Compare commits

...

118 Commits

Author SHA1 Message Date
didericis-claude 4d51b1aa02 fix(claude): don't log the raw identity token in supervise MCP recovery hint
tracker-policy-pr / check-pr (pull_request) Successful in 9s
lint / lint (push) Successful in 2m52s
test / integration-docker (pull_request) Successful in 14s
test / unit (pull_request) Successful in 39s
test / integration-firecracker (pull_request) Successful in 3m45s
test / coverage (pull_request) Successful in 1m0s
test / publish-infra (pull_request) Has been skipped
The previous fix interpolated the real per-bottle identity token into the
`warn()` message shown when `claude mcp add supervise` fails, so a
registration failure would leak the credential into host terminal output
and any collected launch logs (#476 review, PR #471 comment).

Render a `<bottle-identity-token>` placeholder instead. The recovery hint
still shows the required `--header x-bot-bottle-identity: …` shape, and the
operator substitutes the value from inside the bottle (it rides in the
agent's HTTPS_PROXY credentials) — so the token never reaches the host log.
The token truthiness still gates whether the header is shown at all.

Test updated to assert the header name and placeholder are present and the
raw token is absent.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:27:18 +00:00
didericis-claude 7169db1ebe fix(claude): include identity header in supervise MCP fallback recovery command
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Successful in 40s
test / unit (pull_request) Successful in 44s
lint / lint (push) Successful in 51s
test / integration-firecracker (pull_request) Successful in 3m51s
test / coverage (pull_request) Successful in 22s
test / publish-infra (pull_request) Has been skipped
The manual registration command shown in the fallback warning when
`claude mcp add supervise` fails was missing the `--header
x-bot-bottle-identity: <token>` option. Since (source_ip,
identity_token) attribution is now mandatory on the supervise server,
the missing header caused every subsequent supervise request to fail
closed as unattributed — defeating supervision for the entire bottle.

Reuse the already-constructed `token` variable to build the same
`manual_header` string the automatic path uses; include it in the
printed command. Tests verify both the happy path and the fallback
include (or correctly omit) the header.

Addresses P2 finding in #471 (comment 7, didericis-codex 2026-07-25).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 22:20:49 +00:00
didericis 26f4d484d5 Merge remote-tracking branch 'origin/fix/db-off-data-plane-469' into fix/db-off-data-plane-469
tracker-policy-pr / check-pr (pull_request) Successful in 9s
test / integration-docker (pull_request) Successful in 16s
test / unit (pull_request) Successful in 43s
lint / lint (push) Successful in 2m44s
test / integration-firecracker (pull_request) Successful in 3m42s
test / coverage (pull_request) Successful in 15s
test / publish-infra (pull_request) Has been skipped
# Conflicts:
#	tests/unit/test_macos_infra.py
2026-07-25 18:01:49 -04:00
didericis e3376b8809 chore(backend): drop dead code left by the InfraService refactor
Remove symbols with no remaining consumer:

- `gateway_name` property on Docker/MacosInfraService — nothing uses it; the
  launch flow reads `gateway().name` off the Gateway service. (+ its docker test.)
- firecracker `infra_vm` EGRESS_PORT / SUPERVISE_PORT / GIT_HTTP_PORT — dead
  duplicates; launch.py uses supervisor.types / docker.egress / a local const.
- macOS `INFRA_DB_VOLUME` back-compat alias — unused since the DB-volume test
  moved to `ORCHESTRATOR_DB_VOLUME`.
- docker `infra.py`: the unused `ORCHESTRATOR_SOURCE_HASH_LABEL` import and the
  `__all__` re-exports of the orchestrator constants (consumers import them from
  `docker.orchestrator`, not `docker.infra`).
- macOS `infra.py`: the unused `GatewayError` import + re-export.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 16:47:18 -04:00
didericis afb92ca155 refactor(backend): InfraService ABC + FirecrackerInfraService
Add a backend-neutral `InfraService` ABC (backend/infra_service.py) so all three
backends present the same composer contract: `orchestrator()` / `gateway()`
accessors, `ensure_running(*, startup_timeout) -> str` (the host control-plane
URL), `stop()`, plus concrete `url()` / `is_healthy()` that delegate to the
orchestrator. `DockerInfraService` and `MacosInfraService` now subclass it.

Give firecracker the missing class: `FirecrackerInfraService`
(backend/firecracker/infra.py) wraps the `infra_vm` substrate as the pair
coordinator (adopt-or-boot-both under the singleton flock). `infra_vm` keeps only
the plane-agnostic substrate; its `ensure_running` / `_adopt` / `InfraEndpoint`
move to the class, and the coordinator helpers it now calls cross-module go
public (`singleton_lock` / `expected_version` / `adoptable` /
`record_booted_version`).

Unify the return type on the way: `ensure_running` returns the URL string
everywhere (was `str` for docker, two different `InfraEndpoint`s for
macOS/firecracker), and those `InfraEndpoint`s are deleted — the services are the
source of truth (callers read `gateway().address()` / `orchestrator().url()`).
docker's `url` property becomes the inherited `url()`. backend.py /
consolidated_launch / image_builder + tests updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 16:10:21 -04:00
didericis cb2d778a8f refactor: remove leftovers from the orchestrator/gateway consolidation
Sweep for vestiges of the old combined-plane model and the pre-split shared
rootfs. Two are load-bearing, the rest are stale docs/comments:

- Bug: macOS `enumerate_active` only excluded the gateway container from the
  agent list, so after the split the orchestrator container
  (`bot-bottle-mac-orchestrator`, also `bot-bottle-`-prefixed) was enumerated as
  a phantom agent. Exclude both infra containers; test covers it.
- Dead code: the gateway `bootstrap.py` still carried an `orchestrator` daemon
  spec + `_OPT_IN_DAEMONS` + a signing-key/JWT env branch, all for the old
  combined container where the gateway process could also run the control plane.
  No backend ever requests it now — removed; the key-stripping stays as
  defense-in-depth.

Stale-comment reframes: "the/single infra container" -> the orchestrator +
gateway pair (or the specific plane); "shared rootfs / bb_role init / one
published rootfs" -> the per-plane rootfs + `role_init`; the deleted
Dockerfile.infra references in Dockerfile.orchestrator/.gateway; and the macOS
"one infra container ... same address" docstring + its now-false
share-one-address test (the planes are distinct containers with distinct
addresses).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 15:48:16 -04:00
didericis d0d1da612e refactor(firecracker): split the combined rootfs into per-plane artifacts
Each infra VM now boots its own rootfs instead of a shared combined image, so
the exposed gateway VM no longer carries buildah + the control-plane code it
never runs (a fatter, less-isolated exposed surface — the opposite of the plane
split's intent). The combined image bought only "one artifact"; each VM already
kept a full copy of the shared rootfs, so nothing was saved at boot.

  * orchestrator rootfs — control plane + buildah (Dockerfile.orchestrator.fc,
    FROM orchestrator); the slim gateway rootfs boots bot-bottle-gateway:latest
    directly. Dockerfile.infra.fc (the combined image) is deleted.
  * `_infra_init` splits into `_orchestrator_init` / `_gateway_init` (shared
    preamble via `_init_head`); each per-plane rootfs bakes only its role init,
    so the `bb_role` cmdline branch is gone.
  * infra_artifact is role-parametrized: a per-role package
    (bot-bottle-firecracker-<role>), version hash, URL, cache dir, and candidate
    subdir. `ensure_built` pulls both; `_expected_version` combines both markers.
  * publish_infra builds the images once, then builds + publishes an
    orchestrator and a gateway artifact under DIR/<role>/.

coverage.sh's candidate-dir plumbing is layout-agnostic (publish_infra --output
now fills DIR/<role>/, which ensure_artifact_gz reads). Full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 15:21:03 -04:00
didericis b7599ed146 refactor(firecracker): drop the orphaned Dockerfile.infra intermediate
The orchestrator/gateway container split retired the docker backend's combined
`bot-bottle-infra` container, leaving `Dockerfile.infra` used only as the base
layer for the firecracker rootfs image — and its header still (falsely) claimed
"used directly by the Docker backend."

Fold its one `COPY --from orchestrator` into `Dockerfile.infra.fc` (now
`FROM bot-bottle-gateway` directly + the copy + buildah), delete Dockerfile.infra,
and drop the now-dead build step + artifact-version entry. `build_infra_images_with_docker`
builds three images instead of four; the firecracker rootfs is unchanged in
content (still gateway + orchestrator package + buildah, one artifact).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 15:05:04 -04:00
didericis 8d583789d2 refactor(firecracker): orchestrator under the Orchestrator service ABC
Pull the control plane's host-side logic out of `infra_vm` into
`FirecrackerOrchestrator` (backend/firecracker/orchestrator.py): booting the
orchestrator microVM on its link with the persistent registry volume (/dev/vdb),
seeding the host-canonical signing key over SSH, waiting for /health, and the
buildah SSH target. `url()` == `gateway_url()` (the gateway resolves the
orchestrator by guest IP via bb_orch). This completes the trio — docker, macOS,
firecracker — behind both the Gateway and Orchestrator ABCs.

`infra_vm` stays the pair coordinator + shared substrate: `ensure_running` now
mints nothing itself — it constructs both services (lazy import to break the
cycle), calls `orchestrator.ensure_running()` first, then
`gateway.connect_to_orchestrator(orch.gateway_url(), orch.mint_gateway_token())`.
The plane-agnostic boot primitives graduate to public API (`_boot_vm`/
`_push_secret` -> `boot_vm`/`push_secret`) now that they're consumed only across
modules; `InfraVm` loses its `orchestrator_url` (moved to the service) and
`InfraEndpoint.orchestrator` is now the `FirecrackerOrchestrator` service.

Container-lifecycle tests split into test_firecracker_orchestrator; the infra_vm
tests keep the substrate + pair-coordinator coverage.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:54:28 -04:00
didericis 5465379654 refactor(orchestrator): macOS orchestrator under the Orchestrator ABC
Extract the Apple-Container control-plane lifecycle out of `MacosInfraService`
into `MacosOrchestrator` (backend/macos_container/orchestrator.py): the
container run on the host-only control network, the container-only DB volume,
the source-hash recreate gate, health polling against the resolved
control-network address, and `probe_orchestrator_url`. `url()` == `gateway_url()`
here (Apple has no container DNS, so one resolved address serves the CLI and the
gateway).

`MacosInfraService` now composes `orchestrator()` + `gateway()`: ensure the
networks, build both images (each service self-builds via `ensure_built` — added
to `MacosGateway` too), bring the orchestrator up first, then connect the gateway
with the orchestrator-minted token. `ORCHESTRATOR_IMAGE` + the DB-volume constant
move to the orchestrator module (with back-compat aliases where imported).

Container-lifecycle tests split into test_macos_orchestrator; test_macos_infra
now covers the composition.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:44:21 -04:00
didericis 06718ca761 fix(firecracker): reap legacy pre-split infra VM on cutover
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 59s
test / unit (pull_request) Successful in 1m53s
test / integration-firecracker (pull_request) Successful in 3m36s
test / coverage (pull_request) Successful in 14s
test / publish-infra (pull_request) Has been skipped
The two-VM stop() only knew the new orchestrator/gateway config paths
(<infra>/{orchestrator,gateway}/config.json), so a host still running the
OLD single combined-VM (booted from <infra>/config.json on the orchestrator
link) kept it alive across the migration — and it held the orchestrator TAP,
so the first split boot died with "Open tap device failed: Resource busy"
(seen on the CI runner's bborchci0 pool).

stop() now also reaps that legacy singleton (both its drifted pidfile and any
firecracker whose --config-file is the legacy path), so the cutover is
self-healing with no manual per-host teardown. Scoped to the legacy infra
config path, so pool agent VMs are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:37:32 -04:00
didericis cc4e29c3da refactor(orchestrator): introduce the Orchestrator service ABC (docker impl)
Mirror the Gateway work for the control plane. Add the backend-neutral
`Orchestrator` service ABC in orchestrator/lifecycle.py — build the image/rootfs,
`ensure_running` (start + health-gate), `is_running`/`is_healthy`/`stop`,
`url()` (host-facing) vs `gateway_url()` (what the gateway resolves against),
and a concrete `mint_gateway_token()` (the orchestrator holds the signing key,
so it issues the gateway's token — #469).

`DockerOrchestrator` (backend/docker/orchestrator.py) is the first impl,
extracted out of `DockerInfraService`: the control-plane container run, the
`--internal` control network, the source-hash recreate gate, health polling,
and the orchestrator constants (name/label/network/image/dockerfile/hash-label).
`DockerInfraService` now composes `orchestrator()` + `gateway()` — each builds
its own image via `ensure_built`, the orchestrator comes up first, then the
gateway is connected with the orchestrator-minted token. Its `url`/`is_healthy`
delegate to the orchestrator service.

Split the container-lifecycle tests into test_docker_orchestrator; test_docker_infra
now covers the composition. macOS + firecracker follow.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:36:35 -04:00
didericis 57d32d2c5c refactor(orchestrator): rename in-guest core Orchestrator -> OrchestratorCore
Free the `Orchestrator` name for the incoming host-side control-plane service
ABC (parallel to `Gateway`). The in-guest control-plane core (registry + broker,
in orchestrator/service.py) becomes `OrchestratorCore`; server.py, __main__.py,
the lazy facade, and the two service/server tests updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:29:32 -04:00
didericis 07c975636d refactor(backend): verb-first names for the provisioning modules
Rename the module files to match their functions' verb-first names:
`bottle_provision.py` -> `provision_bottle.py`,
`gateway_provision.py` -> `provision_gateway.py` (and the test file to match).
Imports, docstrings, and the git_gate/service.py pointer updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:13:54 -04:00
didericis 96ab8e15a2 refactor(backend): rename teardown_consolidated -> deprovision_consolidated
The backends' public teardown entry point now pairs with `launch_consolidated`
under the provision/deprovision verb used everywhere else
(`deprovision_bottle`, `deprovision_git_gate`). Renamed across all three
backends' consolidated_launch + launch modules and the tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:07:57 -04:00
didericis a98f559bbe refactor(backend): split bottle vs gateway provisioning modules
Rename `consolidated_util` -> `bottle_provision` and align it with
`gateway_provision` as the two provisioning domains: `bottle_provision` owns
the bottle lifecycle (`provision_bottle` + `deprovision_bottle`, the latter
renamed from `teardown_consolidated`), `gateway_provision` owns the git-gate
state pushed through the transport (`provision_git_gate`/`deprovision_git_gate`).

The three backends' consolidated_launch modules import from the new module;
their own public `teardown_consolidated` wrappers (called by launch.py) keep
their names.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 14:05:17 -04:00
didericis 343f3a0735 refactor(backend): move git-gate provisioning to a neutral module
`backend/docker/gateway_provision.py` had no docker-specific code left once
`DockerGatewayTransport` moved out — `provision_git_gate` / `deprovision_git_gate`
drive any `GatewayTransport`, and the guest-side paths they write
(`/git-gate/creds/<id>`, `/git/<id>`, `/etc/git-gate/...`) are identical inside
every backend's gateway. Yet the neutral `backend/consolidated_util.py` reached
into the docker package to import them.

Move it to `backend/gateway_provision.py`, a sibling of its only importer. The
gateway *package* can't host it (git_gate already imports gateway.git_gate,
so gateway importing git_gate would cycle), but the backend layer uses git_gate
freely. Docstrings + the git_gate/service.py pointer updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:59:11 -04:00
didericis e1fd8ef1b4 refactor(gateway): each backend's transport in its own gateway_transport.py
Give every backend a `gateway_transport.py` holding just its `GatewayTransport`
impl, named consistently with the backend: `DockerGatewayTransport`,
`MacosGatewayTransport` (was `AppleGatewayTransport`), and
`FirecrackerGatewayTransport` (was `SshGatewayTransport`).

- docker: split `DockerGatewayTransport` out of `gateway_provision.py`, which
  keeps only the backend-neutral provisioning logic (provision_git_gate /
  deprovision_git_gate) the other backends share.
- macOS: `gateway_provision.py` held only the transport, so it's replaced by
  `gateway_transport.py`.
- firecracker: move the transport out of `gateway.py`.

Behaviour-preserving; importers + tests updated.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:52:31 -04:00
didericis 6e90522664 chore(docker): remove the empty provision subpackage
The `backend/docker/provision` package held no modules — a placeholder kept
"in case backend-specific provisioners are added later" after PRD 0050 moved
provisioning to the AgentProvider plugin. Nothing imports it, so drop it; a
new package is cheap to add if that need ever materializes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:44:34 -04:00
didericis 5bf9f052b9 refactor(firecracker): move gateway logic into the gateway service
Pull the gateway's host-side logic out of `infra_vm` and into
`FirecrackerGateway`'s own methods, so the gateway is self-contained and
`infra_vm` shrinks toward being just the pair coordinator (a step toward
removing it). Now living in the gateway service: the gateway VM boot +
`bb_orch` cmdline, the pre-minted JWT push, the mitmproxy CA fetch over SSH,
the `SshGatewayTransport` exec/cp, and the lifecycle predicates
(is_running/stop/address). `ensure_running` / `_adopt` construct the gateway
and call `connect_to_orchestrator` (lazy import to break the cycle); the
InfraEndpoint's gateway is now the `FirecrackerGateway` service.

What deliberately stays shared in `infra_vm` (documented at the top of
gateway.py): the plane-agnostic VM substrate the orchestrator VM also needs
(`_boot_vm`, the stable SSH keypair, the secret-push retry, PID lifecycle), the
pair coordinator (`ensure_running`: orchestrator-first health gate + singleton
lock + adoption/version marker), and the single shared `bb_role`-branched init
baked into the one published rootfs both VMs boot — the guest-side gateway
startup can't live in a host method. These are the seam that moves to a neutral
module when the Orchestrator service lands and `infra_vm` dissolves.

CA fetch now raises GatewayError (was die/SystemExit) to match the ABC.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:34:36 -04:00
didericis d54c559a3a refactor(gateway): Firecracker gateway under the Gateway service ABC
Add `FirecrackerGateway`, the Firecracker implementation of the shared
`Gateway` service — completing the trio (docker, macOS, firecracker) behind one
uniform contract. The gateway here is a whole microVM, so the adapter composes
`infra_vm`'s VM primitives (boot, SSH secret push, netpool links) into the ABC
surface: `connect_to_orchestrator(url, token)` parses the orchestrator guest IP
off the URL and boots the gateway VM seeding the pre-minted token; `address()`
is the gateway link's guest IP; `ca_cert_pem()` fetches over SSH (adapting the
running VM when this process didn't boot it); `provisioning_transport()` is the
SSH exec/cp transport.

Trust-model alignment with the other backends: minting moves out of
`_push_gateway_jwt` to the host composition point — `ensure_running` mints the
role-scoped `gateway` JWT and threads it through `boot_gateway(orch_ip, token)`,
so the push step only writes the token the gateway presents (#469).
`infra_vm` keeps driving the orchestrator+gateway pair as a singleton and gains
gateway-only helpers (`gateway_running`/`stop_gateway`/`gateway_guest_ip`/
`adopted_gateway_vm`); the consolidated launch reads the gateway's transport +
CA off the service.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:12:39 -04:00
didericis c4ccd74f9b refactor(gateway): macOS gateway under the Gateway service ABC
Extract the Apple-Container gateway data plane out of `MacosInfraService` into
`MacosGateway`, the macOS implementation of the shared `Gateway` service.
`connect_to_orchestrator(url, gateway_token)` runs the triple-homed gateway
container (egress + agent + control networks) carrying the mitmproxy CA and the
pre-minted `gateway` token; `address()` / `ca_cert_pem()` /
`provisioning_transport()` round out the contract.

The infra service now composes the gateway: `ensure_running` mints the
role-scoped `gateway` JWT (it holds the signing key; the gateway never does —
#469) and hands it to `gateway().connect_to_orchestrator(...)`, and
`ca_cert_pem` delegates to the service. The inlined `_ensure_gateway_container`
+ CA-read are gone. `AppleGatewayTransport` + the gateway container name/label
now live with the gateway module, breaking the old infra→provision coupling.

Splits the gateway-run + CA tests out of test_macos_infra into a dedicated
test_macos_gateway; the infra tests mock `svc.gateway`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 13:04:53 -04:00
didericis 05df21f210 refactor(gateway): introduce the Gateway service ABC (docker impl)
Replace the docker backend's ad-hoc gateway wiring with the shared `Gateway`
service ABC (in the gateway package), the first of the two per-host service
classes that supersede the per-backend infra glue.

The contract is backend-neutral: `connect_to_orchestrator(url, gateway_token)`
binds the gateway to its control plane and brings it up, `address()` reports
the agent-facing proxy target, `ca_cert_pem()` vends the mitmproxy CA, and
`provisioning_transport()` hands back the exec/cp seam git-gate provisioning
stages per-bottle repos + deploy keys through. `GatewayProvisionError` +
`GatewayTransport` move to the ABC so every backend shares them.

Key trust-model change: the gateway no longer mints its own token. It never
holds the signing key (#469), so the orchestrator mints the role-scoped
`gateway` JWT and hands it in via `connect_to_orchestrator`; the gateway only
injects it. `DockerGateway.ensure_running` becomes `connect_to_orchestrator`
(stashing the URL + token as instance state), and the docker launch flow reads
`address()` / `provisioning_transport()` off the service rather than
re-deriving the container IP + transport itself.

macOS + firecracker gateways move under the ABC in following commits.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:58:00 -04:00
didericis eb9723027b test(macos): mock ensure_networks in TestInfraEnsureRunning
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Successful in 16s
test / unit (pull_request) Successful in 43s
test / integration-firecracker (pull_request) Failing after 2m43s
lint / lint (push) Successful in 2m46s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
These four ensure_running() tests patched container_mod but not
ensure_networks (imported into infra.py from .gateway), so they shelled out
to the real Apple `container` CLI and errored on Linux
(FileNotFoundError: 'container') — the dev + CI platform for this branch.
Mock it so they're true unit tests, green on any platform.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 12:01:01 -04:00
didericis 18d9b81add feat(firecracker): split orchestrator and gateway into separate VMs (PRD 0070)
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 53s
test / unit (pull_request) Failing after 1m46s
test / integration-firecracker (pull_request) Failing after 2m42s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
Now that #469 got the DB off the data plane, the Firecracker infra runs as
two microVMs instead of one — mirroring the docker/macos plane split:

  * orchestrator VM (ORCH_IFACE) — control plane + buildah image builds; sole
    DB opener; host-seeded signing key. No gateway daemons.
  * gateway VM (new GW_IFACE) — egress / git-http / supervise data plane;
    mitmproxy CA + a host-minted `gateway` JWT (never the key). Reaches the
    orchestrator only over the one nft forward rule its link allows.

Both boot the SAME shared infra rootfs; a `bb_role=` kernel-cmdline arg
selects which plane a VM's PID-1 init starts, so there is still one published
artifact. The gateway learns the orchestrator's address via `bb_orch=` on the
cmdline (no IP baked into the artifact).

Isolation is nearly free: agents were already nft-dropped except the DNAT'd
gateway ports, so re-pointing that single DNAT rule at the gateway VM
(`dnat to gw_guest`) severs every agent's L3 route to the control plane. The
only added nft is the second infra link's mirror block (masquerade egress +
forward accept, which subsumes gateway->orchestrator) in the shared shell
script and the NixOS module.

netpool gains GW_IFACE + gw_slot() (the /31 above the orch link);
firecracker_vm.boot gains extra_boot_args for the role cmdline; infra_vm
ensure_running() boots + adopts the pair (orchestrator first, then the gateway
that resolves policy against it) and returns an InfraEndpoint mirroring the
docker/macos shape. Builds stay in the orchestrator (PRD 0070 v1); the gateway
is the slim unit.

Unit-tested (test_firecracker_infra_vm rewritten for two VMs; gw_slot helper
test added); the KVM boot / L3-isolation checks are validated on a Firecracker
host.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 03:44:15 -04:00
didericis a74894c6f6 docs(prd-0070): correct the firecracker-split difficulty framing
test / integration-docker (pull_request) Successful in 15s
tracker-policy-pr / check-pr (pull_request) Successful in 15s
test / unit (pull_request) Failing after 36s
test / integration-firecracker (pull_request) Successful in 3m23s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
Reevaluated against the actual nft ruleset: the firecracker split's isolation
is nearly free, not "the real work." Agent VMs are already dropped except the
DNAT'd gateway ports, so re-pointing that single DNAT rule at the gateway VM
(dnat to $(gw_guest)) isolates agents from a separate orchestrator VM with zero
new agent rules; the only added nft is a mirrored gateway link + one
gateway->orchestrator forward rule. The effort is the mechanical second-VM
lifecycle, and it's validatable on a Firecracker host.

Sharpened the Access bullet accordingly and added a note to Sequencing
decoupling the original consolidation's "real work" (the broker shim) from the
split, whose difficulty actually runs the other way (docker/macOS do the real
work — new control network, dual-homed gateway, the token-only->L3 upgrade;
firecracker is nearly free).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 03:17:24 -04:00
didericis b676ce3156 feat(macos): split orchestrator and gateway into separate containers (PRD 0070)
test / integration-docker (pull_request) Successful in 18s
tracker-policy-pr / check-pr (pull_request) Successful in 22s
test / unit (pull_request) Failing after 42s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m22s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
The single Apple container existed only because two guests writing one
bot-bottle.db over virtiofs would race incoherent fcntl locks; #469 removed
that (the data plane no longer opens the DB), so the macOS backend now runs
the planes as two containers like docker:

  * bot-bottle-mac-orchestrator — lean control plane on the host-only
    `bot-bottle-mac-control` network only (image Dockerfile.orchestrator,
    `-m bot_bottle.orchestrator`). Sole mounter of the container-only DB
    volume; holds the signing key. The CLI + gateway reach it at its
    control-network address.
  * bot-bottle-mac-infra — the gateway, triple-homed on the NAT egress net,
    the host-only agent net, and the control net. Resolves the orchestrator by
    IP (Apple has no container DNS) via BOT_BOTTLE_ORCHESTRATOR_URL; holds the
    CA + gateway JWT.

Agents sit on the agent network only, so they have no route to the control
plane. ensure_networks gains the control network; MacosInfraService brings up
the orchestrator then the gateway; probe_orchestrator_url + ca_cert_pem target
the right containers. pyright 0 errors; unit suite green (2274). Needs a real
Apple-container host to validate the networking.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 02:39:37 -04:00
didericis d62d19a2e7 feat(docker): split orchestrator and gateway into separate containers (PRD 0070)
tracker-policy-pr / check-pr (pull_request) Successful in 6s
test / integration-docker (pull_request) Successful in 13s
test / unit (pull_request) Failing after 37s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m19s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
Now that #469 got the DB off the data plane, the docker backend runs the
control plane and data plane as two containers instead of the combined
`bot-bottle-infra` container:

  * bot-bottle-orchestrator — the lean control-plane container (image
    Dockerfile.orchestrator, `-m bot_bottle.orchestrator`). Joins the new
    `bot-bottle-orchestrator` control network (`--internal`) only, plus a
    host-loopback publish for the CLI. Sole opener of bot-bottle.db; holds the
    signing key; no CA, no gateway daemons.
  * bot-bottle-orch-gateway — the data-plane container (DockerGateway),
    dual-homed on the agent `bot-bottle-gateway` network AND the control
    network, so it resolves the orchestrator by name
    (http://bot-bottle-orchestrator:8099) while agents — never on the control
    network — have no route to the control plane (the L3 block, not just the
    JWT). Holds the gateway JWT + the mitmproxy CA.

DockerInfraService now orchestrates the pair (builds gateway + orchestrator
images, brings up the orchestrator then the gateway) and exposes gateway_name;
the launch flow attributes agents against the gateway container. DockerGateway
gains a control_network it joins after run. rotate_ca / the CA read target the
gateway container. The combined Dockerfile.infra is no longer built by docker
(firecracker/macOS still use it until their splits).

pyright 0 errors; unit suite green (2273). Integration tests updated for the
two-container shape but need a real-docker run to validate the networking.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 02:32:05 -04:00
didericis 7e9ad8a78d Merge main into fix/db-off-data-plane-469
tracker-policy-pr / check-pr (pull_request) Successful in 13s
test / integration-docker (pull_request) Successful in 35s
test / unit (pull_request) Successful in 47s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 32s
test / publish-infra (pull_request) Has been skipped
Brings in main's 8 commits — #470 (backend-agnostic CI guards: BackendStatus
enum, quiet status(), is_backend_available/is_backend_ready, tests/_backend.py
skip_unless_backend) plus firecracker status() readiness checks (kvm/kernel/
dropbear/mke2fs).

Reconciled #470's additions into the reorg'd backend structure:
  * BackendStatus enum + the status(*, quiet=) ABC signature -> backend/base.py
  * is_backend_available / is_backend_ready -> backend/selection.py
  * exposed all three through the lazy backend facade (__init__).
Integration-test conflicts were our control_plane->orchestrator renames vs
main's skip_unless_docker -> skip_unless_backend swap: kept our refactored
import paths + main's guard. Repointed main's new is_backend_* tests to patch
selection._backends (our moved home) instead of the facade.

pyright: 0 errors. Full unit suite green (2272).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 21:35:58 -04:00
didericis-claude a68ee778f1 firecracker status(): open /dev/kvm O_RDWR; check kernel, dropbear, mke2fs
test / integration-docker (push) Successful in 15s
test / unit (push) Successful in 40s
Update Quality Badges / update-badges (push) Successful in 44s
lint / lint (push) Successful in 58s
test / integration-firecracker (push) Successful in 5m5s
test / coverage (push) Successful in 49s
test / publish-infra (push) Successful in 2m4s
Two reviewer findings addressed:

1. _kvm_accessible() now opens /dev/kvm with O_RDWR|O_CLOEXEC instead
   of read-only. VM creation requires write access; a read-only
   descriptor can satisfy KVM_GET_API_VERSION but fails at boot time.

2. status() now checks every hard prerequisite that require_firecracker()
   checks at launch: guest kernel image, static dropbear binary, and
   mke2fs. Previously a host with a configured TAP pool but missing
   artifacts could pass status() and then fail during launch.

Regression coverage: TestFirecrackerKvmCheck gains test_kvm_open_rdwr_fails;
TestFirecrackerArtifactCheck covers each missing artifact; existing
TestFirecrackerStatus fixtures updated to stub the new checks.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis-claude de02d13ccf firecracker status(): add binary and KVM readiness checks
Adds `_firecracker_binary_ok()` (runs `firecracker --version`) and
`_kvm_accessible()` (opens /dev/kvm, issues KVM_GET_API_VERSION ioctl)
and gates `status()` on both, so the launch preflight reports missing
binary or inaccessible KVM before the caller ever attempts a boot.

Regression tests in TestFirecrackerBinaryCheck, TestFirecrackerKvmCheck,
and TestFirecrackerStatusRuntime cover all branches. Updated the
pre-existing TestFirecrackerStatus fixture to also stub the two new
helpers so it still exercises only the tap-pool/nft logic it was
designed for.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis-claude c740d1e145 fix: add missing type annotations and remove unused variables in new tests
Pyright flagged three inner class `status` methods missing a type
annotation on the `quiet` parameter, and two unused variables
(`written`, `real_status`) left over from an earlier draft of
test_docker_quiet_true_suppresses_stderr.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis-claude 177721c286 test: cover is_backend_available, is_backend_ready, and status(quiet=True)
Add unit tests for the three new public symbols introduced by this PR:

- TestIsBackendAvailable — delegates to has_backend; returns True/False
  correctly for available and unavailable backends
- TestIsBackendReady — unknown names return False; known names return
  True/False based on status() return code; quiet flag is forwarded
- TestBackendStatusQuiet — verifies the quiet=True branch in
  DockerBottleBackend, FirecrackerBottleBackend, and
  MacosContainerBottleBackend: return code is propagated and
  diagnostic stderr is suppressed

Together these bring diff-coverage for the PR's changed lines to 100%.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis-claude 5bcf3db1f8 fix: align skip-guard tests with is_backend_ready and drop unused import
tests/_backend.py imported is_backend_available but never called it,
causing a pyright reportUnusedImport error. Remove the unused import.

test_backend_skip_guards.py was patching tests._backend.has_backend but
_backend.py calls is_backend_ready — the mock never intercepted the real
call, producing AttributeError at test runtime. Update all patch targets
to is_backend_ready and add quiet=False to the assert_called_once_with
assertion to match the actual call signature.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis-claude dcd658ece1 feat(backend): BackendStatus enum, quiet status(), is_backend_ready()
Add a module-level BackendStatus(IntEnum) with READY=0 to replace magic
0 comparisons. Extend BottleBackend.status() with a quiet parameter:
quiet=False (default) prints diagnostics; quiet=True returns the code
silently for programmatic checks.

Add is_backend_available() (cheap PATH check, alias for has_backend) and
is_backend_ready(name, *, quiet=False) (full status() check) to the
backend package for callers that need to distinguish availability from
readiness.

Update tests/_backend.py guards to use is_backend_ready(quiet=False) so
diagnostic output is printed during test discovery, giving the operator a
concrete reason for each skip rather than a bare "prerequisites
unavailable" message.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis-claude dfb56f8fe9 ci: reuse backend.has_backend / backend status for readiness
Review feedback: the hand-rolled `/dev/kvm` + docker-daemon capability
probes duplicated logic the CLI already owns. Delegate instead.

- tests/_backend.py skip guards now gate on `bot_bottle.backend.has_backend`
  (each backend's `is_available()` classmethod — the same probe behind
  `./cli.py backend status`), dropping the bespoke `Capability` probes.
- Remove tests/backend_preflight.py; the docker integration job runs
  `./cli.py backend status --backend=docker` as its preflight (clear
  per-check summary, non-zero exit when unready), matching the firecracker
  job. The firecracker preflight reverts to its original binary/KVM checks
  (backend status doesn't cover those).
- Unit test + docs updated to match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis-claude 64adf23775 ci: backend-agnostic integration guards + per-backend preflight
Integration tests now select their backend from BOT_BOTTLE_BACKEND and
skip on the capability that backend actually needs, instead of gating
every backend on unrelated Docker availability.

Task 1 — backend-agnostic guards (tests/_backend.py):
- Capability probes: docker_capability() (reachable daemon) and
  firecracker_capability() (accessible /dev/kvm + firecracker on PATH,
  Docker-independent). backend_capability()/selected_backend() resolve
  the target from BOT_BOTTLE_BACKEND (default docker).
- skip_unless_selected_backend_available() for backend-agnostic tests
  (test_sandbox_escape) — runs through whichever backend is selected and
  checks that backend's real capability.
- skip_unless_backend("docker") for Docker-implementation tests
  (DockerBroker, DockerGateway, backend.docker.*) — they no-op under a
  non-Docker run rather than testing internals that run doesn't target.
- Retires tests/_docker.py; the KVM job no longer needs SKIP_DOCKER_TESTS
  to steer Docker-only classes.

Task 2 — explicit per-backend skip visibility:
- tests/backend_preflight.py prints a clear PASS/FAIL capability line and
  exits non-zero when the selected backend is missing.
- Both integration jobs run it as a preflight, so absent infrastructure
  is surfaced at the job level instead of hidden among unittest.skip
  lines. The docker job replaces its soft "Show environment" step; the
  firecracker job keeps its richer backend-status check.

Docs (tests/README.md, docs/ci.md) updated; unit coverage for the probes,
guards, and preflight in test_backend_skip_guards.py.

Closes #414

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 21:25:20 -04:00
didericis 3d3d8fd7e8 docs(prd-0070): record remote-terminal resolution; track detail in #478
tracker-policy-pr / check-pr (pull_request) Successful in 19s
test / integration-docker (pull_request) Successful in 42s
test / unit (pull_request) Successful in 52s
test / integration-firecracker (pull_request) Successful in 3m32s
test / coverage (pull_request) Successful in 26s
test / publish-infra (pull_request) Has been skipped
Remote terminal is an orchestrator operation, not a data-plane one: the
agent PTY is exec_agent-sourced (ssh -t / container exec --tty / docker
exec), which the orchestrator owns — the gateway never holds a PTY. A
--remote flag publishes a running agent's session; local + remote clients
share it via a session multiplexer (one PTY, many clients),
operator-role-gated, so the split's "one authenticated remote door" holds
for terminals too. Detailed design (multiplexer location, input
arbitration, per-client rendering, always-on vs --remote) tracked in #478.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 21:14:20 -04:00
didericis 195e0f249d docs(prd-0070): resolve build-placement open question — builds stay in the orchestrator
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 31s
test / unit (pull_request) Successful in 46s
test / integration-firecracker (pull_request) Successful in 3m23s
test / coverage (pull_request) Successful in 42s
test / publish-infra (pull_request) Has been skipped
v1 decision: agent-image builds run in the orchestrator (the control-plane
side already owns launches and hosts builds in today's combined unit; the
data-plane gateway must not build). Consequence recorded in the memory
section: the orchestrator keeps the ~4 GB buildah ceiling and the gateway
is the slim unit (~1 GB), so the split is ~memory-neutral-to-+1 GB for now
— the ~1.5 GB win waits on a later move to a dedicated slim build unit
(#468-adjacent). Also flags the tension of co-locating buildah with the
control plane. Removed the corresponding open question.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 20:59:30 -04:00
didericis ee26e9044f docs(prd-0070): fold in the orchestrator/gateway plane-separation design
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / integration-docker (pull_request) Successful in 41s
test / unit (pull_request) Successful in 46s
test / integration-firecracker (pull_request) Successful in 3m40s
test / coverage (pull_request) Successful in 20s
test / publish-infra (pull_request) Has been skipped
Now that #469 got the DB off the data plane (removing the cross-kernel
SQLite-coherence reason the planes were consolidated), separating them
into distinct runtimes is possible again. Adds a Design subsection to
0070 covering:

  * Access — the only hop that changes is gateway->orchestrator (loopback
    -> network). A dedicated `bot-bottle-orchestrator` --internal control
    network with a dual-homed gateway gives Docker/macOS the L3 agent
    isolation that only Firecracker's nft currently has (both are
    token-only today); Firecracker gets it nearly free (agents already
    can't reach non-DNAT'd ports).
  * Memory — firecracker's mem_size_mib is a fixed ceiling (no reclaim
    without a balloon); the 4 GB combined VM is really buildah headroom.
    Right-sized: ~512 MiB orchestrator + ~1 GB gateway < today's 4 GB,
    if agent-image builds move off the gateway.
  * iroh belongs in the orchestrator (control-plane remote entry point);
    keep the data-plane gateway slim and P2P-free.

Plus open questions: where agent-image builds run post-split, and whether
agent-PTY-to-mobile relays through the orchestrator's iroh endpoint.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 20:41:34 -04:00
didericis-claude e7fe00e2f5 fix(firecracker): keep the host key canonical instead of clobbering the host token file
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / integration-docker (pull_request) Successful in 21s
lint / lint (push) Successful in 1m3s
test / unit (pull_request) Successful in 1m55s
test / integration-firecracker (pull_request) Successful in 3m29s
test / coverage (pull_request) Successful in 17s
test / publish-infra (pull_request) Has been skipped
Reimplements the dropped firecracker fix on the refactored code, addressing
didericis-codex's blocking review on #471.

The previous firecracker path let the VM generate its own signing key on the
guest volume and had `_with_signing_key()` unconditionally overwrite the single
host-wide orchestrator-token with it. That breaks a co-running Docker/macOS
control plane: their orchestrators still verify with the old key while new host
clients start signing `cli` tokens with the guest key, so
supervise/teardown/policy calls against those backends begin returning 401
(cross-backend operation is supported).

Keep the host token file the single source of truth. The launcher now pushes the
host-canonical key into the freshly booted infra VM over SSH (atomic write,
mirroring persist_env_var_secret) via `_push_signing_key()`; the VM's init waits
for it and refuses to start the control plane — rather than run OPEN — if it
never arrives. Nothing writes back to the host file, so other backends are
untouched, and the best-effort silent path is gone (both the push and the guest
fail loudly).

Ported onto the reorg'd names (host_orchestrator_token, BOT_BOTTLE_ORCHESTRATOR_
TOKEN, orchestrator-token, bot_bottle.orchestrator_auth, bot_bottle.gateway.
bootstrap). pyright: 0 errors. Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 19:47:59 -04:00
didericis 466f4ee13a Merge main into fix/db-off-data-plane-469
tracker-policy-pr / check-pr (pull_request) Successful in 13s
test / integration-docker (pull_request) Successful in 34s
test / unit (pull_request) Successful in 49s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m29s
test / coverage (pull_request) Successful in 20s
test / publish-infra (pull_request) Has been skipped
Brings in the two commits main gained: #472 (CLI cleanup — remove `info`,
split `list active` into a top-level `active`, simplify `list`) and the
terminology glossary. #472 was written against the old flat cli/ layout,
so its semantics are reconciled into the reorg'd cli/commands/ structure:

  * new `active` command lives at cli/commands/active.py, registered in
    the lazy registry and listed in `help` (not left at the flat path).
  * `info` removed: deleted cli/commands/info.py, dropped from the
    registry and `help`.
  * `list` simplified to list-available-only, using the reorg's
    os.getcwd()/constants.PROG conventions.
  * the `list active` -> `active` doc wording applied to the backend
    docstrings that our split moved into base.py / selection.py.

Our reorg wins for the fully-rewritten dispatcher (cli/__init__ shim,
backend facade). pyright: 0 errors. Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 19:35:49 -04:00
didericis bef45348f5 fix(types): satisfy pyright strict on the refactored facades
tracker-policy-pr / check-pr (pull_request) Successful in 13s
test / integration-docker (pull_request) Successful in 33s
test / unit (pull_request) Successful in 51s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m25s
test / coverage (pull_request) Successful in 43s
test / publish-infra (pull_request) Has been skipped
Two facade issues from the reorg:

  * orchestrator/__init__.py used `__all__ = list(_LAZY)`, which pyright
    strict can't analyze (reportUnsupportedDunderAll) — and because it
    isn't a static literal, the TYPE_CHECKING re-export imports read as
    unused. Replaced with the explicit literal list the other lazy
    facades already use.
  * manifest/index.py imported six piece types (ManifestAgentProvider,
    EGRESS_AUTH_SCHEMES, ManifestEgressConfig/Route, ManifestGitEntry,
    ManifestKeyConfig) it never referenced — the package facade re-exports
    those straight from their submodules, so index.py needn't import them.
    Dropped the dead imports.

pyright: 0 errors (was 24). Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 19:11:37 -04:00
didericis e2740842a0 refactor: make the two remaining heavy __init__ modules lazy
tracker-policy-pr / check-pr (pull_request) Successful in 13s
test / integration-docker (pull_request) Successful in 35s
test / unit (pull_request) Successful in 51s
lint / lint (push) Failing after 58s
test / integration-firecracker (pull_request) Successful in 3m32s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
Audit of package __init__ import cost turned up two eager ones:

orchestrator/__init__.py (27 -> 2): was pure `from .X import Y`
re-exports (registry_store, broker, docker_broker, gateway, service,
server). Because importing any submodule runs the parent __init__, the
CLI migration gate (orchestrator.store.store_manager, hit on every
command) transitively dragged backend.docker, docker_broker, server, and
http.server. Converted to the same lazy __getattr__ + _LAZY facade used
by manifest/backend/egress/git_gate; the re-export API is unchanged and
the docker/http drag is gone.

cli/commands/__init__.py (78 -> 23): the registry imported all twelve
handlers to build COMMANDS, so importing one command pulled all of them
plus their deps. COMMANDS now maps each name to a thin lazy wrapper that
imports its handler's module on first dispatch. Values stay callable, so
the dispatcher and the patch.dict dispatch tests are unchanged; a CLI run
now loads only the one command it dispatches. The remaining 23 is the
dispatcher's own baseline (store_manager migration gate + help + log).

Full unit suite green (2243); `bb help` + dispatch/migration-gate tests
verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 18:54:41 -04:00
didericis 3efb014ace refactor(orchestrator): move registry into store/registry_store
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 18s
lint / lint (push) Failing after 58s
test / unit (pull_request) Successful in 2m14s
test / integration-firecracker (pull_request) Successful in 3m25s
test / coverage (pull_request) Successful in 35s
test / publish-infra (pull_request) Has been skipped
registry.py is a SQLite-backed store (its class is already RegistryStore),
so it belongs with the other orchestrator-owned stores rather than at the
package root. Moved orchestrator/registry.py -> orchestrator/store/
registry_store.py, joining queue_store / secret_store / config_store.

Repointed the in-package importers (.registry -> .store.registry_store)
and the tests, bumped the moved file's relative-import depths, renamed
the test file to match, and listed it in the store package docstring.
Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 18:24:20 -04:00
didericis cb3a74edb0 refactor(orchestrator): collapse supervisor package into supervisor.py
orchestrator/supervisor/ held only __init__.py, so the package no longer
earns its own directory. Moved it to orchestrator/supervisor.py and
removed the dir. External imports are unchanged (orchestrator.supervisor
resolves identically as a module); only the file's own relative-import
depths shift up one level. Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 18:22:03 -04:00
didericis ca1d341d4f refactor: unify component naming — control_plane/control_auth -> orchestrator
tracker-policy-pr / check-pr (pull_request) Successful in 14s
test / integration-docker (pull_request) Successful in 18s
test / unit (pull_request) Successful in 46s
lint / lint (push) Failing after 54s
test / integration-firecracker (pull_request) Successful in 3m21s
test / coverage (pull_request) Successful in 18s
test / publish-infra (pull_request) Has been skipped
The codebase used "control plane" both as an architectural role term AND
as an identifier alias for the orchestrator component, producing
duplicate names for one thing (control_plane_url vs orchestrator_url,
CONTROL_PLANE_PORT, host_control_plane_token, …). Going forward the
concrete component is always named for what it is — Gateway or
Orchestrator — and the plane vocabulary is reserved for prose (module
descriptions, the security argument).

Renamed (identifiers + the in-repo env/wire/file string values, all
setters/getters are in this repo so the change is atomic):

  ControlPlaneServer            -> OrchestratorServer
  control_plane_url             -> orchestrator_url
  probe_control_plane_url       -> probe_orchestrator_url
  host_control_plane_token      -> host_orchestrator_token
  CONTROL_PLANE_PORT            -> ORCHESTRATOR_PORT
  CONTROL_PLANE_TOKEN_ENV/FILE  -> ORCHESTRATOR_TOKEN_ENV/FILENAME
  BOT_BOTTLE_CONTROL_PLANE_TOKEN-> BOT_BOTTLE_ORCHESTRATOR_TOKEN
  control-plane-token (file)    -> orchestrator-token

  control_auth (module)         -> orchestrator_auth  (stays top-level;
                                   the gateway imports it and must not
                                   import the orchestrator/ package)
  CONTROL_AUTH_HEADER           -> ORCHESTRATOR_AUTH_HEADER
  x-bot-bottle-control-auth     -> x-bot-bottle-orchestrator-auth
  CONTROL_AUTH_JWT_ENV          -> ORCHESTRATOR_AUTH_JWT_ENV
  BOT_BOTTLE_CONTROL_AUTH_JWT   -> BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT
  _control_auth_headers         -> _orchestrator_auth_headers

Prose plane-terms ("control plane", "data plane") are preserved,
including the test name test_data_plane_daemons_get_jwt_not_key (it
names the security invariant). Gateway and orchestrator verified to
agree on the renamed wire header; full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 18:17:48 -04:00
didericis 4166057abc refactor(orchestrator): rename control_plane module to server
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 50s
lint / lint (push) Failing after 1m0s
test / integration-firecracker (pull_request) Successful in 3m25s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 12m7s
orchestrator/control_plane.py -> orchestrator/server.py. Within the
orchestrator package the "control_plane" filename stutters (the
orchestrator *is* the control plane), and `orchestrator.server` reads as
"the orchestrator's HTTP server", pairing with service.py (domain logic)
and matching gateway/supervisor/server.py.

Scope is the module name only. The class ControlPlaneServer, the
CONTROL_AUTH_HEADER constant, and the "control plane" architectural term
(CONTROL_PLANE_PORT, host_control_plane_token, control_plane_url, …) are
deliberately unchanged — that's the load-bearing control-plane/data-plane
distinction. Test file renamed to match; full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:57:49 -04:00
didericis b276227cbb refactor(cli): registry in commands/__init__, dispatcher main() in __main__
tracker-policy-pr / check-pr (pull_request) Successful in 8s
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 48s
lint / lint (push) Failing after 2m56s
test / integration-firecracker (pull_request) Successful in 3m33s
test / coverage (pull_request) Successful in 21s
test / publish-infra (pull_request) Has been skipped
Split the cli package's two responsibilities out of __init__:

  * commands/__init__.py now assembles the COMMANDS registry and
    NO_MIGRATION_COMMANDS from the per-command modules — the command
    surface lives entirely under commands/.
  * __main__.py now owns main() (dispatch + migration gate + exit-code
    mapping) alongside the runnable entry guard.

cli/__init__.py shrinks to a shim that re-exports main / COMMANDS /
NO_MIGRATION_COMMANDS, so bot_bottle.cli.main (repo-root cli.py entry)
and the tests' bot_bottle.cli.COMMANDS keep working unchanged. Verified
`python -m bot_bottle.cli` (runpy) and `cli.py` both dispatch; full unit
suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:52:52 -04:00
didericis 3ccd308613 refactor(cli): make help a first-class command; drop usage() from dispatcher
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Successful in 34s
test / unit (pull_request) Successful in 46s
lint / lint (push) Failing after 57s
test / integration-firecracker (pull_request) Successful in 3m25s
test / coverage (pull_request) Successful in 17s
test / publish-infra (pull_request) Has been skipped
Move the top-level usage/command-list text out of the dispatcher into
commands/help.py as cmd_help, registered in COMMANDS so `bb help` now
works as a real command. The dispatcher's -h/--help, no-args, and
unknown-command fallbacks call cmd_help() for the text while keeping
their exit codes (0 for help/-h, 2 for bare no-args, Die for unknown).

Added `help` to NO_MIGRATION_COMMANDS so it prints without a migrated
DB, and listed it in the command summary. Full unit suite green (2243);
dispatch + migration-gate tests pass, `bb help` verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:40:05 -04:00
didericis a21f2358c6 refactor(cli): trim _common to a PROG constants leaf; inline os.getcwd()
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / integration-docker (pull_request) Successful in 21s
lint / lint (push) Failing after 58s
test / unit (pull_request) Successful in 2m11s
test / integration-firecracker (pull_request) Successful in 3m27s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
_common had rotted into a junk drawer. Trim it to what actually
justifies a shared leaf module and rename for legibility:

  * REPO_DIR: deleted (dead — nothing imported it).
  * read_tty_line: dropped the pointless re-export; cleanup/start/init
    now import it straight from bot_bottle.util.
  * USER_CWD: deleted. It captured os.getcwd() at import time, but
    nothing chdirs and no test patched it, so it was equivalent to a
    live os.getcwd() — inlined at the six call sites.
  * PROG: kept, now the sole member. It's still a leaf (both the
    dispatcher and the commands it imports need it) so it can't move
    into __init__ without a circular import.

_common.py -> constants.py. Full unit suite green (2243); CLI dispatch
verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:31:14 -04:00
didericis 50a67c04bd refactor(cli): group subcommand handlers under cli/commands/
test / integration-docker (pull_request) Successful in 18s
tracker-policy-pr / check-pr (pull_request) Successful in 17s
test / unit (pull_request) Failing after 41s
lint / lint (push) Failing after 57s
test / integration-firecracker (pull_request) Successful in 3m26s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
Move the eleven per-command modules (backend, cleanup, commit, edit,
info, init, list, login, resume, start, supervise) into a new
bot_bottle/cli/commands/ package, leaving the dispatcher (__init__),
entrypoint (__main__), and shared helpers (_common, tui) at the cli
root. Makes the command surface obvious at a glance and separates
handlers from the plumbing that registers them.

Updated the dispatcher's COMMANDS imports to .commands.*, bumped the
moved files' relative-import depths (.._common, .. import tui, sibling
.start unchanged), and repointed test references to
bot_bottle.cli.commands.*. Full unit suite green (2243); CLI dispatch
verified via `python -m bot_bottle.cli --help`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:25:25 -04:00
didericis 82d02d2c4b refactor(gateway): move egress entrypoint into gateway/egress
tracker-policy-pr / check-pr (pull_request) Successful in 15s
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 46s
lint / lint (push) Failing after 58s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
egress_entrypoint.sh is the mitmdump launcher for the egress data plane,
so it belongs with the gateway egress service, not at the package root.
Moved to bot_bottle/gateway/egress/entrypoint.sh (prefix stripped now
that the package namespaces it).

Only the Dockerfile.gateway COPY source changes; the runtime target
(/app/egress-entrypoint.sh, referenced by bootstrap's egress DaemonSpec)
is unchanged, and the infra images inherit it via FROM gateway. Updated
the test that resolves the script path and the infra-artifact docstring
example. Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:14:26 -04:00
didericis 74dc984cf8 refactor(gateway): rename bootstrap's _Supervisor to _DaemonManager
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 17s
lint / lint (push) Failing after 53s
test / unit (pull_request) Successful in 2m4s
test / integration-firecracker (pull_request) Successful in 3m32s
test / coverage (pull_request) Successful in 18s
test / publish-infra (pull_request) Has been skipped
The PID-1 process manager in gateway/bootstrap.py was named _Supervisor,
which collided conceptually with the supervise permissions service
(Supervisor / gateway.supervisor). It manages the _DaemonSpec child set,
so _DaemonManager names what it does without the "supervise" echo.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:08:34 -04:00
didericis ce744a85c4 refactor(gateway): split data-plane files into egress/supervisor/git_gate services
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 49s
lint / lint (push) Failing after 2m49s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 18s
test / publish-infra (pull_request) Has been skipped
Group the gateway's data-plane modules into three service sub-packages
mirroring the host-side trio (bot_bottle.egress / .supervisor / .git_gate):

  gateway/egress/     addon_core, addon, dlp_config, dlp_detectors
  gateway/supervisor/ server            (was supervise_server)
  gateway/git_gate/   render, http_backend

Prefix-stripped filenames now that the package namespaces them; each
sub-package has a thin docstring __init__ (no eager imports, cheap leaf
loads). The two cross-cutting files stay at the gateway root:
policy_resolver (shared per-client lookup) and gateway_init, renamed to
bootstrap now that gateway/ already namespaces it.

Updated all importers (bot_bottle + tests), the in-VM/container `-m`
launch strings, the Dockerfile.gateway addon shim + ENTRYPOINT, and the
five gateway entries in scripts/critical-modules.txt. Full unit suite
green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 17:00:51 -04:00
didericis a446551acb refactor(egress): make Egress a service class in an egress package
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 46s
lint / lint (push) Failing after 2m44s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 20s
test / publish-infra (pull_request) Has been skipped
Split the flat egress.py into an egress/ package: neutral DTOs
(EgressRoute, EgressPlan) in plan.py, and the concrete Egress service
class plus rendering/env helpers in service.py, behind a thin
__getattr__ facade so leaf imports stay cheap.

Egress drops ABC and becomes a concrete service the backends call:
resolve_token_values / agent_env_entries / prepare are now methods.
Backend launch paths (docker, firecracker, macos_container) call
Egress().<method>(...) instead of the module-level functions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 16:43:59 -04:00
didericis 450037b7e9 refactor(git-gate): make GitGate a service class in a git_gate package
test / integration-docker (pull_request) Successful in 15s
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / unit (pull_request) Successful in 47s
lint / lint (push) Failing after 58s
test / integration-firecracker (pull_request) Successful in 3m17s
test / coverage (pull_request) Successful in 20s
test / publish-infra (pull_request) Has been skipped
Turn the git_gate module into a package with GitGate as a concrete service class
(dropping the ABC), mirroring the Supervisor shape. The host-side git-gate
operations are now methods the backend drives:

  git_gate/
    __init__.py  — thin __getattr__ facade (keeps `from bot_bottle.git_gate
                   import …` working; render/hook names lazily forwarded to
                   gateway.git_gate_render)
    plan.py      — GitGatePlan (the launch DTO the backend contract references)
    service.py   — GitGate: prepare / provision_dynamic_keys /
                   revoke_provisioned_keys / preflight_host_keys
    provision.py — deploy-key lifecycle (was git_gate_provision.py)
    host_key.py  — host-key preflight (was git_gate_host_key.py)

The backend launch + base.py preflight now call GitGate() methods instead of the
free functions. The runtime rendering / in-gateway hook execution stay in
gateway/git_gate_render.py (data plane) — only the host-side service moved.

Because the facade forwards names lazily, the ~25 `from bot_bottle.git_gate
import GitGatePlan` call-sites are unchanged, and importing git_gate.plan (the
contract's dependency) no longer drags in the provisioning / forge-API code.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 16:27:56 -04:00
didericis 14c28946a7 refactor(manifest): move Manifest/ManifestIndex to manifest.index; thin the package init
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Failing after 1m3s
test / unit (pull_request) Successful in 2m6s
test / integration-firecracker (pull_request) Successful in 3m26s
test / coverage (pull_request) Successful in 21s
test / publish-infra (pull_request) Has been skipped
manifest/__init__.py was a facade that eagerly imported every piece (agent,
bottle, egress, git, loader, schema, util) and defined the aggregate Manifest /
ManifestIndex, so importing a leaf like `manifest.util` paid 19 modules.

Move the aggregate model — Manifest, ManifestIndex, and the bottle-resolution
helpers — into manifest/index.py (which keeps the framework imports it needs),
and make __init__ a thin __getattr__ facade over the 12 __all__ names (mapping
each to its submodule), with a TYPE_CHECKING block. The package docstring is
preserved.

`import bot_bottle.manifest.util` drops 19 -> 3; the package itself is 2. All
`from bot_bottle.manifest import Manifest/ManifestIndex/ManifestError/…`
call-sites keep working via the lazy facade; no importer changes needed.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 16:04:15 -04:00
didericis eab9d15130 refactor(docker): fold run_docker into backend/docker/util; drop the docker_cmd shim
tracker-policy-pr / check-pr (pull_request) Successful in 8s
test / integration-docker (pull_request) Successful in 13s
test / unit (pull_request) Successful in 45s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m17s
test / coverage (pull_request) Successful in 35s
test / publish-infra (pull_request) Has been skipped
docker_cmd.py existed as a top-level module solely so the orchestrator's docker
components could share run_docker without dragging the (then-heavy) backend
layer in. Now that backend/__init__ and backend/docker/__init__ are thin,
importing backend.docker.util costs 6 modules instead of 76, so the shim's whole
reason to exist is gone.

Move run_docker into backend/docker/util.py (where the other docker subprocess
primitives live) and delete docker_cmd.py. Backend siblings import it via
`from .util import run_docker`; the two docker-specific orchestrator modules
(docker_broker, rotate_ca) import `from ..backend.docker.util import run_docker`.
No import cycle (backend.docker.util pulls nothing from orchestrator);
orchestrator.__main__ stays lean at 28 modules. run_docker patch targets are
unchanged (tests patch it in the importing module).

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:58:23 -04:00
didericis 923d44bc09 refactor(backend): thin the docker/firecracker/macos-container package inits
tracker-policy-pr / check-pr (pull_request) Successful in 15s
test / integration-docker (pull_request) Successful in 20s
lint / lint (push) Successful in 53s
test / unit (pull_request) Failing after 2m5s
test / integration-firecracker (pull_request) Successful in 3m27s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
The three concrete-backend package inits eagerly imported their
`*BottleBackend` (which pulls the whole framework), so importing any leaf under
them (e.g. `backend.docker.util`) paid ~78 modules. Give each the same thin
`__getattr__` treatment as the top-level backend package: re-export the public
class(es) lazily on first access, with a TYPE_CHECKING block for checkers.

`import bot_bottle.backend.docker.util` drops from 78 -> 6 bot_bottle modules;
the docker/firecracker/macos-container packages themselves are ~3 each. All
`from bot_bottle.backend.docker import DockerBottleBackend` call-sites and the
lazy selection path keep working.

This is what makes backend leaves cheap — and unblocks moving docker_cmd into
backend/docker/util if we want.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:53:48 -04:00
didericis 3e62f31d8b refactor(backend): split the heavy backend/__init__ into base + selection, thin the package init
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 38s
test / unit (pull_request) Successful in 52s
lint / lint (push) Successful in 1m1s
test / integration-firecracker (pull_request) Successful in 3m33s
test / coverage (pull_request) Successful in 17s
test / publish-infra (pull_request) Has been skipped
backend/__init__.py eagerly imported the whole framework (manifest, egress,
git-gate, env, workspace, agent-provider), so importing ANY backend.* submodule
paid ~32 modules just to run the package init. Split it:

  - backend/base.py     — the abstract contract (BottleSpec, BottlePlan,
                          BottleCleanupPlan, ExecResult, ActiveAgent, Bottle,
                          BottleImages, BottleBackend). Carries the framework
                          imports; loaded only when a caller needs the contract.
  - backend/selection.py — backend registry / selection / enumeration
                          (get_bottle_backend, _get_backends, _auto_select_backend,
                          has_backend, known_backend_names, enumerate_active_agents).
                          Concrete backends still imported lazily inside.
  - backend/__init__.py  — thin: a __getattr__ that resolves the public names
                          from those submodules on first access (+ a TYPE_CHECKING
                          block for checkers). Existing `from bot_bottle.backend
                          import X` and patch.object call-sites keep working.

`import bot_bottle.backend` drops from 32 -> 2 bot_bottle modules. Repointed the
backend-selection test's patch targets to the `selection` module (the functions
reference each other there now; the FirecrackerBottleBackend class-method
patches are unchanged — same class object).

Note: backend.docker.util is still heavy because backend/docker/__init__ eagerly
imports DockerBottleBackend — slimming the three sub-package inits is the
follow-up that makes the leaves cheap.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:47:29 -04:00
didericis b96a8b44e0 refactor(docker): rename OrchestratorService -> DockerInfraService, move to backend/docker
test / integration-docker (pull_request) Successful in 22s
lint / lint (push) Successful in 1m7s
test / unit (pull_request) Successful in 2m10s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 15s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 13m2s
OrchestratorService wasn't the orchestrator — it's the host-side lifecycle of
the docker infra *container* (the one that runs the Orchestrator). It read as
"the orchestrator as a service" and lived in orchestrator/lifecycle.py, while
its siblings (macOS MacosInfraService, Firecracker infra_vm) live under their
backend package. Rename it DockerInfraService and move it to
backend/docker/infra.py alongside the docker backend, with its docker-only
constants (INFRA_*/ORCHESTRATOR_* image + container names, daemon list, mount
paths).

orchestrator/lifecycle.py keeps only the backend-neutral pieces the other infra
services share — DEFAULT_PORT, DEFAULT_STARTUP_TIMEOUT_SECONDS,
OrchestratorStartError, source_hash — which macOS / firecracker / client still
import from there. backend/docker/infra.py imports those (backend -> orchestrator
is an allowed direction). Renamed the unit test to test_docker_infra.py.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:26:52 -04:00
didericis d41236c376 refactor(util): make render_diff generic (caller supplies side titles); move to root util
tracker-policy-pr / check-pr (pull_request) Successful in 8s
test / integration-docker (pull_request) Successful in 36s
test / unit (pull_request) Successful in 43s
lint / lint (push) Successful in 56s
test / integration-firecracker (pull_request) Successful in 3m27s
test / coverage (pull_request) Successful in 21s
test / publish-infra (pull_request) Has been skipped
Generalize render_diff so it isn't supervise-flavored: the "(current)" /
"(proposed)" side titles are no longer baked in — callers pass `before_title`
and `after_label` (required). Move it from orchestrator/supervisor/util.py to
the base bot_bottle.util alongside sha256_hex, and delete the now-empty
supervisor/util.py.

The supervise callsite (Orchestrator.supervise_respond) assigns
before_title="current", after_label="proposed"; the facade no longer re-exports
render_diff (callers import from bot_bottle.util). Behavior at the supervise
callsite is unchanged.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:09:27 -04:00
didericis db6a151803 refactor(supervise): hash the proposed file inside Proposal.new; sha256_hex → root util
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / integration-docker (pull_request) Successful in 39s
test / unit (pull_request) Successful in 53s
lint / lint (push) Failing after 1m3s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
Move sha256_hex out of orchestrator/supervisor/util.py to the root
bot_bottle.util (it's a generic string hash, not supervise-specific), and have
Proposal.new take the proposed file contents and compute current_file_hash
itself instead of every caller passing sha256_hex(proposed_file).

Every call site set current_file_hash to the hash of the proposed file, and
nothing reads the field except the DB round-trip (schema + _row_to_proposal +
from_dict, all untouched), so folding the hash into the factory is
behavior-preserving and removes the boilerplate. Callers now pass just the
file; the orchestrator no longer imports sha256_hex at all.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 15:03:37 -04:00
didericis 8abccf7ffe refactor(supervise): make Supervisor a service class the orchestrator calls
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Failing after 55s
test / unit (pull_request) Successful in 2m8s
test / integration-firecracker (pull_request) Successful in 3m25s
test / coverage (pull_request) Successful in 39s
test / publish-infra (pull_request) Has been skipped
Turn the loose queue/audit functions in orchestrator/supervisor/queue.py into
methods on a concrete Supervisor service. The Orchestrator now owns an
injectable `self._supervisor` and calls `self._supervisor.write_proposal(...)`
etc., instead of module-level free functions — the supervise dependency is
explicit and mockable, and a Supervisor can be scoped to a `db_path` (defaults
to the host DB) so tests can point it at a temp database.

  - Supervisor (in the package __init__) drops the ABC and gains write_proposal,
    read_proposal, list_pending_proposals, list_all_pending_proposals,
    write_response, read_response, archive_all_proposals, write_audit_entry,
    read_audit_entries, plus the launch-time prepare().
  - render_diff / sha256_hex are pure and stateless, so they move to
    orchestrator/supervisor/util.py (re-exported from the facade) rather than
    becoming methods.
  - queue.py is deleted; service.py + the supervise tests call through a
    Supervisor instance.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 14:52:17 -04:00
didericis 27a122e24b refactor(supervise): drop obsolete queue helpers (archive_proposal, wait_for_response, audit_dir/path)
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Successful in 17s
lint / lint (push) Failing after 59s
test / unit (pull_request) Successful in 2m3s
test / integration-firecracker (pull_request) Successful in 3m28s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
Remove the supervise queue helpers that no live path uses anymore — they're
mechanisms the PRD-0070 / idempotent-poll migration superseded:

  - archive_proposal (+ QueueStore.archive_proposal): the poll stopped archiving
    on read (#469 review), so single-proposal archiving has no caller; decided
    proposals drop off the pending list via their response row and are reaped in
    bulk by archive_all_proposals on teardown/reconcile.
  - wait_for_response: the data plane polls the queue over the control-plane RPC
    now, so nothing block-waits on it.
  - audit_dir / audit_log_path: file-based audit paths, replaced by the SQLite
    AuditStore.

Also drops the tests that covered those obsolete helpers. Kept the prod-unused
read accessors list_pending_proposals and read_audit_entries: those aren't dead
mechanisms — they're the read side of live write paths that many tests use to
verify real queue/audit behavior.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 14:36:32 -04:00
didericis 3e2cbcab88 refactor(supervise): fold Supervise into the facade as Supervisor; drop dead flat-fallbacks
tracker-policy-pr / check-pr (pull_request) Successful in 8s
test / integration-docker (pull_request) Successful in 12s
test / unit (pull_request) Successful in 43s
lint / lint (push) Failing after 59s
test / integration-firecracker (pull_request) Successful in 3m16s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
Move the `Supervise` lifecycle out of its own `orchestrator/supervisor/
supervise.py` and into the package `__init__`, renaming the class to
`Supervisor`. Callers now import it from `bot_bottle.orchestrator.supervisor`
alongside the queue surface it belongs with.

Remove the dead `try/except ImportError` flat-import fallbacks from the
package-only store modules (db_store, audit_store, config_store, queue_store)
and image_cache. Those fallbacks existed for when the store files were
flat-copied into the gateway; post-PRD-0070 the data plane never opens the DB,
so these modules are only ever imported as part of the package. The two gateway
data-plane files that may still be loaded flat (egress_addon_core,
git_gate_render) keep their fallbacks.

Full unit suite green (2251).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 14:24:06 -04:00
didericis 44e2b5a897 refactor(supervise): split the supervise plane by tier; per-service store managers
test / integration-docker (pull_request) Successful in 13s
tracker-policy-pr / check-pr (pull_request) Successful in 14s
test / unit (pull_request) Successful in 42s
lint / lint (push) Failing after 54s
test / integration-firecracker (pull_request) Successful in 3m17s
test / coverage (pull_request) Successful in 17s
test / publish-infra (pull_request) Has been skipped
Give each service its own store package + manager, and cut the supervise module
along the control/data-plane boundary so nothing in the shared layer reaches up
into the orchestrator.

Stores, by owner:
  - bot_bottle/store/ keeps only the shared base (DbStore, migrations) and the
    concrete stores that aren't service-owned (audit_store, config_store).
  - bot_bottle/orchestrator/store/ now houses the orchestrator-owned stores —
    queue_store (supervise queue), secret_store, config_store — plus a new
    orchestrator store_manager that migrates them (composing audit/config
    downward from the base). The old shared store_manager is gone.

Supervise plane, by tier:
  - bot_bottle/supervisor/ (NEUTRAL, importable by every tier including the
    gateway): types.py (the Proposal/Response/AuditEntry wire types + the tool/
    status/poll constants + the shared daemon constants moved out of
    supervise.py) and plan.py (SupervisePlan, a pure DTO).
  - bot_bottle/orchestrator/supervisor/ (orchestrator-only): queue.py (the
    queue/audit I/O wrappers + render_diff + sha256_hex) and supervise.py (the
    Supervise lifecycle that stages the DB via the store manager). Its __init__
    re-exports the neutral vocabulary so orchestrator-side callers import from
    one place.

The gateway now imports only bot_bottle.supervisor.types (never
bot_bottle.supervise), so the data plane holds no code dependency on the
orchestrator — it reaches the queue over the control-plane RPC. This removes
the circular import that moving queue_store under orchestrator introduced
(supervise -> orchestrator -> service -> supervise).

supervise_types.py -> supervisor/types.py; supervise.py deleted (split). Full
unit suite green (2251).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 14:15:04 -04:00
didericis f77023db1d refactor(gateway): move the data-plane daemons into a bot_bottle.gateway package
test / integration-docker (pull_request) Successful in 11s
test / unit (pull_request) Successful in 43s
lint / lint (push) Successful in 56s
test / integration-firecracker (pull_request) Successful in 3m19s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 7s
Separate the gateway (data plane) from the orchestrator (control plane) at the
module level. The gateway runtime files move out of the package root — and the
backend-neutral Gateway lifecycle ABC + GATEWAY_* constants move out of
orchestrator/ — into a new bot_bottle/gateway/ package:

  gateway/__init__.py      (was orchestrator/gateway.py: Gateway ABC + consts
                            + rotate_gateway_ca)
  gateway/gateway_init.py  (the PID-1 daemon supervisor)
  gateway/egress_addon.py, egress_addon_core.py, egress_dlp_config.py,
          dlp_detectors.py            (the egress mitmproxy daemon)
  gateway/git_http_backend.py         (the git-http daemon)
  gateway/git_gate_render.py          (the git-gate pre-receive rendering)
  gateway/supervise_server.py         (the supervise MCP daemon)
  gateway/policy_resolver.py          (the data-plane control-plane RPC client)

orchestrator/ now holds only control-plane files. The shared plan/types/auth
layer (egress.py=EgressPlan, git_gate.py=GitGatePlan, supervise.py,
supervise_types.py, control_auth.py) and the launch-time git-gate provisioning
helpers stay at root, so orchestrator/ and backend/ still own them.

Because these daemons are invoked as `python3 -m bot_bottle.<name>`, loaded flat
by mitmproxy, and referenced in Dockerfile.gateway, the move updates more than
Python imports: the `-m` invocations (firecracker/macOS infra scripts), the
Dockerfile.gateway addon shim + ENTRYPOINT, gateway_init's _DAEMONS module
paths, and the git-gate CGI heredocs all now point at bot_bottle.gateway.*.

No behavior change; full unit suite green (2251).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 13:25:10 -04:00
didericis a845cba925 refactor(manifest): consolidate the manifest modules into a bot_bottle.manifest package
tracker-policy-pr / check-pr (pull_request) Failing after 11s
test / integration-docker (pull_request) Successful in 20s
lint / lint (push) Successful in 59s
test / unit (pull_request) Successful in 2m10s
test / integration-firecracker (pull_request) Successful in 3m29s
test / coverage (pull_request) Successful in 38s
test / publish-infra (pull_request) Has been skipped
Move the nine root-level manifest modules into a bot_bottle/manifest/ package,
dropping the redundant manifest_ prefix: the facade (manifest.py) becomes the
package __init__ and keeps re-exporting Manifest / ManifestIndex and the piece
types, while manifest_<x>.py become manifest/<x>.py (agent, bottle, egress,
extends, git, loader, schema, util).

Because the facade stays the package __init__, the ~13 callers that import
`from bot_bottle.manifest import ...` are unchanged; only direct-piece imports
move to `bot_bottle.manifest.<name>`. Inside the package, intra-manifest
imports drop the prefix and the non-manifest siblings (log, yaml_subset,
agent_provider) move to `..`.

No behavior change; full unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 13:04:39 -04:00
didericis d7a58e52fd refactor(store): consolidate the SQLite store family into bot_bottle.store
lint / lint (push) Successful in 54s
test / integration-docker (pull_request) Successful in 38s
test / unit (pull_request) Successful in 45s
test / integration-firecracker (pull_request) Successful in 3m31s
test / coverage (pull_request) Successful in 18s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 6s
Move the DbStore base (db_store), the shared schema-migration base (migrations /
TableMigrations), the concrete stores (audit_store, config_store, queue_store),
and StoreManager out of the package root into a bot_bottle/store/ package, so
persistence lives in one place rather than scattered across the root.

Callers use direct submodule imports (from bot_bottle.store.store_manager
import StoreManager). Inside the moved modules the relative imports point back
up to their non-store siblings (..paths, ..supervise_types) while co-moved
siblings stay package-local; the flat-import fallbacks are untouched. The
orchestrator's own stores (config_store, secret_store, registry) stay in
orchestrator/ and repoint to the moved DbStore / TableMigrations.

No behavior change; full unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 12:53:52 -04:00
didericis dfce3d9505 refactor(gateway): move DockerGateway to the backend layer, drop the dead standalone-gateway path
lint / lint (push) Successful in 54s
The `DockerGateway` container-lifecycle impl now lives in
`backend/docker/gateway.py` (the shape backend gateway classes will share);
`orchestrator/gateway.py` keeps only the backend-neutral pieces (the `Gateway`
ABC, constants, `rotate_gateway_ca`).

Delete the standalone-gateway path it was the only consumer of. `--gateway` on
`python -m bot_bottle.orchestrator` was invoked nowhere — the production docker
flow runs the gateway data plane inside the combined `bot-bottle-infra`
container via `OrchestratorService`, never this class. Removing it takes with it
`Orchestrator.ensure_gateway()` and the `gateway` ctor arg; `gateway_status()`
becomes a stub reporting `configured: false` so the documented `GET /gateway`
control-plane route keeps its contract.

Fix a latent bug the move surfaced: `launch.py` read the shared gateway CA via
`docker exec bot-bottle-orch-gateway`, a container the consolidated flow never
creates — so the agent CA install was reaching a nonexistent name. Read it from
`bot-bottle-infra` (INFRA_NAME) instead.

Repoint the affected tests to the new module; drop the unit test + fake that
covered the deleted standalone wiring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 12:39:58 -04:00
didericis-claude aac27d8a40 fix(firecracker): provision role-scoped control-plane auth in the infra VM
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / integration-docker (pull_request) Successful in 18s
test / unit (pull_request) Failing after 45s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m29s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
The Firecracker infra VM started the control plane with no signing key, so it
ran open and granted every unauthenticated caller the `cli` role. The nft
boundary only fences off the separate agent VM — the egress / supervise /
git-http daemons run in this same VM and reach the orchestrator over 127.0.0.1,
so a compromised data-plane daemon could still drive the operator routes
(approve its own supervise proposals, rewrite policy, read injected tokens).

Generate the signing key on the persistent volume, hand it only to the
orchestrator process, mint a `gateway` JWT for the data-plane daemons, and
mirror the key back to the host token file so the CLI signs `cli` tokens the VM
verifies — the same per-process scoping the docker / macOS launchers use.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 07:03:05 +00:00
didericis-claude d8b61b3658 fix(supervise): make the response poll idempotent and slow it to 250ms
tracker-policy-pr / check-pr (pull_request) Successful in 15s
test / integration-docker (pull_request) Successful in 42s
test / unit (pull_request) Successful in 46s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m22s
test / coverage (pull_request) Successful in 23s
test / publish-infra (pull_request) Has been skipped
The control-plane supervise poll archived the proposal on the same call
that returned the decision, so a dropped connection lost the operator's
decision (the retry saw `unknown`). Poll no longer archives — a re-poll
returns the same decision. Decided proposals still drop off the operator's
pending list (a response row exists); their rows are reaped when the
bottle is torn down or reconciled.

Also raise the grace-window poll interval from 50ms to 250ms now that each
poll is an HTTP round trip rather than a local file read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 06:26:06 +00:00
didericis-claude 1db2a9eb67 feat(control-plane): role-scoped signed tokens so the gateway can't drive operator routes
test / integration-docker (pull_request) Successful in 17s
tracker-policy-pr / check-pr (pull_request) Successful in 16s
test / unit (pull_request) Successful in 39s
lint / lint (push) Successful in 56s
test / integration-firecracker (pull_request) Successful in 3m21s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
Review follow-up on #469: the data plane held the same control-plane secret
that authorizes every route, so a compromised egress/git-gate could queue a
supervise proposal AND approve it (or rewrite policy, read injected tokens) —
the (source_ip, identity_token) checks attribute the *bottle*, not the caller.

Replace the single shared bearer secret with role-scoped, HMAC-signed tokens
(compact HS256 JWTs, stdlib-only — no new dependency):

  * new `control_auth` mints/verifies `{role}` tokens; roles are `gateway`
    (data plane) and `cli` (host operator/launcher).
  * the orchestrator holds only the signing *key* and verifies; `dispatch`
    gates each route by role — `gateway` reaches /resolve + /supervise/
    {propose,poll}, everything else is `cli`-only (401 unauthenticated,
    403 wrong role).
  * the gateway is handed a pre-minted `gateway` token it cannot rewrite into
    `cli`; the host CLI mints its own `cli` token from the host key.
  * `gateway_init` scopes the signing key to the orchestrator process and the
    gateway token to the data-plane daemons, so even in the combined infra
    container a compromised data-plane daemon never sees the key.

Launchers (docker gateway + infra, macOS infra) inject the minted token(s);
Firecracker stays open behind its nft boundary. Open mode (no key) still grants
full `cli` access — the fail-visible fallback for tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 05:22:56 +00:00
didericis-claude 1972c8c6e9 docs: add glossary of canonical bot-bottle terminology
lint / lint (push) Successful in 56s
tracker-policy-pr / check-pr (pull_request) Successful in 15s
Defines Agent Provider, Agent Runtime, Agent/Agent Definition,
Bottle/Bottle Definition, Sealed Bottle, Bottled Agent, and Active Bottle.
Links from docs/README.md and AGENTS.md for discoverability.

Closes #474

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 00:12:32 -04:00
didericis-claude 5d109ea290 CLI cleanup: remove info, rename list subcommands (#472)
test / unit (push) Successful in 44s
test / integration-docker (push) Successful in 47s
lint / lint (push) Successful in 54s
Update Quality Badges / update-badges (push) Successful in 55s
test / integration-firecracker (push) Successful in 4m47s
test / coverage (push) Successful in 15s
test / publish-infra (push) Successful in 1m49s
- Remove `info` command
- Rename `list active` → `active` (new top-level command)
- Rename `list available` → `list` (no subcommand argument)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 00:02:19 -04:00
didericis-claude 72fdb1d14b test: annotations + coverage for the supervise RPC seam
tracker-policy-pr / check-pr (pull_request) Successful in 6s
test / integration-docker (pull_request) Successful in 15s
test / unit (pull_request) Successful in 39s
test / integration-firecracker (pull_request) Successful in 3m20s
test / coverage (pull_request) Successful in 44s
test / publish-infra (pull_request) Has been skipped
Add pyright-strict parameter/return annotations to the fake resolvers and
test helpers, and cover the new control-plane validation/403 branches
(/supervise/propose + /supervise/poll) plus the supervise-server and egress
poll-error fail-closed paths, so the diff-coverage gate stays above 90%.
No production behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 03:24:30 +00:00
didericis-claude 2c496dc3d0 fix(supervise): reach the queue over RPC, get bot-bottle.db off the data plane
PRD 0070's rule — only the orchestrator opens bot-bottle.db; the data
plane reaches state through the control-plane RPC — was not in force.
Three data-plane daemons held a direct read-write handle on the shared
SQLite file: the supervise MCP server, the egress DLP addon (the most
attack-exposed process, TLS-bumping hostile traffic), and the git-gate
pre-receive hook. An RCE in any of them could read every bottle's
plaintext identity_token and forge attribution fleet-wide (issue #469).

Add the agent half of the supervise flow to the control plane:

  POST /supervise/propose  -> queue a proposal, 201 {proposal_id}
  POST /supervise/poll     -> non-blocking decision poll, 200 {status,...}

Both attribute the caller by (source_ip, identity_token) exactly like
/resolve — never a caller-supplied slug — so a bottle can only ever queue
or read its own proposals even if the data plane is compromised. A decided
poll archives server-side, preserving the archive-after-read contract.

Data plane: the supervise server, egress addon, and git-gate hook now
queue/poll through PolicyResolver.propose_supervise / poll_supervise
instead of opening the DB. supervise_server keeps its ~30s grace window
by polling the RPC; egress keeps its safelist keyed by resolved bottle;
the git-gate hook gets (source_ip, identity_token) from the CGI env.

Packaging: drop the DB bind-mount and SUPERVISE_DB_PATH from the
data-plane containers/VMs (docker gateway + infra, macOS infra,
firecracker infra). The orchestrator remains the sole opener of the one
file via BOT_BOTTLE_ROOT / host_db_path().

Update PRD 0070: the rule is now in force; remove the transitional caveat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 02:49:51 +00:00
didericis-codex f24ae45d13 test(gateway): cover single and simultaneous daemon crashes
test / integration-docker (push) Successful in 21s
lint / lint (push) Successful in 1m4s
test / unit (push) Successful in 1m44s
test / integration-firecracker (push) Successful in 4m55s
test / coverage (push) Successful in 21s
test / publish-infra (push) Successful in 1m50s
Update Quality Badges / update-badges (push) Failing after 14m42s
2026-07-23 20:15:18 -04:00
didericis-codex 594d07410a fix(gateway): restart dead children before completion check 2026-07-23 20:15:18 -04:00
didericis-claude c9c62f256d fix(egress): restart dead daemons and cap inbound scan body size (#455)
Two complementary fixes for the egress gateway OOM:

1. gateway_init: auto-restart any daemon that dies unexpectedly. The
   supervisor already had restart_daemon()/request_restart() logic; this
   wires it into tick() so an OOM-killed mitmdump is respawned without
   operator intervention. Implements the "eventual" failure policy
   described in the original module docstring.

2. egress_addon: cap response body bytes passed to the DLP inbound scan
   at EGRESS_INBOUND_SCAN_LIMIT_BYTES (default 1 MiB). mitmproxy buffers
   the full response before the hook fires; capping at scan time limits
   the additional amplification from decoded text and regex strings. Bodies
   over the limit emit an egress_scan_truncated log event. Set to 0 to
   disable the cap.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 20:15:18 -04:00
didericis 10150ae9f5 docs(paths): correct root docstring after SQLite queue migration
lint / lint (push) Successful in 2m37s
test / unit (push) Successful in 1m42s
test / integration-firecracker (push) Successful in 4m43s
test / integration-docker (push) Successful in 33s
test / coverage (push) Successful in 19s
Update Quality Badges / update-badges (push) Successful in 1m45s
test / publish-infra (push) Successful in 2m1s
The module docstring still listed "queue" and "audit logs" as things
living under the app data root, which read as directories. Both have
been tables in the shared SQLite DB since PRD 0067; the legacy `queue/`
directory was unplumbed in 29904609 and nothing has written there since.

Lists what the root actually holds and points at the stores that own the
queue/audit rows, so the next reader doesn't go looking for a directory
that isn't there.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 19:40:20 -04:00
Quality Badge Bot 86c7ac1843 chore: update quality badges
- Coverage: 84%
- Core coverage: 94%

[skip ci]
2026-07-23 23:29:52 +00:00
didericis-claude 2e0414f969 fix(test): restore headless unit tests broken by PTY check in e719022
test / integration-docker (push) Successful in 15s
test / unit (push) Successful in 39s
Update Quality Badges / update-badges (push) Successful in 45s
lint / lint (push) Successful in 2m36s
test / integration-firecracker (push) Successful in 5m15s
test / coverage (push) Successful in 16s
test / publish-infra (push) Successful in 1m39s
The PTY check added in e719022 calls sys.stdin.fileno() which raises
io.UnsupportedOperation under pytest's stdin capture. Guard the fileno()
call with a try/except so the check degrades cleanly to "not a tty"
instead of crashing, then stub os.isatty in the test setUp so headless
unit tests aren't blocked on a real TTY.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 23:28:55 +00:00
didericis-claude 12b071833d test: annotate _git_repo helper to satisfy pyright strict mode
test / integration-docker (push) Successful in 13s
Update Quality Badges / update-badges (push) Successful in 40s
lint / lint (push) Successful in 53s
test / integration-firecracker (push) Successful in 4m56s
test / unit (push) Failing after 33s
test / coverage (push) Has been skipped
test / publish-infra (push) Has been skipped
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 18:32:59 -04:00
didericis-claude bf72282f8e fix(manifest): avoid KeyError when override bottle has repos absent from base
merge_bottles_runtime used .get(n, base_repos_by_name[n]) which eagerly
evaluates the default, crashing when a repo exists only in the override
bottle. Replaced with a conditional expression so the base lookup only
runs when needed.

Adds three regression tests covering override-only, base-only, and
name-collision cases.

Closes #457

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 18:32:59 -04:00
Quality Badge Bot f2c3710d0d chore: update quality badges
- Coverage: 83%
- Core coverage: 94%

[skip ci]
2026-07-23 22:13:28 +00:00
didericis-claude e719022698 fix(cli): preflight PTY check in --headless before resource allocation
test / integration-docker (push) Successful in 19s
lint / lint (push) Successful in 52s
Update Quality Badges / update-badges (push) Successful in 48s
test / unit (push) Failing after 1m45s
test / integration-firecracker (push) Successful in 4m55s
test / coverage (push) Has been skipped
test / publish-infra (push) Has been skipped
Fails fast with a clear message and workaround hint instead of the
opaque ENODEV deep in the container exec layer after images are built,
the gateway is started, and deploy keys are provisioned.

Closes #460

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 22:12:31 +00:00
Quality Badge Bot 0fb9b04c01 chore: update quality badges
- Coverage: 84%
- Core coverage: 94%

[skip ci]
2026-07-23 21:49:28 +00:00
didericis-codex 26d0f5e3b2 refactor(secrets): keep ciphertext format minimal
tracker-policy-pr / check-pr (pull_request) Successful in 15s
test / integration-docker (pull_request) Successful in 35s
test / unit (pull_request) Successful in 46s
test / integration-firecracker (pull_request) Successful in 4m17s
test / coverage (pull_request) Successful in 16s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 14s
prd-number / assign-numbers (push) Failing after 31s
test / unit (push) Successful in 41s
Update Quality Badges / update-badges (push) Successful in 44s
lint / lint (push) Successful in 1m0s
test / integration-firecracker (push) Successful in 4m59s
test / coverage (push) Successful in 24s
test / publish-infra (push) Successful in 1m49s
2026-07-23 21:39:07 +00:00
didericis-codex bc4e559775 test(secrets): satisfy static coverage checks 2026-07-23 21:39:07 +00:00
didericis-codex 854f6b5696 fix(secrets): recover encrypted tokens on all backends 2026-07-23 21:39:07 +00:00
didericis-claude 28953bfe0b fix(types): remove unused new_env_var_secret import
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 21:38:33 +00:00
didericis 1ffc553ade test(secrets): cover per-bottle egress secret encryption
Add unit coverage for the encrypted at-rest egress secrets: the
secret_store round-trip (new_env_var_secret/encrypt_value/decrypt_value)
and the registry/service wiring that injects ENV_VAR_SECRET.

Recovered from the bot-bottle-claude-agent-1 VM after the CI runner's
disk filled and forced its rootfs read-only; committed work was already
on origin, these were the agent's uncommitted changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-23 21:38:33 +00:00
didericis-claude 9014c07b86 feat(secrets): encrypt egress tokens at rest with per-bottle ENV_VAR_SECRET
Implements the interim secret-provider design (PRD prd-new-secret-provider):
each agent receives a random ENV_VAR_SECRET injected into its container env
at launch. The host uses this key to encrypt each egress auth token value
(HMAC-SHA256 CTR mode, stdlib-only) and store it in a new
bottled_agent_secrets table (one row per env var, key column plaintext for
auditing). The key never touches the DB.

On infra container restart the in-memory token map is lost. launch_consolidated
now calls _reprovision_running_bottles after ensure_running: for each
registered bottle still alive on the gateway network it execs
`printenv ENV_VAR_SECRET` into the agent container and posts the result to the
new POST /bottles/<id>/reprovision_gateway control-plane endpoint, which
decrypts the stored rows and restores _tokens — no manual intervention needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-23 21:38:33 +00:00
didericis 8e2465e241 docs(prd): draft encrypted at-rest egress secrets (SecretProvider interim)
The orchestrator holds each bottle's egress auth tokens in process memory
only, so recreating the infra container strips every already-running
bottle of its upstream credentials. The registry row and the gateway CA
both survive; the tokens do not, so /resolve serves an intact policy with
an empty token map and the addon fails closed on `token_env unset`.

Drafts the interim slice of #355: persist the tokens encrypted so they
survive a restart, without regressing to plaintext at rest and without
foreclosing the per-request minting end state. Design section is left for
the author to fill in.
2026-07-23 21:38:33 +00:00
didericis-codex a8043be394 fix(types): annotate inspect settings fixture
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 18s
test / unit (pull_request) Successful in 1m41s
test / integration-firecracker (pull_request) Successful in 3m27s
test / coverage (pull_request) Successful in 22s
test / publish-infra (pull_request) Has been skipped
test / integration-docker (push) Successful in 34s
test / unit (push) Successful in 45s
Update Quality Badges / update-badges (push) Successful in 45s
lint / lint (push) Successful in 53s
test / integration-firecracker (push) Successful in 5m5s
test / coverage (push) Successful in 42s
test / publish-infra (push) Successful in 1m44s
2026-07-23 21:28:39 +00:00
didericis-codex 7d401a68c5 refactor(egress): make TLS inspection explicit 2026-07-23 21:28:39 +00:00
didericis-claude ce7a7c9915 refactor: resolve once per HTTPS connection, not per inner request
http_connect now stashes the resolved (config, slug, env) under
_FLOW_CTX_KEY after resolving, so request() can reuse it without
a second orchestrator round-trip. Plain-HTTP flows (no prior CONNECT
stash) still resolve in request() as before.
2026-07-23 21:28:22 +00:00
didericis-claude 83aa6768fc feat: add dlp: false passthrough option for egress routes
Routes with `dlp: false` skip all DLP scanning (including CRLF injection,
which cannot be disabled via `dlp.outbound_detectors: false`) and tunnel
HTTPS connections without TLS interception so the client sees the server's
real certificate. Fixes Docker image pulls, which fail when the proxy MITM's
the TLS handshake and the container doesn't trust the per-bottle CA.

Closes #462
2026-07-23 21:28:22 +00:00
didericis-codex 6fea44067f fix(manifest): preserve declared boolean fields
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 18s
test / unit (pull_request) Successful in 42s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 26s
test / publish-infra (pull_request) Has been skipped
prd-number / assign-numbers (push) Failing after 29s
test / integration-docker (push) Successful in 44s
lint / lint (push) Successful in 1m0s
test / unit (push) Successful in 56s
Update Quality Badges / update-badges (push) Successful in 1m45s
test / integration-firecracker (push) Successful in 6m54s
test / coverage (push) Successful in 26s
test / publish-infra (push) Successful in 2m22s
2026-07-22 18:35:31 +00:00
didericis-claude bf8ff91b31 fix(manifest): presence-aware runtime merge for nested_containers
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 44s
lint / lint (push) Successful in 57s
test / integration-firecracker (pull_request) Successful in 3m31s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 13s
The runtime merge (merge_bottles_runtime / _fold_two_bottles) was using
OR semantics for nested_containers, which meant `nested_containers: false`
in a later bottle could never override an earlier `true`.

Fix: add `nested_containers_explicit: bool` to ManifestBottle, set to
True only when the key is present in the source dict. The runtime merge
functions now use the override's value only when it was explicitly
supplied; otherwise the base's value is preserved. This matches the
documented last-wins / presence-aware semantics used by supervise and
agent_provider, and aligns with _merge_bottles (the file-based extends
path) which already did presence checks via `"nested_containers" in
child_raw`.

Tests: rename existing test to clarify the preserved-when-omitted case,
add explicit-false-overrides-true and explicit-true-overrides-false cases.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 17:58:00 +00:00
didericis-claude 315ed04979 test: update unit tests to match podman-in-nested-containers-layer change
tracker-policy-pr / check-pr (pull_request) Successful in 8s
test / integration-docker (pull_request) Successful in 15s
test / unit (pull_request) Successful in 38s
lint / lint (push) Successful in 54s
test / integration-firecracker (pull_request) Successful in 4m21s
test / coverage (pull_request) Successful in 24s
test / publish-infra (pull_request) Has been skipped
- Replace test_all_install_podman with test_none_install_podman: agent
  images intentionally no longer include podman; the nested-containers
  derived layer is where it lives (see build_image in nested_containers.py).
- Add podman to the package assertions in test_macos_nested_containers so
  its presence in the derived Dockerfile is explicitly verified.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 17:36:51 +00:00
didericis-claude 9a04ab262b review: remove security-boundary note, move podman into nested-containers layer
test / integration-docker (pull_request) Successful in 19s
test / unit (pull_request) Failing after 39s
lint / lint (push) Successful in 56s
tracker-policy-pr / check-pr (pull_request) Failing after 13m51s
test / integration-firecracker (pull_request) Failing after 14m1s
test / coverage (pull_request) Has been skipped
test / publish-infra (pull_request) Has been skipped
- Drop the module-level paragraph emphasizing the single-UID security
  boundary (#issuecomment-5010): redundant given the safe environment.
- Move `podman` from each agent image (claude, codex, pi) into the
  `build_image` derived layer so bottles without `nested_containers`
  pay no image-size cost (#issuecomment-5013).
- Update `build_image` docstring and inline comments to reflect the move.
- Add TODO(#394) noting the planned docker-layer abstraction refactor.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-22 17:30:58 +00:00
didericis cc094765fd docs(readme): make the registry route example correct and complete
test / integration-docker (pull_request) Successful in 14s
lint / lint (push) Successful in 49s
test / unit (pull_request) Successful in 1m37s
test / integration-firecracker (pull_request) Successful in 3m25s
test / coverage (pull_request) Successful in 19s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 7s
The example previously left DLP enabled on the blob CDNs, which is
exactly backwards: those hosts carry the multi-hundred-MB layers whose
buffering triggers #455, while the registry hosts they hang off carry
little. It also listed one of quay's three CDNs and omitted the CDN
requirement from the prose, so a reader following it would authenticate
fine and then 403 partway through a pull.

Adds the by-design DNS behaviour and the BusyBox wget caveat, both of
which cost real time during the #392 verification: nslookup failing
inside a nested container looks like breakage but is expected, and
BusyBox wget reports "error getting response" for requests the proxy
served successfully.

Verified by feeding the snippet through parse_frontmatter +
ManifestBottle.from_dict: all nine routes parse with the intended
preserve_auth and DLP settings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 0ba25352b9 docs(nested-containers): record what live verification established
Documents the proxy-layer precedence that took four attempts to get
right, the blob-CDN routes every registry needs alongside its own, the
podman 5 networking packages and how each one fails when absent, and
the BusyBox wget caveat. Closes the DNS open question: public
resolution inside a nested container fails by design; reaching the
proxy was the real problem.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis dba48706de fix(macos): put the gateway address in the Docker CLI proxy config
This is the layer that actually decides it, and it was in the script
from the start: ~/.docker/config.json's proxies block is what the Docker
CLI copies into every container it starts. Being client-side it beats
the podman service outright, which is why containers.conf `env`,
http_proxy=false, and the service environment each looked correct on
disk and changed nothing a nested container saw.

Also corrects the comment on the service-env substitution, which
claimed to be the one place the address sticks. It is not; it covers
podman's own pulls and non-CLI API clients.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 854a8956ad fix(macos): substitute the gateway address before the service starts
Third attempt at the same symptom, and the first at the right layer.
podman's Docker-compatible API injects proxy settings from the *daemon*
environment after containers.conf is applied, so neither the env list
(892bfc16) nor http_proxy=false (21c144ae) could override it — the same
compat-path blind spot already seen with hosts_file. Nested containers
kept reporting `wget: bad address 'bot-bottle-gateway:9099'` through
both.

Substituting the resolved address into the service's own environment,
before `podman system service` starts, is the one place it sticks.
NO_PROXY keeps the name, which is matched against what a client asks
for.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 7a9628fc03 fix(macos): stop podman re-injecting the gateway name into containers
The address-bearing proxy URLs from 892bfc16 were written correctly but
never took effect: podman copies the host's proxy environment into every
container by default, and that copy overrides containers.conf `env`,
restoring the `bot-bottle-gateway` name a nested container cannot
resolve. Verified on the macOS host — containers still reported
`wget: bad address 'bot-bottle-gateway:9099'` with the addresses in
place.

Setting http_proxy=false disables only podman's own passthrough; the
proxy vars still reach containers, by address, from the env list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis ca8b2a9f2c fix(macos): reach the egress proxy by address in nested containers
podman's containers.conf `hosts_file` turns out to apply only to native
`podman run`; the Docker-compatible API ignores it, and the agent types
`docker`. (`base_hosts_file`, used before this, is a no-op in both
paths on podman 5.4.2.) So no config key can put the gateway name in a
nested container's /etc/hosts.

The name is therefore resolved in the bottle, where /etc/hosts does
carry it, and the address goes into the proxy URL handed to nested
containers. NO_PROXY keeps the name, since that is matched against what
a client asks for.

Verified on the macOS host: with the gateway reachable, a nested
container gets 200 from an allowlisted host and 403 from one that is
not — the egress boundary applies inside nested containers too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 96f5be48a6 fix(egress): emit preserve_auth into the generated routes
The flag from #453 was parsed off the manifest and honored by the
addon, but `_route_to_yaml_fields` never wrote it, so it was dropped on
the way to the proxy and has never worked end to end. Every registry
route silently kept the unconditional Authorization strip.

That is the whole of the Docker Hub / GHCR "unauthorized" blocker in
#392: podman fetched a valid 2658-byte bearer from auth.docker.io, the
proxy stripped it, and registry-1.docker.io answered the resulting
anonymous request with a fresh WWW-Authenticate challenge — which reads
exactly like a credential problem rather than a serializer gap.

Found by diffing a bottle's rendered routes.yaml against its manifest:
preserve_auth was in the manifest and absent from the output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 82cf9bab5a feat(macos): give nested containers working egress
Answers the last open question in #392: DNS from inside a nested
container was not a phantom, but it was also not a DNS problem. Public
resolution fails there by design — everything egresses through the
proxy — and the actual breakage was two missing pieces:

- The proxy URL names `bot-bottle-gateway`, which resolves only through
  the bottle's own /etc/hosts. Containers got their own hosts file, so
  they failed at "Could not resolve proxy", which reads like broken DNS.
  base_hosts_file propagates the bottle's entries.
- The gateway TLS-intercepts, so a container that does not trust the
  bottle's CA bundle fails with "unable to get local issuer
  certificate". The bundle is now mounted read-only and the usual env
  vars point at it, which covers curl, wget, python, and node without
  distro-specific trust commands.

Verified on the macOS host by reproducing each failure and confirming
that these three conditions applied by hand produced HTTP 200 from
inside a nested container. podman already forwards the proxy env, so
that needed nothing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 3c92e79775 fix(macos): install podman 5's full networking stack
netavark shells out to `nft` for the bridge network every compose file
expects, and aardvark-dns serves name resolution inside nested
containers. Neither arrives with the base image's
`--no-install-recommends` podman, and each fails at a different and
misleading layer: without nft containers are created but never start;
without aardvark-dns DNS inside a container fails while the engine, the
pulls, and the bridge all look healthy. That last one is the likeliest
explanation for the "DNS inside nested containers fails entirely" note
in #392 — a missing package, not a design gap.

Verified on the macOS host: with passt alone, `docker run` reached
netavark and died on "unable to execute nft".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis 220620bfcc fix(macos): install passt for podman 5 rootless networking
podman 5 uses pasta as the default rootless network helper; podman 4
used slirp4netns. The spike was written against bookworm's podman 4.3.1,
so its package list never included passt — and the trixie bump (#451)
changed the default out from under it. Every nested container failed to
start with "could not find pasta", while image pulls and the service
itself worked, which disguised a missing dependency as a compat-API bug.

The bootstrap now checks for pasta up front so this fails with a clear
message instead of surfacing at the first `docker run`.

Note the sun_path fix in a12d2191 did NOT cause or cure this; that path
was over the 108-byte limit independently and would have bitten as soon
as attach was reached.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis b0f012b8e6 fix(macos): shorten the podman runtime dir to fit sun_path
podman derives conmon's attach socket as
`$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`. With the
descriptive `/tmp/bot-bottle-podman-run` that path is 116 bytes, past
the 108-byte sun_path limit, so every attached `docker run` failed with
"unable to upgrade to tcp, received 500" — while image pulls and the
service itself worked, which is what made it look like a compat-API bug
rather than a path-length one.

Found on the macOS host: quay.io pulled fine and then no container
could be started. Test pins the budget so a more readable name cannot
silently reintroduce it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis fa9fed4194 fix(manifest): OR nested_containers across composed bottles
merge_bottles_runtime sees resolved bottles, where a bottle that never
mentioned the key is indistinguishable from one that set it false. With
"later replaces" semantics, `--bottle with-containers --bottle claude-dev`
silently dropped the capability — the second bottle's default clobbered
it. OR instead: composing bottles adds capabilities, it does not remove
them. The file-based extends path still sees raw keys, so an explicit
`nested_containers: false` in a child continues to turn it back off.

Also surface the flag in the launch summary, where its absence was the
only reason the clobber went unnoticed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
didericis d0b595828f feat(macos): run containers inside a bottle via guest-local podman
Adds the `nested_containers` bottle flag. On the macOS backend it starts
a rootless podman service inside the bottle and exposes its
Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. No host daemon socket is mounted and the guest gains
no capabilities; backends that cannot run a guest-local engine reject
the flag in the shared prepare template rather than ignoring it.

Podman rather than rootless Docker because Apple Container's capability
bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a
multi-range uid_map via newuidmap. With no subordinate UID range podman
falls back to a single-UID self-mapping an unprivileged process may
write itself, so the image build strips /etc/subuid and /etc/subgid
entries rather than adding them.

That mapping is also why nested containers are not an isolation layer:
root inside one is the agent user outside it. They are a build/test
convenience; the bottle remains the boundary.

Ports the spike branch onto main, renaming docker_access — it implied
Docker and granted access to nothing on the host — and drops podman
from the derived layer now that every built-in image ships it (#451).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-21 22:54:26 -04:00
257 changed files with 12552 additions and 5660 deletions
+8 -6
View File
@@ -103,14 +103,16 @@ jobs:
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
- name: Show environment
# Fail loudly if the backend this job promises isn't actually usable,
# rather than letting every test silently `unittest.skip` and the job
# go green on zero coverage. `backend status` prints a clear per-check
# summary (docker on PATH, daemon reachable) and exits non-zero when a
# prerequisite is missing — the same readiness check the skip guards
# gate on via `has_backend`.
- name: Preflight — Docker backend is ready
run: |
python3 --version
if command -v docker >/dev/null 2>&1; then
docker version || true
else
echo "docker not on PATH — integration tests will skip"
fi
python3 cli.py backend status --backend=docker
- name: Run integration tests (docker) with coverage
env:
+1
View File
@@ -35,6 +35,7 @@ backend remains available with `BOT_BOTTLE_BACKEND=docker` or
- `.bot-bottle/` — per-repo agent and bottle manifests (YAML markdown format).
- `examples/` — example bottles and agents showing the manifest format.
- `docs/README.md` — docs overview; when to write which document.
- `docs/glossary.md` — canonical term definitions (Agent Provider, Bottle, Sealed Bottle, etc.).
- `docs/prds/` — product requirement docs (see `docs/prds/README.md` for format).
- `docs/research/` — research notes (see `docs/research/README.md`).
- `docs/decisions/` — decision records (ADR-lite).
+10 -12
View File
@@ -8,8 +8,8 @@
# this image's mitmproxy / git / gitleaks payload.
#
# Collapses the prior per-daemon images (egress, git-gate,
# supervise) into one. A small stdlib-Python init supervisor at
# /app/gateway_init.py spawns all daemons, forwards SIGTERM, and
# supervise) into one. A small stdlib-Python init supervisor
# (`bot_bottle.gateway.bootstrap`) spawns all daemons, forwards SIGTERM, and
# propagates per-daemon stdout/stderr to the container log with a
# `[name]` prefix. See PRD 0024 for the rationale.
#
@@ -26,8 +26,9 @@
# /git-gate-entrypoint.sh docker-cp'd at start time
# /git-gate/creds/* docker-cp'd at start time
# /git/* bare repos, populated at runtime
# /run/supervise/bot-bottle.db bind-mounted at run time
# /home/mitmproxy/.mitmproxy/ mitmproxy CA dir
# (No bot-bottle.db mount: the data plane reaches the supervise queue over the
# control-plane RPC and never opens the DB — PRD 0070 / issue #469.)
#
# Exposed ports inside the container:
# 9099 egress (mitmproxy, agent-facing HTTPS proxy)
@@ -36,12 +37,9 @@
# 9100 supervise (MCP HTTP)
# Based on `python:3.12-slim` (Debian trixie) rather than the
# `mitmproxy/mitmproxy` image (Debian bookworm) so the whole stack —
# gateway here, and the firecracker infra image that builds FROM this —
# lands on trixie, whose buildah (1.39) can build agent Dockerfiles that
# use heredocs. mitmproxy is pip-installed to the same effect as the
# upstream image. (bookworm's buildah is 1.28, which can't parse
# `RUN ... <<EOF`; see the infra image + PR discussion.)
# `mitmproxy/mitmproxy` image (Debian bookworm), matching the trixie base the
# orchestrator image needs for buildah (Dockerfile.orchestrator.fc). mitmproxy
# is pip-installed to the same effect as the upstream image.
FROM python:3.12-slim
# Runtime system deps:
@@ -101,8 +99,8 @@ RUN pip install --no-cache-dir /src/
# WORKDIR here also creates /app so the shim + COPYs below can write into it
# (nothing created /app before this point).
WORKDIR /app
RUN printf 'from bot_bottle.egress_addon import addons\n' > /app/egress_addon.py
COPY bot_bottle/egress_entrypoint.sh /app/egress-entrypoint.sh
RUN printf 'from bot_bottle.gateway.egress.addon import addons\n' > /app/egress_addon.py
COPY bot_bottle/gateway/egress/entrypoint.sh /app/egress-entrypoint.sh
RUN chmod +x /app/egress-entrypoint.sh
# Pre-create runtime directories the compose renderer + start
@@ -122,4 +120,4 @@ EXPOSE 8888 9099 9418 9420 9100
# PID 1 is the supervisor. It owns signal handling and exit-code
# propagation; no `exec` chain in the entrypoint itself.
ENTRYPOINT ["python3", "-m", "bot_bottle.gateway_init"]
ENTRYPOINT ["python3", "-m", "bot_bottle.gateway.bootstrap"]
-22
View File
@@ -1,22 +0,0 @@
# Shared infra image: gateway data plane + orchestrator control plane.
#
# Used directly by the Docker backend (run as one `bot-bottle-infra`
# container, replacing the prior two-container split). The Firecracker
# backend extends this via Dockerfile.infra.fc, adding buildah/crun/
# netavark for in-VM agent-image building.
#
# Dockerfile.orchestrator is the single definition of the orchestrator
# content (the lean `bot_bottle` package on python:3.12-slim). Both this
# image and Dockerfile.infra.fc pull it in via `COPY --from`.
#
# multi-`FROM` can't union two bases (that's multi-stage, not multiple
# inheritance), so the orchestrator content is pulled in via `COPY --from`
# rather than a second base. Both images share the trixie `python:3.12-slim`
# base, so the copy is clean (same python; future installed deps copy too).
FROM bot-bottle-gateway:latest
# The orchestrator content, from its single definition. The gateway image
# already has the flat daemon modules under /app; this adds the full
# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves —
# used by gateway_init when BOT_BOTTLE_GATEWAY_DAEMONS includes `orchestrator`.
COPY --from=bot-bottle-orchestrator:latest /app/bot_bottle /app/bot_bottle
-23
View File
@@ -1,23 +0,0 @@
# Firecracker infra VM image (PRD 0070 Stage B).
#
# Extends the shared infra base (Dockerfile.infra: gateway + orchestrator
# control plane) with the in-VM agent-image builder. The Firecracker backend
# builds users' agent Dockerfiles *inside this VM* with buildah (rootless,
# daemonless) instead of on the host — no host Docker daemon, no
# root-equivalent `docker` group.
#
# Requires the trixie base from bot-bottle-gateway (buildah 1.39: bookworm's
# 1.28 can't parse Dockerfile heredocs that agent images use).
#
# `crun` is the OCI runtime; `netavark` + `aardvark-dns` are the network
# backend for `FROM` pulls + `RUN` egress. `vfs` + `chroot`: buildah works
# as root in the bare microVM (no fuse-overlayfs / overlay module / subuid
# maps). Matches image_builder.
FROM bot-bottle-infra:latest
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
+8 -10
View File
@@ -4,21 +4,19 @@
# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
# * the docker backend, which runs this image directly as the lean
# control-plane container; and
# * the firecracker infra image (Dockerfile.infra), which `COPY --from`s
# this image's `/app/bot_bottle` so the single infra VM runs the same
# control plane. Keeping it in one place means future orchestrator deps
# (e.g. iroh) are added here once, not duplicated per backend.
# * the firecracker orchestrator VM image (Dockerfile.orchestrator.fc),
# which is `FROM` this image and adds buildah for in-VM agent builds.
# Keeping the content in one place means future orchestrator deps (e.g.
# iroh) are added here once, not duplicated per backend.
#
# It stays deliberately lean: the control plane is **stdlib-only** today, so
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
# builder, and lives only in Dockerfile.infra). Keeping the secret-dense
# control plane on a minimal dependency surface is the point (PRD 0070's
# "secret concentration").
# builder, and lives only in Dockerfile.orchestrator.fc). Keeping the
# secret-dense control plane on a minimal dependency surface is the point
# (PRD 0070's "secret concentration").
#
# Shares the trixie `python:3.12-slim` base with the gateway image, so when
# the orchestrator grows real deps they can be `COPY --from`'d into the
# infra image cleanly (same base/python — installed packages copy safely).
# Shares the trixie `python:3.12-slim` base with the gateway image.
FROM python:3.12-slim
+23
View File
@@ -0,0 +1,23 @@
# Firecracker orchestrator-VM image (PRD 0070).
#
# The control-plane rootfs the orchestrator microVM boots: the lean orchestrator
# image + the in-VM agent-image builder. The Firecracker backend builds users'
# agent Dockerfiles *inside this VM* with buildah (rootless, daemonless) instead
# of on the host — no host Docker daemon, no root-equivalent `docker` group. The
# gateway VM boots a *separate*, slimmer rootfs (bot-bottle-gateway:latest) that
# carries none of this build tooling — the exposed data plane stays minimal.
#
# `crun` is the OCI runtime; `netavark` + `aardvark-dns` are the network backend
# for `FROM` pulls + `RUN` egress. `vfs` + `chroot`: buildah works as root in the
# bare microVM (no fuse-overlayfs / overlay module / subuid maps). The trixie
# base (from Dockerfile.orchestrator's python:3.12-slim) carries buildah 1.39,
# which parses the Dockerfile heredocs agent images use (bookworm's 1.28 can't).
# Matches image_builder.
FROM bot-bottle-orchestrator:latest
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
+91 -9
View File
@@ -5,7 +5,7 @@
# bot-bottle
[![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
[![coverage](https://img.shields.io/badge/coverage-83%25-brightgreen)](https://coverage.readthedocs.io/)
[![coverage](https://img.shields.io/badge/coverage-84%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-94%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
@@ -75,6 +75,88 @@ On compatible macOS hosts, the default backend requires Apple's `container` CLI
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
### Containers inside a bottle
A bottle may set `nested_containers: true`. On the macOS backend this starts a
guest-local, rootless **podman** service after the bottle is registered and
exposes its Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. Nothing is mounted from the host: Docker Desktop's socket
stays out of the bottle and the guest gains no outer VM capabilities. Backends
that cannot do this (`docker`, `firecracker`) reject the flag rather than
silently ignore it.
Rootless Docker was tried first and does not work here at all: Apple
Container's capability bounding set omits `CAP_SYS_ADMIN`, which the kernel
requires to write a multi-range `uid_map`. See
[`docs/research/rootless-docker-in-apple-container-spike.md`](docs/research/rootless-docker-in-apple-container-spike.md).
The tradeoff to understand before enabling it: podman avoids that requirement
by falling back to a single-UID mapping, so nested containers provide **no
isolation from the agent itself** — `root` inside a nested container is the
agent user outside it. Nested containers are a build/test convenience, not a
security boundary. The bottle remains the boundary.
Pulling images goes through the bottle's egress proxy like every other
request, so each registry needs a route — **and so does the CDN it redirects
layer blobs to**, which is a different host. Without the CDN route the pull
authenticates, fetches the manifest, then 403s partway through.
Docker Hub and GHCR additionally need `preserve_auth: true`: their token dance
uses a client-fetched per-scope bearer token that the proxy would otherwise
strip. Turn DLP off on every registry and CDN route — the bodies are
compressed layer blobs that no detector can read, and buffering them is what
triggers the shared-proxy OOM in #455.
```yaml
nested_containers: true
egress:
routes:
# Docker Hub: registry, token endpoint, blob CDN.
- host: registry-1.docker.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: auth.docker.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: production.cloudfront.docker.com
dlp: { outbound_detectors: false, inbound_detectors: false }
# GHCR: registry + blob CDN.
- host: ghcr.io
preserve_auth: true
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: pkg-containers.githubusercontent.com
dlp: { outbound_detectors: false, inbound_detectors: false }
# quay.io: registry + blob CDNs. No preserve_auth needed for public pulls.
- host: quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn01.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn02.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
- host: cdn03.quay.io
dlp: { outbound_detectors: false, inbound_detectors: false }
```
`mcr.microsoft.com` and `registry.k8s.io` follow the same shape and also
redirect blobs elsewhere (`*.data.mcr.microsoft.com` and
`us-*-docker.pkg.dev` respectively); route whichever host the 403 names.
Inside a nested container the same allowlist applies: an allowlisted host
returns 200 and anything else gets a 403 straight from the proxy. The
gateway's CA bundle and proxy settings are wired in automatically, so
`docker run … curl https://…` works with no extra flags — no `--add-host`,
`-e`, or `-v`.
Two things worth knowing when testing that:
- Public DNS inside a nested container fails **by design**. Everything
egresses through the proxy, so `nslookup` failing is expected and is not
evidence of a problem.
- Alpine's BusyBox `wget` drops the connection after the proxy's TLS
interception and reports `error getting response`, even though the proxy
logs the decrypted request and returns a response. Use `curl` to test
egress; BusyBox `wget` will lie to you.
### Firecracker on Linux
On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
@@ -123,14 +205,14 @@ git:
egress:
routes:
- host: gitea.dideric.is
auth:
scheme: token # Bearer | token
token_ref: BOT_BOTTLE_GITEA_TOKEN
matches: # optional — restrict to specific paths/methods/headers
- paths:
- {type: prefix, value: /api/v1/}
methods: [GET, POST, PATCH, DELETE]
dlp: # optional — per-route detector overrides (default: all on)
inspect:
auth:
scheme: token # Bearer | token
token_ref: BOT_BOTTLE_GITEA_TOKEN
matches: # optional — restrict to specific paths/methods/headers
- paths:
- {type: prefix, value: /api/v1/}
methods: [GET, POST, PATCH, DELETE]
outbound_detectors: [token_patterns, known_secrets]
inbound_detectors: false # disable response scanning for this host
---
+83 -810
View File
@@ -1,842 +1,115 @@
"""Per-backend bottle factories.
"""The bottle-backend package: abstract contract + backend selection.
A bottle is a running, isolated environment with claude inside. Each
backend exposes five methods:
Thin by design — nothing framework-heavy is imported at package init, so
importing any `backend.*` submodule (e.g. `backend.docker.util`) doesn't drag
the manifest / egress / git-gate framework into memory. The public names are
re-exported lazily:
prepare(spec, stage_dir=...) -> BottlePlan
Resolves names, validates host-side prerequisites, and writes
scratch files. No remote/runtime resources are created yet.
Safe to call before the y/N preflight.
* the abstract contract (`BottleBackend`, `BottleSpec`, `Bottle`, …) lives in
`backend.base`;
* backend selection / enumeration (`get_bottle_backend`,
`enumerate_active_agents`, …) in `backend.selection`;
* the concrete backends and the freeze helpers in their own submodules.
launch(plan) -> ContextManager[Bottle]
Brings up the container (or VM, or remote machine), provisions
it, yields a Bottle handle, and tears everything down on exit.
prepare_cleanup() -> BottleCleanupPlan
Enumerates orphaned resources left behind by previous bottles
(containers, networks, ...). Idempotent; no side effects.
cleanup(plan) -> None
Actually removes everything described by the cleanup plan.
enumerate_active() -> Sequence[ActiveAgent]
Return every currently-running bottle on this backend, with
enough metadata for callers (CLI `list active`, dashboard
agents pane) to render a row.
Selection is driven by `--backend` on `start` or BOT_BOTTLE_BACKEND
(env var). When neither is set, compatible macOS hosts default to
`macos-container`; Linux hosts with KVM default to `firecracker`;
otherwise `docker`. Per PRD 0003 the manifest does not carry a
backend field; the host picks.
`from bot_bottle.backend import X` resolves X on first access via `__getattr__`
and caches it at package level, so existing call-sites (and
`patch.object(backend_mod, X, …)`) keep working.
"""
from __future__ import annotations
import os
import shlex
import sys
from abc import ABC, abstractmethod
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING, Any, Generator, Generic, Sequence, TypeVar
from ..agent_provider import AgentProvisionPlan, get_provider, build_agent_provision_plan
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..log import die, info, warn
from ..util import read_tty_line
from ..manifest import Manifest, ManifestIndex
from ..supervise import SupervisePlan
from ..util import expand_tilde
from ..env import resolve_env, ResolvedEnv
from ..workspace import WorkspacePlan, workspace_plan
from .print_util import print_multi, visible_agent_env_names
from .util import host_skill_dir
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .base import (
ActiveAgent,
BackendStatus,
Bottle,
BottleBackend,
BottleCleanupPlan,
BottleImages,
BottlePlan,
BottleSpec,
ExecResult,
)
from .selection import (
enumerate_active_agents,
get_bottle_backend,
has_backend,
is_backend_available,
is_backend_ready,
known_backend_names,
)
from .docker import DockerBottleBackend
from .firecracker import FirecrackerBottleBackend
from .macos_container import MacosContainerBottleBackend
from .freeze import CommitCancelled, Freezer, get_freezer
@dataclass(frozen=True)
class BottleSpec:
"""CLI-supplied intent. Backend-agnostic — each backend's prepare
step consumes it and produces its own backend-specific plan.
Resolved values (image names, container name, scratch paths, runsc
availability) live on the plan, not the spec."""
manifest: ManifestIndex
agent_name: str
copy_cwd: bool
user_cwd: str
# PRD 0016 follow-up: when set, the backend's prepare step uses
# this identity instead of minting a fresh one — the resume path
# (`cli.py resume <identity>`) sets this to continue an existing
# bottle's state. Empty string for a fresh `start`.
identity: str = ""
label: str = ""
color: str = ""
# Ordered bottle names selected at launch (issue #269). When non-empty
# they are merged in order and replace the agent's `bottle:` field.
bottle_names: tuple[str, ...] = ()
# True when launched via --headless (no TTY, no interactive prompts).
# The git-gate host-key preflight uses this to error rather than prompt.
headless: bool = False
# Image startup policy. "fresh" preserves the normal build path;
# "cached" reuses the current local image/artifact without rebuilding.
image_policy: str = "fresh"
@dataclass(frozen=True)
class BottlePlan(ABC):
"""Base output of a backend's prepare step. Concrete subclasses
(e.g. DockerBottlePlan) add backend-specific resolved fields."""
spec: BottleSpec
manifest: Manifest
stage_dir: Path
git_gate_plan: GitGatePlan
@property
def guest_home(self) -> str:
return self.agent_provision.guest_home
@property
def git_gate_insteadof_host(self) -> str:
"""Host (and optional port) used in git-gate insteadOf URLs.
Docker uses the compose-network DNS alias; VM backends may
override with an IP:port when the guest has no DNS."""
return "git-gate"
@property
def git_gate_insteadof_scheme(self) -> str:
"""URL scheme for git-gate insteadOf rewrites. 'git' for
Docker (git daemon); VM backends may override (e.g. 'http'
over a published host port)."""
return "git"
egress_plan: EgressPlan
supervise_plan: SupervisePlan | None
agent_provision: AgentProvisionPlan
@property
def workspace_plan(self) -> WorkspacePlan:
return workspace_plan(self.spec, guest_home=self.guest_home)
def print(self) -> None:
"""Render the y/N preflight summary to stderr."""
spec = self.spec
manifest = self.manifest
agent = manifest.agent
bottle = manifest.bottle
env_names = visible_agent_env_names(
sorted(
set(bottle.env.keys())
| set(self.agent_provision.guest_env.keys())
),
hidden_env_names=self.agent_provision.hidden_env_names,
)
print(file=sys.stderr)
info(f"agent : {spec.agent_name}")
info(f"provider : {self.agent_provision.template}")
print_multi("env ", env_names)
print_multi("skills ", list(agent.skills))
effective_bottles = (
list(spec.bottle_names) if spec.bottle_names
else ([agent.bottle] if agent.bottle else [])
)
print_multi("bottle ", effective_bottles)
identity = manifest.git_identity_summary()
if identity:
info(f" git identity : {identity}")
git_lines = [
f"{u.name}{u.upstream_host}:{u.upstream_port}"
for u in self.git_gate_plan.upstreams
]
if git_lines:
print_multi(" git gate ", git_lines)
if self.egress_plan.routes:
egress_lines = []
for r in self.egress_plan.routes:
auth = f" [auth:{r.auth_scheme}]" if r.auth_scheme else ""
egress_lines.append(f"{r.host}{auth}")
print_multi(" egress ", egress_lines)
print(file=sys.stderr)
@dataclass(frozen=True)
class BottleCleanupPlan(ABC):
"""Base output of a backend's prepare_cleanup step. Concrete
subclasses (e.g. DockerBottleCleanupPlan) carry backend-specific
lists of resources to be removed and implement `print` + `empty`."""
@abstractmethod
def print(self) -> None:
"""Render the cleanup y/N summary to stderr."""
@property
@abstractmethod
def empty(self) -> bool:
"""True iff there is nothing to clean up; the CLI uses this to
short-circuit before showing the y/N."""
@dataclass(frozen=True)
class ExecResult:
"""Captured result of `Bottle.exec`. Backend-neutral: the Docker
impl populates it from a `subprocess.CompletedProcess`, but a
VM backend could populate it from any source that produces a
returncode + captured streams."""
returncode: int
stdout: str
stderr: str
@dataclass(frozen=True)
class ActiveAgent:
"""One currently-running agent, as the CLI `list active` and
dashboard agents pane render it. ("Agent" is the project's
consistent name for the thing running inside a bottle — the
bottle is the container, the agent is what runs in it.)
Fields are deliberately backend-neutral. `services` is the set
of gateway daemons currently up for this bottle (`egress`,
`git-gate`, `supervise`); the dashboard uses it to
gate edit verbs. `backend_name` is the matching key in
`_BACKENDS` (`docker` / `firecracker` / `macos-container`) — used by the active-
list rendering to disambiguate and by the dashboard's
re-attach path."""
backend_name: str
slug: str
agent_name: str # from metadata.json; "?" if missing
started_at: str # ISO 8601 from metadata.json; "" if missing
services: tuple[str, ...] # alphabetical
label: str = ""
color: str = ""
class Bottle(ABC):
"""Handle to a running bottle. Yielded by a backend's launch step.
`exec_agent` runs the selected agent CLI inside the bottle and
blocks until the session ends. `exec` runs a POSIX shell script inside the bottle
and returns the captured result. `cp_in` copies a host path into
the bottle. `close` is an idempotent alias for context-manager
teardown.
"""
name: str
@abstractmethod
def agent_argv(
self, argv: list[str], *, tty: bool = True,
) -> list[str]:
"""Return the host-side argv that runs the selected agent
inside the bottle. Used by `exec_agent` for foreground
handoffs and by the dashboard's tmux `respawn-pane` flow,
which needs the argv up front (it spawns claude in a tmux
pane rather than as a child of the current process).
Implementations transparently inject
`--append-system-prompt-file` when the bottle was launched
with a provisioned prompt path."""
...
@abstractmethod
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int: ...
@abstractmethod
def exec(self, script: str, *, user: str = "node") -> ExecResult:
"""Run `script` as a POSIX shell script inside the bottle as
`user` (default `node`, matching the agent image's USER
directive) and return the captured stdout/stderr/returncode.
The bottle's environment (including HTTPS_PROXY pointing at
the egress daemon) is inherited by the child. Non-zero
exit does not raise — callers inspect `returncode`
themselves.
Pass `user="root"` for shell-outs that need privileged file
writes / package install — provisioning calls that need root
bypass `Bottle.exec` and use the backend-specific raw
machine-exec helper, but the tests have a legitimate use
case for arbitrary-user runs."""
@abstractmethod
def cp_in(self, host_path: str, container_path: str) -> None: ...
@abstractmethod
def close(self) -> None: ...
PlanT = TypeVar("PlanT", bound=BottlePlan)
CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
@dataclass(frozen=True)
class BottleImages:
"""Resolved image references (or artifact paths) for a bottle launch.
For Docker/macOS-container backends, `agent` and `sidecar` are string
image refs. For the smolmachines backend they are Path objects pointing
to pre-built `.smolmachine` artifacts."""
agent: str | Path
sidecar: str | Path = ""
class BottleBackend(ABC, Generic[PlanT, CleanupT]):
"""Abstract base for selectable bottle backends. Concrete subclasses
(e.g. DockerBottleBackend) own their own prepare/launch impls.
Parameterized over the backend's concrete plan + cleanup-plan types
so subclass methods get the narrow type without isinstance
boilerplate."""
name: str
def prepare(self, spec: BottleSpec, stage_dir: Path) -> PlanT:
"""Template method: run cross-backend host-side validation, then
delegate to the subclass's `_resolve_plan` for the
backend-specific resolution (names, scratch files, etc.). The
validation step is enforced here so a future backend cannot
accidentally skip it. No remote/runtime resources are created."""
from .resolve_common import (
merge_provision_env_vars,
mint_slug,
prepare_agent_state_dir,
prepare_egress,
prepare_git_gate,
prepare_supervise,
resolve_manifest_dockerfile,
write_launch_metadata,
)
manifest = self._validate(spec)
self._preflight()
from ..git_gate_host_key import preflight_host_keys
manifest = preflight_host_keys(
manifest,
headless=spec.headless,
home_md=spec.manifest.home_md,
)
manifest_bottle = manifest.bottle
manifest_agent_provider = manifest_bottle.agent_provider
agent_provider = get_provider(manifest_agent_provider.template)
resolved_env = resolve_env(manifest)
workspace = workspace_plan(spec, guest_home=agent_provider.guest_home)
slug = mint_slug(spec)
write_launch_metadata(slug, spec, compose_project="", backend=self.name)
# Manifest may override the Dockerfile per-bottle; otherwise fall
# back to the provider plugin's bundled Dockerfile (next to its
# agent_provider.py module).
if manifest_agent_provider.dockerfile:
agent_dockerfile_path = resolve_manifest_dockerfile(
manifest_agent_provider.dockerfile, spec,
)
else:
agent_dockerfile_path = str(agent_provider.dockerfile)
agent_dir, prompt_file = prepare_agent_state_dir(slug, manifest)
agent_provision_plan = build_agent_provision_plan(
template=manifest_agent_provider.template,
dockerfile=agent_dockerfile_path,
state_dir=agent_dir,
instance_name=f"bot-bottle-{slug}",
prompt_file=prompt_file,
guest_env=self._build_guest_env(resolved_env),
forward_host_credentials=manifest_agent_provider.forward_host_credentials,
auth_token=manifest_agent_provider.auth_token,
host_env=dict(os.environ),
trusted_project_path=workspace.workdir,
label=spec.label,
color=spec.color,
provider_settings=manifest_agent_provider.settings,
)
agent_provision_plan = merge_provision_env_vars(agent_provision_plan)
egress_plan = prepare_egress(manifest_bottle, slug, agent_provision_plan)
supervise_plan = prepare_supervise(manifest_bottle, slug)
git_gate_plan = prepare_git_gate(manifest_bottle, slug)
return self._resolve_plan(
spec,
manifest=manifest,
slug=slug,
resolved_env=resolved_env,
agent_provision_plan=agent_provision_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
git_gate_plan=git_gate_plan,
stage_dir=stage_dir,
)
def _build_guest_env(self, resolved_env: ResolvedEnv) -> dict[str, str]:
return {}
def _preflight(self) -> None:
"""
tasks to do before resolving a plan
"""
pass
def _validate(self, spec: BottleSpec) -> Manifest:
"""Cross-backend pre-launch checks. Parses the selected agent and
its bottle (raising ManifestError on invalid content), confirms
skills are present on the host, and every git IdentityFile resolves.
Returns the loaded Manifest for the selected agent. Subclasses with
additional preconditions should override and call
`super()._validate(spec)` first."""
manifest = spec.manifest.load_for_agent(spec.agent_name, spec.bottle_names)
self._validate_skills(manifest.agent.skills)
self._validate_agent_provider_dockerfile(spec, manifest)
return manifest
def _validate_skills(self, skills: Sequence[str]) -> None:
"""Each named skill must be a directory under the host's
`~/.claude/skills/`. The check is purely host-side, so the
default impl covers every backend."""
for name in skills:
path = host_skill_dir(name)
if not os.path.isdir(path):
die(
f"skill '{name}' not found on host at {path}. "
f"Create it under ~/.claude/skills/, then re-run."
)
def _validate_agent_provider_dockerfile(self, spec: BottleSpec, manifest: Manifest) -> None:
bottle = manifest.bottle
dockerfile = bottle.agent_provider.dockerfile
if not dockerfile:
return
path = Path(expand_tilde(dockerfile))
if not path.is_absolute():
path = Path(spec.user_cwd) / path
if not path.is_file():
effective = (
", ".join(spec.bottle_names) if spec.bottle_names else manifest.agent.bottle
)
die(
f"agent_provider.dockerfile for bottle "
f"'{effective}' not found: {path}"
)
@abstractmethod
def _resolve_plan(self,
spec: BottleSpec,
*,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
supervise_plan: SupervisePlan | None,
stage_dir: Path) -> PlanT:
"""Backend-specific plan resolution: image/container names,
env-file, prompt-file, proxy plan, runtime detection. Called by
`prepare` after `_validate` succeeds. Instance name, image,
prompt file, Dockerfile path, and guest home all live on
`agent_provision_plan` — the source of truth."""
def prelaunch_checks(self, plan: PlanT) -> None:
"""Raise StaleImageError if any cached image used by this plan is stale.
No-op default; backends override to call the shared check_stale*
helpers on their image/artifact timestamps. Called by the CLI before
launch so the operator can be prompted outside the launch context."""
@contextmanager
def launch(self, plan: PlanT) -> Generator[Bottle, None, None]:
"""Template: build or load images, then delegate to _launch_impl."""
images = self._build_or_load_images(plan)
with self._launch_impl(plan, images) as bottle:
yield bottle
@abstractmethod
def _build_or_load_images(self, plan: PlanT) -> BottleImages:
"""Return the agent and sidecar image references (or artifact paths)
for this plan, building fresh images when the policy requires it."""
@abstractmethod
def _launch_impl(self, plan: PlanT, images: BottleImages) -> AbstractContextManager[Bottle]:
"""Bring up the bottle using pre-resolved images; yield a handle; tear down on exit."""
def provision(self, plan: PlanT, bottle: "Bottle") -> str | None:
"""Copy host-side files (CA cert, prompt, skills, .git) into
the running bottle. Called from `launch` after the container
/ machine is up. Returns the in-container prompt path if a
prompt was provisioned, else None — the Bottle handle uses it
to decide whether to add provider-specific prompt args to the
agent's argv.
Default orchestration: ca → prompt → provider apply → skills
→ workspace → git → supervise-mcp. CA install runs first so
the agent's trust store is rebuilt before anything inside the
agent makes a TLS call.
Per PRD 0050 the per-provider steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin. The backend only owns the
steps that are about backend infrastructure (CA, workspace,
git) and surfaces the supervise daemon URL its launch step
knows about via `supervise_mcp_url`.
PRD 0017: cred-proxy's agent-side dotfile rewrites (~/.npmrc,
~/.gitconfig insteadOf, tea config) are gone. Egress-proxy is
on the agent's HTTP_PROXY path so every tool that respects
HTTPS_PROXY (claude-code, git over HTTPS, npm, curl) is
intercepted without per-tool reconfiguration."""
provider = get_provider(plan.agent_provision.template)
provider.provision_ca(bottle, plan)
prompt_path = provider.provision_prompt(plan, bottle)
provider.provision(plan, bottle)
provider.provision_skills(plan, bottle)
self.provision_workspace(plan, bottle)
provider.provision_git(bottle, plan)
provider.provision_supervise_mcp(
plan, bottle, self.supervise_mcp_url(plan),
)
return prompt_path
def provision_workspace(self, plan: PlanT, bottle: "Bottle") -> None:
"""Copy the operator workspace into the running bottle.
This is the only supported workspace-provisioning path: Docker
does not build a derived image containing the current
workspace."""
workspace = plan.workspace_plan
if not (workspace.enabled and workspace.copy_contents):
return
guest_parent = workspace.guest_path.rsplit("/", 1)[0] or "/"
guest_path = shlex.quote(workspace.guest_path)
guest_parent = shlex.quote(guest_parent)
owner = shlex.quote(workspace.owner)
mode = shlex.quote(workspace.mode)
info(f"copying {workspace.host_path} -> {bottle.name}:{workspace.guest_path}")
bottle.exec(
f"rm -rf {guest_path} && mkdir -p {guest_parent}",
user="root",
)
bottle.cp_in(str(workspace.host_path), workspace.guest_path)
bottle.exec(
f"chown -R {owner} {guest_path} && chmod {mode} {guest_path}",
user="root",
)
def supervise_mcp_url(self, plan: PlanT) -> str:
"""Return the agent-side URL of the per-bottle supervise
gateway, or "" when this bottle has no gateway. The provider
plugin's `provision_supervise_mcp` uses it to register the
MCP entry inside the guest.
Default returns "" so backends without supervise support
don't have to implement it. Docker and firecracker override."""
del plan
return ""
def ensure_orchestrator(self) -> str:
"""Bring up this backend's per-host orchestrator + shared gateway
(idempotent) and return the host-reachable control-plane URL.
This is the backend-agnostic bring-up entry point: `launch` calls
it as part of starting a bottle, and operator tools (`supervise`)
call it to start the control plane on demand when none is running
yet. Docker starts the orchestrator + gateway containers;
firecracker boots the infra VM. Backends with no orchestrator
(macos-container) die with a pointer — the default here."""
die(f"backend {self.name!r} has no orchestrator control plane")
@abstractmethod
def prepare_cleanup(self) -> CleanupT:
"""Enumerate orphaned resources from previous bottles. No side
effects; safe to call before the y/N."""
@abstractmethod
def cleanup(self, plan: CleanupT) -> None:
"""Remove everything described by the cleanup plan."""
@abstractmethod
def enumerate_active(self) -> Sequence[ActiveAgent]:
"""Return every currently-running agent on this backend.
Empty when none. Backend-specific: docker queries `docker
compose ls`; firecracker cross-references its running gateway
containers against per-bottle metadata."""
@classmethod
@abstractmethod
def is_available(cls) -> bool:
"""Whether this backend's runtime prerequisites are satisfied
on the current host. Docker → `docker` on PATH; firecracker →
Linux + KVM. Used by the cross-backend
`enumerate_active_agents` / `cmd_cleanup` to skip backends
the operator hasn't installed, so a docker-only host
doesn't fail when `cli.py list active` walks past
firecracker."""
@classmethod
@abstractmethod
def setup(cls) -> int:
"""Emit this backend's one-time host setup — privileged network
pool, daemon bring-up, install pointers, etc. — as
host-appropriate config or commands. Prints to stdout/stderr and
returns a shell exit code (0 = nothing to report / success). A
backend that needs no host setup prints a short note and returns
0. Invoked generically by `./cli.py backend setup [--backend=…]`
so operators can provision any backend without a
backend-specific command. Classmethod (like `is_available`) —
it's a host query, not per-bottle state."""
@classmethod
@abstractmethod
def status(cls) -> int:
"""Report whether this backend's prerequisites are satisfied on
the host — binaries, daemon reachability, network pool, range
conflicts, etc. Prints a human-readable summary; returns 0 when
the backend is ready to launch and non-zero when something is
missing. Invoked by `./cli.py backend status [--backend=…]`."""
@classmethod
@abstractmethod
def teardown(cls) -> int:
"""Undo `setup()` — the inverse operation, surfaced as
`./cli.py backend teardown [--backend=…]` (uninstall). Symmetric
with setup: where setup is advisory (prints the privileged
commands / declarative config to apply), teardown prints the
commands / config change to remove the host prerequisites. A
backend with no host setup prints a short note and returns 0.
Not called by the launch path or the test suite."""
# _backends is None until the first call to _get_backends(), at which
# point all three concrete backend classes are imported and instantiated.
# Keeping the imports out of module scope means that importing any
# backend sub-module (e.g. `backend.docker.util`) no longer drags the
# firecracker and macos-container implementations into memory.
#
# Tests may replace _backends with a {name: fake} dict via patch.object;
# _get_backends() returns the current module-level value as-is when it
# is not None, so test fakes take effect without triggering real imports.
_backends: dict[str, BottleBackend[Any, Any]] | None = None
def _get_backends() -> dict[str, BottleBackend[Any, Any]]:
"""Return the registry of all backend instances, loading lazily on first call."""
global _backends # pylint: disable=global-statement
if _backends is None:
from .docker import DockerBottleBackend
from .firecracker import FirecrackerBottleBackend
from .macos_container import MacosContainerBottleBackend
_backends = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
}
return _backends
# Public name -> submodule that defines it. Contract types resolve from `base`,
# selection/enumeration from `selection`, the concrete backends + freeze helpers
# from their own subpackages.
_LAZY_MODULES: dict[str, str] = {
"BottleSpec": "base",
"BottlePlan": "base",
"BottleCleanupPlan": "base",
"ExecResult": "base",
"ActiveAgent": "base",
"Bottle": "base",
"BottleImages": "base",
"BottleBackend": "base",
"BackendStatus": "base",
"get_bottle_backend": "selection",
"known_backend_names": "selection",
"has_backend": "selection",
"is_backend_available": "selection",
"is_backend_ready": "selection",
"enumerate_active_agents": "selection",
"_print_vm_install_instructions": "selection",
"DockerBottleBackend": "docker",
"FirecrackerBottleBackend": "firecracker",
"MacosContainerBottleBackend": "macos_container",
"CommitCancelled": "freeze",
"Freezer": "freeze",
"get_freezer": "freeze",
}
def __getattr__(name: str) -> Any:
"""Lazily surface concrete backend classes and freeze symbols at the
package level so existing `from bot_bottle.backend import X` and
`patch.object(backend_mod, X, ...)` call-sites keep working without
forcing an import of every backend at module-init time."""
if name == "DockerBottleBackend":
from .docker import DockerBottleBackend
globals()[name] = DockerBottleBackend
return DockerBottleBackend
if name == "FirecrackerBottleBackend":
from .firecracker import FirecrackerBottleBackend
globals()[name] = FirecrackerBottleBackend
return FirecrackerBottleBackend
if name == "MacosContainerBottleBackend":
from .macos_container import MacosContainerBottleBackend
globals()[name] = MacosContainerBottleBackend
return MacosContainerBottleBackend
if name == "CommitCancelled":
from .freeze import CommitCancelled
globals()[name] = CommitCancelled
return CommitCancelled
if name == "Freezer":
from .freeze import Freezer
globals()[name] = Freezer
return Freezer
if name == "get_freezer":
from .freeze import get_freezer
globals()[name] = get_freezer
return get_freezer
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
"""Lazily surface the package's public names from their submodules and
cache them at package level so `from bot_bottle.backend import X` and
`patch.object(backend_mod, X, …)` keep working without importing the
framework (or every backend) at package-init time."""
mod = _LAZY_MODULES.get(name)
if mod is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from importlib import import_module
def get_bottle_backend(
name: str | None = None,
*,
prompt: bool = True,
) -> BottleBackend[Any, Any]:
"""Resolve the bottle backend.
`name` precedence:
1. explicit arg (e.g. resume passes the recorded backend name)
2. BOT_BOTTLE_BACKEND env var
3. auto-selection: VM backend first, docker fallback with prompt
`prompt` controls whether auto-selection may block on an interactive
[i/d/q] prompt when falling back to docker. Pass `prompt=False` in
non-interactive contexts (headless launches, CI) so the call dies
with an actionable message instead of hanging.
Dies with a pointer at the known backends if the chosen name
isn't implemented."""
resolved = name or os.environ.get("BOT_BOTTLE_BACKEND")
if resolved is None:
resolved = _auto_select_backend(prompt=prompt)
backends = _get_backends()
if resolved not in backends:
known = ", ".join(sorted(backends))
die(f"unknown backend {resolved!r}; known backends: {known}")
return backends[resolved]
def _platform_vm_suggestion() -> str:
"""Platform-appropriate VM backend name for install suggestions."""
return "macos-container" if sys.platform == "darwin" else "firecracker"
def _print_vm_install_instructions() -> None:
"""Print platform-appropriate VM backend install instructions to stderr."""
vm = _platform_vm_suggestion()
if vm == "macos-container":
info("Install Apple Container: https://github.com/apple/container/releases")
info("Then start the service: container system start")
else:
info("Install Firecracker: https://github.com/firecracker-microvm/firecracker/releases")
info("Configure the host: ./cli.py backend setup")
def _auto_select_backend(prompt: bool = True) -> str:
"""Tier-1 / tier-2 backend auto-selection.
Tier 1: VM backend — macos-container on macOS when Apple Container is
installed; firecracker on KVM-capable Linux even before the binary is
present (its preflight prints an install pointer).
Tier 2: docker, with a security warning and an interactive prompt.
When `prompt=False` (headless / CI), dies with an actionable message
instead of blocking on a TTY read. When docker is also absent, prints
VM install instructions and exits.
"""
# --- Tier 1: VM backend -----------------------------------------
if has_backend("macos-container"):
return "macos-container"
# A KVM-capable Linux host defaults to firecracker even when the
# `firecracker` binary isn't installed yet: selecting it here routes
# start through firecracker's preflight, which prints an install
# pointer, instead of silently falling back to docker.
from .firecracker import FirecrackerBottleBackend
if FirecrackerBottleBackend.is_host_capable():
return "firecracker"
# --- Tier 2: docker fallback ------------------------------------
if not has_backend("docker"):
info("No backend available on this host.")
_print_vm_install_instructions()
die("no backend available; install a VM backend and re-run")
vm = _platform_vm_suggestion()
warn(
"docker is less secure than VM backends — "
"containers share the host kernel."
)
if not prompt:
die(
f"no VM backend available; set BOT_BOTTLE_BACKEND=docker to proceed "
f"with docker, or install the {vm!r} backend."
)
sys.stderr.write(
f"bot-bottle: For better isolation, install the {vm!r} backend.\n"
f" [i] show {vm} install instructions and exit\n"
" [d] use docker anyway\n"
" [q] quit\n"
"bot-bottle: choice [i/d/q]: "
)
sys.stderr.flush()
reply = read_tty_line().strip().lower()
if reply == "d":
return "docker"
if reply == "i":
_print_vm_install_instructions()
die("not proceeding with docker; install a VM backend or set BOT_BOTTLE_BACKEND=docker")
def known_backend_names() -> tuple[str, ...]:
"""Sorted tuple of all backend keys in `_get_backends()`. Used by
argparse (`--backend` choices) and the dashboard's backend
picker."""
return tuple(sorted(_get_backends()))
def has_backend(name: str) -> bool:
"""Whether the named backend's runtime prerequisites are
available on the current host. Cross-backend callers (list,
cleanup) skip unavailable backends so a docker-only host
doesn't fail when the firecracker backend isn't usable,
and vice versa.
Returns False for unknown names so callers can pass
arbitrary input without separate validation."""
backends = _get_backends()
if name not in backends:
return False
return backends[name].is_available()
def enumerate_active_agents() -> list[ActiveAgent]:
"""All currently-running agents, across every available
backend. Used by CLI `list active` and the dashboard's agents
pane so neither has to know which backends exist. Skips
backends whose `is_available()` reports False.
Sorted by `(started_at, slug)` so the list is stable across
dashboard refresh ticks — agents don't shift position while
the operator navigates with arrow keys. ISO 8601 timestamps
sort lexicographically in chronological order; `slug` is the
deterministic tiebreaker. Agents with missing metadata
(`started_at == ""`) sort first."""
out: list[ActiveAgent] = []
backends = _get_backends()
for name in sorted(backends):
if not backends[name].is_available():
continue
out.extend(backends[name].enumerate_active())
out.sort(key=lambda a: (a.started_at, a.slug))
return out
value = getattr(import_module(f"{__name__}.{mod}"), name)
globals()[name] = value
return value
__all__ = [
"ActiveAgent",
"BackendStatus",
"Bottle",
"BottleBackend",
"BottleCleanupPlan",
"BottleImages",
"BottlePlan",
"BottleSpec",
"CommitCancelled",
"ExecResult",
"CommitCancelled",
"Freezer",
"get_freezer",
"DockerBottleBackend",
"FirecrackerBottleBackend",
"MacosContainerBottleBackend",
"enumerate_active_agents",
"get_bottle_backend",
"get_freezer",
"has_backend",
"is_backend_available",
"is_backend_ready",
"known_backend_names",
]
+619
View File
@@ -0,0 +1,619 @@
"""The abstract backend contract (PRD 0018 / 0070).
The backend-neutral types every bottle backend implements: the launch
`BottleSpec`, the `BottlePlan` / `BottleCleanupPlan` ABCs, the running-`Bottle`
+ `ExecResult` shapes, `BottleImages`, and the `BottleBackend` ABC itself.
This carries the framework imports (manifest, egress, git-gate, env, workspace,
agent-provider) the contract's signatures and helpers need — which is why it
lives here rather than in `backend/__init__.py`: touching an unrelated
`backend.*` module then doesn't drag the whole framework into memory. The
thin package `__init__` re-exports these names lazily.
"""
from __future__ import annotations
import enum
import os
import shlex
import sys
from abc import ABC, abstractmethod
from contextlib import AbstractContextManager, contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Generator, Generic, Sequence, TypeVar
from ..agent_provider import AgentProvisionPlan, get_provider, build_agent_provision_plan
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..log import die, info
from ..util import expand_tilde
from ..manifest import Manifest, ManifestIndex
from ..supervisor.plan import SupervisePlan
from ..env import resolve_env, ResolvedEnv
from ..workspace import WorkspacePlan, workspace_plan
from .print_util import print_multi, visible_agent_env_names
from .util import host_skill_dir
class BackendStatus(enum.IntEnum):
"""Return codes for BottleBackend.status(). READY == 0 so callsites
can compare against 0 or the named constant interchangeably."""
READY = 0
@dataclass(frozen=True)
class BottleSpec:
"""CLI-supplied intent. Backend-agnostic — each backend's prepare
step consumes it and produces its own backend-specific plan.
Resolved values (image names, container name, scratch paths, runsc
availability) live on the plan, not the spec."""
manifest: ManifestIndex
agent_name: str
copy_cwd: bool
user_cwd: str
# PRD 0016 follow-up: when set, the backend's prepare step uses
# this identity instead of minting a fresh one — the resume path
# (`cli.py resume <identity>`) sets this to continue an existing
# bottle's state. Empty string for a fresh `start`.
identity: str = ""
label: str = ""
color: str = ""
# Ordered bottle names selected at launch (issue #269). When non-empty
# they are merged in order and replace the agent's `bottle:` field.
bottle_names: tuple[str, ...] = ()
# True when launched via --headless (no TTY, no interactive prompts).
# The git-gate host-key preflight uses this to error rather than prompt.
headless: bool = False
# Image startup policy. "fresh" preserves the normal build path;
# "cached" reuses the current local image/artifact without rebuilding.
image_policy: str = "fresh"
@dataclass(frozen=True)
class BottlePlan(ABC):
"""Base output of a backend's prepare step. Concrete subclasses
(e.g. DockerBottlePlan) add backend-specific resolved fields."""
spec: BottleSpec
manifest: Manifest
stage_dir: Path
git_gate_plan: GitGatePlan
@property
def guest_home(self) -> str:
return self.agent_provision.guest_home
@property
def git_gate_insteadof_host(self) -> str:
"""Host (and optional port) used in git-gate insteadOf URLs.
Docker uses the compose-network DNS alias; VM backends may
override with an IP:port when the guest has no DNS."""
return "git-gate"
@property
def git_gate_insteadof_scheme(self) -> str:
"""URL scheme for git-gate insteadOf rewrites. 'git' for
Docker (git daemon); VM backends may override (e.g. 'http'
over a published host port)."""
return "git"
egress_plan: EgressPlan
supervise_plan: SupervisePlan | None
agent_provision: AgentProvisionPlan
@property
def workspace_plan(self) -> WorkspacePlan:
return workspace_plan(self.spec, guest_home=self.guest_home)
def print(self) -> None:
"""Render the y/N preflight summary to stderr."""
spec = self.spec
manifest = self.manifest
agent = manifest.agent
bottle = manifest.bottle
env_names = visible_agent_env_names(
sorted(
set(bottle.env.keys())
| set(self.agent_provision.guest_env.keys())
),
hidden_env_names=self.agent_provision.hidden_env_names,
)
print(file=sys.stderr)
info(f"agent : {spec.agent_name}")
info(f"provider : {self.agent_provision.template}")
print_multi("env ", env_names)
print_multi("skills ", list(agent.skills))
effective_bottles = (
list(spec.bottle_names) if spec.bottle_names
else ([agent.bottle] if agent.bottle else [])
)
print_multi("bottle ", effective_bottles)
identity = manifest.git_identity_summary()
if identity:
info(f" git identity : {identity}")
git_lines = [
f"{u.name}{u.upstream_host}:{u.upstream_port}"
for u in self.git_gate_plan.upstreams
]
if git_lines:
print_multi(" git gate ", git_lines)
if self.egress_plan.routes:
egress_lines = []
for r in self.egress_plan.routes:
auth = f" [auth:{r.auth_scheme}]" if r.auth_scheme else ""
egress_lines.append(f"{r.host}{auth}")
print_multi(" egress ", egress_lines)
print(file=sys.stderr)
@dataclass(frozen=True)
class BottleCleanupPlan(ABC):
"""Base output of a backend's prepare_cleanup step. Concrete
subclasses (e.g. DockerBottleCleanupPlan) carry backend-specific
lists of resources to be removed and implement `print` + `empty`."""
@abstractmethod
def print(self) -> None:
"""Render the cleanup y/N summary to stderr."""
@property
@abstractmethod
def empty(self) -> bool:
"""True iff there is nothing to clean up; the CLI uses this to
short-circuit before showing the y/N."""
@dataclass(frozen=True)
class ExecResult:
"""Captured result of `Bottle.exec`. Backend-neutral: the Docker
impl populates it from a `subprocess.CompletedProcess`, but a
VM backend could populate it from any source that produces a
returncode + captured streams."""
returncode: int
stdout: str
stderr: str
@dataclass(frozen=True)
class ActiveAgent:
"""One currently-running agent, as the CLI `active` and
dashboard agents pane render it. ("Agent" is the project's
consistent name for the thing running inside a bottle — the
bottle is the container, the agent is what runs in it.)
Fields are deliberately backend-neutral. `services` is the set
of gateway daemons currently up for this bottle (`egress`,
`git-gate`, `supervise`); the dashboard uses it to
gate edit verbs. `backend_name` is the matching key in
`_BACKENDS` (`docker` / `firecracker` / `macos-container`) — used by the active-
list rendering to disambiguate and by the dashboard's
re-attach path."""
backend_name: str
slug: str
agent_name: str # from metadata.json; "?" if missing
started_at: str # ISO 8601 from metadata.json; "" if missing
services: tuple[str, ...] # alphabetical
label: str = ""
color: str = ""
class Bottle(ABC):
"""Handle to a running bottle. Yielded by a backend's launch step.
`exec_agent` runs the selected agent CLI inside the bottle and
blocks until the session ends. `exec` runs a POSIX shell script inside the bottle
and returns the captured result. `cp_in` copies a host path into
the bottle. `close` is an idempotent alias for context-manager
teardown.
"""
name: str
@abstractmethod
def agent_argv(
self, argv: list[str], *, tty: bool = True,
) -> list[str]:
"""Return the host-side argv that runs the selected agent
inside the bottle. Used by `exec_agent` for foreground
handoffs and by the dashboard's tmux `respawn-pane` flow,
which needs the argv up front (it spawns claude in a tmux
pane rather than as a child of the current process).
Implementations transparently inject
`--append-system-prompt-file` when the bottle was launched
with a provisioned prompt path."""
...
@abstractmethod
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int: ...
@abstractmethod
def exec(self, script: str, *, user: str = "node") -> ExecResult:
"""Run `script` as a POSIX shell script inside the bottle as
`user` (default `node`, matching the agent image's USER
directive) and return the captured stdout/stderr/returncode.
The bottle's environment (including HTTPS_PROXY pointing at
the egress daemon) is inherited by the child. Non-zero
exit does not raise — callers inspect `returncode`
themselves.
Pass `user="root"` for shell-outs that need privileged file
writes / package install — provisioning calls that need root
bypass `Bottle.exec` and use the backend-specific raw
machine-exec helper, but the tests have a legitimate use
case for arbitrary-user runs."""
@abstractmethod
def cp_in(self, host_path: str, container_path: str) -> None: ...
@abstractmethod
def close(self) -> None: ...
PlanT = TypeVar("PlanT", bound=BottlePlan)
CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
@dataclass(frozen=True)
class BottleImages:
"""Resolved image references (or artifact paths) for a bottle launch.
For Docker/macOS-container backends, `agent` and `sidecar` are string
image refs. For the smolmachines backend they are Path objects pointing
to pre-built `.smolmachine` artifacts."""
agent: str | Path
sidecar: str | Path = ""
class BottleBackend(ABC, Generic[PlanT, CleanupT]):
"""Abstract base for selectable bottle backends. Concrete subclasses
(e.g. DockerBottleBackend) own their own prepare/launch impls.
Parameterized over the backend's concrete plan + cleanup-plan types
so subclass methods get the narrow type without isinstance
boilerplate."""
name: str
# Whether this backend can run a container engine *inside* the bottle.
# Backends that cannot must reject `nested_containers: true` rather than
# reach for a host daemon socket (issue #392).
supports_nested_containers: bool = False
def prepare(self, spec: BottleSpec, stage_dir: Path) -> PlanT:
"""Template method: run cross-backend host-side validation, then
delegate to the subclass's `_resolve_plan` for the
backend-specific resolution (names, scratch files, etc.). The
validation step is enforced here so a future backend cannot
accidentally skip it. No remote/runtime resources are created."""
from .resolve_common import (
merge_provision_env_vars,
mint_slug,
prepare_agent_state_dir,
prepare_egress,
prepare_git_gate,
prepare_supervise,
reject_nested_containers,
resolve_manifest_dockerfile,
write_launch_metadata,
)
manifest = self._validate(spec)
if not self.supports_nested_containers:
reject_nested_containers(self.name, manifest)
self._preflight()
from ..git_gate import GitGate
manifest = GitGate().preflight_host_keys(
manifest,
headless=spec.headless,
home_md=spec.manifest.home_md,
)
manifest_bottle = manifest.bottle
manifest_agent_provider = manifest_bottle.agent_provider
agent_provider = get_provider(manifest_agent_provider.template)
resolved_env = resolve_env(manifest)
workspace = workspace_plan(spec, guest_home=agent_provider.guest_home)
slug = mint_slug(spec)
write_launch_metadata(slug, spec, compose_project="", backend=self.name)
# Manifest may override the Dockerfile per-bottle; otherwise fall
# back to the provider plugin's bundled Dockerfile (next to its
# agent_provider.py module).
if manifest_agent_provider.dockerfile:
agent_dockerfile_path = resolve_manifest_dockerfile(
manifest_agent_provider.dockerfile, spec,
)
else:
agent_dockerfile_path = str(agent_provider.dockerfile)
agent_dir, prompt_file = prepare_agent_state_dir(slug, manifest)
agent_provision_plan = build_agent_provision_plan(
template=manifest_agent_provider.template,
dockerfile=agent_dockerfile_path,
state_dir=agent_dir,
instance_name=f"bot-bottle-{slug}",
prompt_file=prompt_file,
guest_env=self._build_guest_env(resolved_env),
forward_host_credentials=manifest_agent_provider.forward_host_credentials,
auth_token=manifest_agent_provider.auth_token,
host_env=dict(os.environ),
trusted_project_path=workspace.workdir,
label=spec.label,
color=spec.color,
provider_settings=manifest_agent_provider.settings,
)
agent_provision_plan = merge_provision_env_vars(agent_provision_plan)
egress_plan = prepare_egress(manifest_bottle, slug, agent_provision_plan)
supervise_plan = prepare_supervise(manifest_bottle, slug)
git_gate_plan = prepare_git_gate(manifest_bottle, slug)
return self._resolve_plan(
spec,
manifest=manifest,
slug=slug,
resolved_env=resolved_env,
agent_provision_plan=agent_provision_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
git_gate_plan=git_gate_plan,
stage_dir=stage_dir,
)
def _build_guest_env(self, resolved_env: ResolvedEnv) -> dict[str, str]:
return {}
def _preflight(self) -> None:
"""
tasks to do before resolving a plan
"""
pass
def _validate(self, spec: BottleSpec) -> Manifest:
"""Cross-backend pre-launch checks. Parses the selected agent and
its bottle (raising ManifestError on invalid content), confirms
skills are present on the host, and every git IdentityFile resolves.
Returns the loaded Manifest for the selected agent. Subclasses with
additional preconditions should override and call
`super()._validate(spec)` first."""
manifest = spec.manifest.load_for_agent(spec.agent_name, spec.bottle_names)
self._validate_skills(manifest.agent.skills)
self._validate_agent_provider_dockerfile(spec, manifest)
return manifest
def _validate_skills(self, skills: Sequence[str]) -> None:
"""Each named skill must be a directory under the host's
`~/.claude/skills/`. The check is purely host-side, so the
default impl covers every backend."""
for name in skills:
path = host_skill_dir(name)
if not os.path.isdir(path):
die(
f"skill '{name}' not found on host at {path}. "
f"Create it under ~/.claude/skills/, then re-run."
)
def _validate_agent_provider_dockerfile(self, spec: BottleSpec, manifest: Manifest) -> None:
bottle = manifest.bottle
dockerfile = bottle.agent_provider.dockerfile
if not dockerfile:
return
path = Path(expand_tilde(dockerfile))
if not path.is_absolute():
path = Path(spec.user_cwd) / path
if not path.is_file():
effective = (
", ".join(spec.bottle_names) if spec.bottle_names else manifest.agent.bottle
)
die(
f"agent_provider.dockerfile for bottle "
f"'{effective}' not found: {path}"
)
@abstractmethod
def _resolve_plan(self,
spec: BottleSpec,
*,
manifest: Manifest,
slug: str,
resolved_env: ResolvedEnv,
agent_provision_plan: AgentProvisionPlan,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
supervise_plan: SupervisePlan | None,
stage_dir: Path) -> PlanT:
"""Backend-specific plan resolution: image/container names,
env-file, prompt-file, proxy plan, runtime detection. Called by
`prepare` after `_validate` succeeds. Instance name, image,
prompt file, Dockerfile path, and guest home all live on
`agent_provision_plan` — the source of truth."""
def prelaunch_checks(self, plan: PlanT) -> None:
"""Raise StaleImageError if any cached image used by this plan is stale.
No-op default; backends override to call the shared check_stale*
helpers on their image/artifact timestamps. Called by the CLI before
launch so the operator can be prompted outside the launch context."""
@contextmanager
def launch(self, plan: PlanT) -> Generator[Bottle, None, None]:
"""Template: build or load images, then delegate to _launch_impl."""
images = self._build_or_load_images(plan)
with self._launch_impl(plan, images) as bottle:
yield bottle
@abstractmethod
def _build_or_load_images(self, plan: PlanT) -> BottleImages:
"""Return the agent and sidecar image references (or artifact paths)
for this plan, building fresh images when the policy requires it."""
@abstractmethod
def _launch_impl(self, plan: PlanT, images: BottleImages) -> AbstractContextManager[Bottle]:
"""Bring up the bottle using pre-resolved images; yield a handle; tear down on exit."""
def provision(self, plan: PlanT, bottle: "Bottle") -> str | None:
"""Copy host-side files (CA cert, prompt, skills, .git) into
the running bottle. Called from `launch` after the container
/ machine is up. Returns the in-container prompt path if a
prompt was provisioned, else None — the Bottle handle uses it
to decide whether to add provider-specific prompt args to the
agent's argv.
Default orchestration: ca → prompt → provider apply → skills
→ workspace → git → supervise-mcp. CA install runs first so
the agent's trust store is rebuilt before anything inside the
agent makes a TLS call.
Per PRD 0050 the per-provider steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin. The backend only owns the
steps that are about backend infrastructure (CA, workspace,
git) and surfaces the supervise daemon URL its launch step
knows about via `supervise_mcp_url`.
PRD 0017: cred-proxy's agent-side dotfile rewrites (~/.npmrc,
~/.gitconfig insteadOf, tea config) are gone. Egress-proxy is
on the agent's HTTP_PROXY path so every tool that respects
HTTPS_PROXY (claude-code, git over HTTPS, npm, curl) is
intercepted without per-tool reconfiguration."""
provider = get_provider(plan.agent_provision.template)
provider.provision_ca(bottle, plan)
prompt_path = provider.provision_prompt(plan, bottle)
provider.provision(plan, bottle)
provider.provision_skills(plan, bottle)
self.provision_workspace(plan, bottle)
provider.provision_git(bottle, plan)
provider.provision_supervise_mcp(
plan, bottle, self.supervise_mcp_url(plan),
)
return prompt_path
def provision_workspace(self, plan: PlanT, bottle: "Bottle") -> None:
"""Copy the operator workspace into the running bottle.
This is the only supported workspace-provisioning path: Docker
does not build a derived image containing the current
workspace."""
workspace = plan.workspace_plan
if not (workspace.enabled and workspace.copy_contents):
return
guest_parent = workspace.guest_path.rsplit("/", 1)[0] or "/"
guest_path = shlex.quote(workspace.guest_path)
guest_parent = shlex.quote(guest_parent)
owner = shlex.quote(workspace.owner)
mode = shlex.quote(workspace.mode)
info(f"copying {workspace.host_path} -> {bottle.name}:{workspace.guest_path}")
bottle.exec(
f"rm -rf {guest_path} && mkdir -p {guest_parent}",
user="root",
)
bottle.cp_in(str(workspace.host_path), workspace.guest_path)
bottle.exec(
f"chown -R {owner} {guest_path} && chmod {mode} {guest_path}",
user="root",
)
def supervise_mcp_url(self, plan: PlanT) -> str:
"""Return the agent-side URL of the per-bottle supervise
gateway, or "" when this bottle has no gateway. The provider
plugin's `provision_supervise_mcp` uses it to register the
MCP entry inside the guest.
Default returns "" so backends without supervise support
don't have to implement it. Docker and firecracker override."""
del plan
return ""
def ensure_orchestrator(self) -> str:
"""Bring up this backend's per-host orchestrator + shared gateway
(idempotent) and return the host-reachable control-plane URL.
This is the backend-agnostic bring-up entry point: `launch` calls
it as part of starting a bottle, and operator tools (`supervise`)
call it to start the control plane on demand when none is running
yet. Docker starts the orchestrator + gateway containers;
firecracker boots the infra VM. Backends with no orchestrator
(macos-container) die with a pointer — the default here."""
die(f"backend {self.name!r} has no orchestrator control plane")
@abstractmethod
def prepare_cleanup(self) -> CleanupT:
"""Enumerate orphaned resources from previous bottles. No side
effects; safe to call before the y/N."""
@abstractmethod
def cleanup(self, plan: CleanupT) -> None:
"""Remove everything described by the cleanup plan."""
@abstractmethod
def enumerate_active(self) -> Sequence[ActiveAgent]:
"""Return every currently-running agent on this backend.
Empty when none. Backend-specific: docker queries `docker
compose ls`; firecracker cross-references its running gateway
containers against per-bottle metadata."""
@classmethod
@abstractmethod
def is_available(cls) -> bool:
"""Whether this backend's runtime prerequisites are satisfied
on the current host. Docker → `docker` on PATH; firecracker →
Linux + KVM. Used by the cross-backend
`enumerate_active_agents` / `cmd_cleanup` to skip backends
the operator hasn't installed, so a docker-only host
doesn't fail when `cli.py active` walks past
firecracker."""
@classmethod
@abstractmethod
def setup(cls) -> int:
"""Emit this backend's one-time host setup — privileged network
pool, daemon bring-up, install pointers, etc. — as
host-appropriate config or commands. Prints to stdout/stderr and
returns a shell exit code (0 = nothing to report / success). A
backend that needs no host setup prints a short note and returns
0. Invoked generically by `./cli.py backend setup [--backend=…]`
so operators can provision any backend without a
backend-specific command. Classmethod (like `is_available`) —
it's a host query, not per-bottle state."""
@classmethod
@abstractmethod
def status(cls, *, quiet: bool = False) -> int:
"""Report whether this backend's prerequisites are satisfied on
the host — binaries, daemon reachability, network pool, range
conflicts, etc. Returns BackendStatus.READY (0) when the backend
is ready to launch and non-zero when something is missing.
When quiet=False (default) prints a human-readable summary to
stderr. When quiet=True returns the status code silently —
useful for cheap programmatic checks.
Invoked by `./cli.py backend status [--backend=…]` (quiet=False)
and by is_backend_ready() (caller-controlled)."""
@classmethod
@abstractmethod
def teardown(cls) -> int:
"""Undo `setup()` — the inverse operation, surfaced as
`./cli.py backend teardown [--backend=…]` (uninstall). Symmetric
with setup: where setup is advisory (prints the privileged
commands / declarative config to apply), teardown prints the
commands / config change to remove the host prerequisites. A
backend with no host setup prints a short note and returns 0.
Not called by the launch path or the test suite."""
-60
View File
@@ -1,60 +0,0 @@
"""Shared helpers for the consolidated launch sequence (PRD 0070).
Logic that was duplicated across the docker, macos_container, and
firecracker consolidated_launch modules — extracted so each backend
imports it rather than re-implementing it.
"""
from __future__ import annotations
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..orchestrator.client import OrchestratorClient
from ..orchestrator.registration import registration_inputs
from .docker.gateway_provision import GatewayTransport, deprovision_git_gate, provision_git_gate
def provision_bottle(
client: OrchestratorClient,
source_ip: str,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
transport: GatewayTransport,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
):
"""Register the bottle and provision its git-gate state. Rolls back the
registration if provisioning fails so no orphan is left. Returns the
`RegisteredBottle` from the orchestrator."""
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(transport, reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
return reg
def teardown_consolidated(
bottle_id: str,
transport: GatewayTransport,
*,
orchestrator_url: str,
timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state. Both steps are
idempotent so this is safe from a cleanup trap."""
from ..orchestrator.config_store import DEFAULT_TEARDOWN_TIMEOUT_SECONDS
OrchestratorClient(
orchestrator_url,
timeout=timeout if timeout is not None else DEFAULT_TEARDOWN_TIMEOUT_SECONDS,
).teardown_bottle(bottle_id)
deprovision_git_gate(transport, bottle_id)
__all__ = ["provision_bottle", "teardown_consolidated"]
+31 -7
View File
@@ -11,18 +11,42 @@ The bulk of the implementation lives in sibling modules:
- launch: bring-up + teardown context manager
- cleanup: orphan enumeration, removal, active listing
- backend: DockerBottleBackend façade wiring the above
- infra: DockerInfraService (the per-host orchestrator + gateway pair)
This file only re-exports the public names so
`from bot_bottle.backend.docker import DockerBottleBackend` keeps
working.
Thin by design: the public names are re-exported lazily via `__getattr__`, so
importing a leaf like `backend.docker.util` doesn't drag `DockerBottleBackend`
(and the whole framework it pulls) into memory.
"""
from __future__ import annotations
from .backend import DockerBottleBackend
from .bottle import DockerBottle
from .bottle_cleanup_plan import DockerBottleCleanupPlan
from .bottle_plan import DockerBottlePlan
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .backend import DockerBottleBackend
from .bottle import DockerBottle
from .bottle_cleanup_plan import DockerBottleCleanupPlan
from .bottle_plan import DockerBottlePlan
_LAZY_MODULES: dict[str, str] = {
"DockerBottleBackend": "backend",
"DockerBottle": "bottle",
"DockerBottleCleanupPlan": "bottle_cleanup_plan",
"DockerBottlePlan": "bottle_plan",
}
def __getattr__(name: str) -> Any:
mod = _LAZY_MODULES.get(name)
if mod is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from importlib import import_module
value = getattr(import_module(f"{__name__}.{mod}"), name)
globals()[name] = value
return value
__all__ = [
"DockerBottle",
+10 -6
View File
@@ -20,16 +20,17 @@ infrastructure: CA install and git copy-in.
from __future__ import annotations
import shutil
from contextlib import contextmanager
import io
from contextlib import contextmanager, redirect_stderr
from pathlib import Path
from typing import Generator, Sequence
from ...supervise import SUPERVISE_HOSTNAME, SUPERVISE_PORT
from ...supervisor.types import SUPERVISE_HOSTNAME, SUPERVISE_PORT
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervise import SupervisePlan
from ...supervisor.plan import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from . import cleanup as _cleanup
@@ -60,8 +61,11 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
return _setup.setup()
@classmethod
def status(cls) -> int:
def status(cls, *, quiet: bool = False) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -112,8 +116,8 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
yield bottle
def ensure_orchestrator(self) -> str:
from ...orchestrator.lifecycle import OrchestratorService
return OrchestratorService().ensure_running()
from .infra import DockerInfraService
return DockerInfraService().ensure_running()
def supervise_mcp_url(self, plan: DockerBottlePlan) -> str:
"""Docker bottles reach the supervise daemon via the
+4
View File
@@ -39,6 +39,10 @@ class DockerBottlePlan(BottlePlan):
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
# Encryption key for the agent's stored egress secrets; injected into the
# agent container as ENV_VAR_SECRET via the compose subprocess env (bare
# name — value never written to the compose file). Empty pre-registration.
env_var_secret: str = ""
@property
def container_name(self) -> str:
@@ -16,7 +16,8 @@ from __future__ import annotations
from typing import Any
from ...egress import egress_agent_env_entries
from ...egress import Egress
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from .bottle_plan import DockerBottlePlan
from .egress import EGRESS_PORT
@@ -58,7 +59,11 @@ def consolidated_agent_compose(
# the secret value never lands on argv or in the compose file.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
# ENV_VAR_SECRET: bare name so the value comes from the compose subprocess
# env (set in launch.py) and is never written to the compose file on disk.
if getattr(plan, "env_var_secret", ""):
env.append(ENV_VAR_SECRET_NAME)
env.extend(Egress().agent_env_entries(plan.egress_plan))
service: dict[str, Any] = {
"image": plan.image,
@@ -2,7 +2,7 @@
Composes the orchestrator primitives into the register/teardown sequence:
1. ensure the single infra container (control plane + gateway) is up;
1. ensure the per-host pair (orchestrator + gateway containers) is up;
2. allocate the bottle a pinned source IP on the gateway network;
3. register it and provision its git-gate repos/creds into the gateway.
@@ -15,15 +15,17 @@ from __future__ import annotations
from dataclasses import dataclass
from ...docker_cmd import run_docker
from ... import log
from .util import run_docker
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.gateway import GATEWAY_NETWORK
from ...orchestrator.lifecycle import INFRA_NAME, OrchestratorService
from ..consolidated_util import provision_bottle
from ..consolidated_util import teardown_consolidated as _teardown_util
from .gateway_provision import DockerGatewayTransport
from ...gateway import GATEWAY_NETWORK
from .infra import INFRA_NAME, DockerInfraService
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ...orchestrator.reprovision import reprovision_bottles
from ..provision_bottle import deprovision_bottle, provision_bottle
from .gateway_transport import DockerGatewayTransport
from .gateway_net import next_free_ip
@@ -41,6 +43,7 @@ class LaunchContext:
network: str # the shared gateway network to attach to
gateway_ip: str # the gateway's address — the agent's proxy target
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def _network_cidr(network: str) -> str:
@@ -57,23 +60,9 @@ def _network_cidr(network: str) -> str:
return cidr
def _container_ip(name: str, network: str) -> str:
"""A container's IPv4 address on `network`, or raise."""
proc = run_docker([
"docker", "inspect", "--format",
f'{{{{(index .NetworkSettings.Networks "{network}").IPAddress}}}}', name,
])
ip = proc.stdout.strip()
if proc.returncode != 0 or not ip:
raise ConsolidatedLaunchError(
f"container {name} has no address on {network}: {proc.stderr.strip()}"
)
return ip
def _network_container_ips(network: str) -> list[str]:
"""Every address currently assigned on the gateway network — the ground
truth for "in use": the infra container and every live agent. Read from
truth for "in use": the gateway container and every live agent. Read from
the network so a new bottle can't collide with anything actually attached."""
proc = run_docker([
"docker", "network", "inspect", "--format",
@@ -85,27 +74,85 @@ def _network_container_ips(network: str) -> list[str]:
return ips
def _reprovision_running_bottles(
orchestrator_url: str,
network: str = GATEWAY_NETWORK,
infra_name: str = INFRA_NAME,
) -> None:
"""Re-inject egress tokens for any registered bottles that lost their
in-memory tokens (e.g., after an orchestrator restart).
For each registered bottle whose source IP maps to a live container on the
gateway network, reads ENV_VAR_SECRET via ``docker exec … printenv`` and
calls ``POST /bottles/<id>/reprovision_gateway``. Idempotent — a no-op
when the orchestrator already has all tokens loaded. Best-effort: a single
container exec failure never blocks a new bottle launch."""
client = OrchestratorClient(orchestrator_url)
# Build {source_ip: container_name} from live containers on the gateway
# network, excluding the gateway container itself.
try:
proc = run_docker([
"docker", "network", "inspect",
"--format", "{{range .Containers}}{{.Name}} {{.IPv4Address}}\n{{end}}",
network,
])
except OSError as exc:
log.info(f"egress token reprovision skipped: {exc}")
return
ip_to_container: dict[str, str] = {}
for line in proc.stdout.splitlines():
parts = line.strip().split()
if len(parts) >= 2 and parts[0] != infra_name:
ip = parts[1].split("/", 1)[0]
if ip:
ip_to_container[ip] = parts[0]
secrets_by_ip: dict[str, str] = {}
for source_ip, container_name in ip_to_container.items():
proc = run_docker(
["docker", "exec", container_name, "printenv", ENV_VAR_SECRET_NAME]
)
if proc.returncode == 0 and proc.stdout.strip():
secrets_by_ip[source_ip] = proc.stdout.strip()
reprovisioned = reprovision_bottles(client, secrets_by_ip)
if reprovisioned:
log.info(
"reprovisioned egress tokens",
context={"count": reprovisioned},
)
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
service: OrchestratorService | None = None,
service: DockerInfraService | None = None,
infra_name: str = INFRA_NAME,
network: str = GATEWAY_NETWORK,
) -> LaunchContext:
"""Ensure the infra container is up, allocate + register the bottle, and
provision its git-gate state. Returns the agent's attach context."""
service = service or OrchestratorService()
"""Ensure the orchestrator + gateway pair is up, allocate + register the bottle, and
provision its git-gate state. Returns the agent's attach context.
Also reprovisiones egress tokens for any already-running bottles that lost
their in-memory credentials (e.g. after an orchestrator restart), so
they regain egress access before the new bottle is registered."""
service = service or DockerInfraService()
url = service.ensure_running()
# Agents attribute against the *gateway* container (data plane), not the
# orchestrator — the two planes are now separate containers. Read its
# agent-facing address + provisioning transport off the Gateway service.
gateway = service.gateway()
_reprovision_running_bottles(url, network=network, infra_name=gateway.name)
client = OrchestratorClient(url)
cidr = _network_cidr(network)
gateway_ip = _container_ip(infra_name, network)
gateway_ip = gateway.address()
source_ip = next_free_ip(cidr, _network_container_ips(network))
transport = DockerGatewayTransport(infra_name)
transport = gateway.provisioning_transport()
reg = provision_bottle(
client, source_ip, egress_plan, git_gate_plan, transport,
image_ref=image_ref, tokens=tokens,
@@ -117,21 +164,22 @@ def launch_consolidated(
network=network,
gateway_ip=gateway_ip,
orchestrator_url=url,
env_var_secret=reg.env_var_secret,
)
def teardown_consolidated(
def deprovision_consolidated(
bottle_id: str, *, orchestrator_url: str, infra_name: str = INFRA_NAME,
timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state. Idempotent."""
_teardown_util(bottle_id, DockerGatewayTransport(infra_name),
deprovision_bottle(bottle_id, DockerGatewayTransport(infra_name),
orchestrator_url=orchestrator_url, timeout=timeout)
__all__ = [
"LaunchContext",
"launch_consolidated",
"teardown_consolidated",
"deprovision_consolidated",
"ConsolidatedLaunchError",
]
+2 -2
View File
@@ -1,6 +1,6 @@
"""Active-agent enumeration for the docker backend.
Returns `ActiveAgent` records the CLI `list active` command and the
Returns `ActiveAgent` records the CLI `active` command and the
dashboard agents pane consume. Empty when docker isn't reachable
— gated by `has_backend('docker')` at the cross-backend caller
so this module trusts that docker is available when called.
@@ -60,7 +60,7 @@ def _parse_services_by_project(stdout: str) -> dict[str, set[str]]:
def _query_services_by_project() -> dict[str, set[str]]:
"""One `docker ps` call → `{project: {service, ...}}`. Used
by the CLI's `list active` and the dashboard's agents pane —
by the CLI's `active` and the dashboard's agents pane —
one subprocess per refresh tick, not one per bottle."""
try:
r = subprocess.run(
@@ -1,136 +1,20 @@
"""The consolidated per-host gateway (PRD 0070).
The core consolidation win: **one** persistent gateway per host, shared by
every bottle, instead of a gateway per bottle. It's safe to share
because the attribution invariant (source IP + identity token, see
`registry`) lets the gateway attribute each request to the right bottle
so per-bottle policy lives in one long-lived process keyed on who's calling.
`Gateway` is the backend-neutral lifecycle contract (mirrors `LaunchBroker`):
ensure the single instance is up, report it, tear it down. `DockerGateway`
is the docker implementation; a firecracker gateway VM slots in later.
The defining behaviour is **idempotent singleton**: `ensure_running` starts
the instance if absent and is a no-op if it's already up, so N bottle
launches never spawn N gateways.
"""
from __future__ import annotations
import abc
import os
import time
from pathlib import Path
from ..docker_cmd import run_docker
from ..paths import (
CONTROL_PLANE_TOKEN_ENV,
host_control_plane_token,
host_db_path,
from .util import run_docker
from .gateway_transport import DockerGatewayTransport
from ...paths import (
ORCHESTRATOR_AUTH_JWT_ENV,
host_gateway_ca_dir,
)
from ..supervise import DB_PATH_IN_CONTAINER
# The host DB dir is bind-mounted here so the gateway's supervise daemon
# writes its queued proposals into the ONE host DB (the same file the
# orchestrator container opens and the operator reaches over HTTP).
_SUPERVISE_DB_DIR_IN_CONTAINER = os.path.dirname(DB_PATH_IN_CONTAINER)
# The gateway's mitmproxy writes its CA a beat after the container starts, so
# reads poll for it rather than assuming it's there on a fresh launch.
_CA_POLL_SECONDS = 0.5
DEFAULT_CA_TIMEOUT_SECONDS = 30.0
GATEWAY_NAME = "bot-bottle-orch-gateway"
GATEWAY_LABEL = "bot-bottle-orch-gateway=1"
# The single user-defined network the gateway and every agent bottle share.
# Agents attach here with a pinned IP and reach the gateway's egress /
# git-http / supervise ports by its address — no host port publishing, and
# the source IP the gateway attributes by is the address on this network.
GATEWAY_NETWORK = "bot-bottle-gateway"
# mitmproxy's CA dir in the bundle. The host's gateway-CA dir (see
# `host_gateway_ca_dir`) is bind-mounted here so the gateway's self-generated
# CA stays STABLE across container recreation — every agent installs this one
# CA to trust the shared gateway's TLS interception, so it must not rotate when
# the gateway restarts. A host bind-mount rather than a named volume: a named
# volume is silently wiped by `docker volume prune`, minting a fresh CA that
# breaks every running bottle (issue #450).
MITMPROXY_HOME = "/home/mitmproxy/.mitmproxy"
GATEWAY_CA_CERT = f"{MITMPROXY_HOME}/mitmproxy-ca-cert.pem"
# The CA material mitmproxy writes into its confdir. mitmproxy reuses these on
# startup when present and generates them only on first run, so persisting them
# is what makes the CA stable; deleting them (see `rotate_gateway_ca`) forces a
# fresh CA on the next start. `mitmproxy-ca.pem` (cert + private key) is the
# signing identity; the rest are derived encodings agents/clients consume.
GATEWAY_CA_GLOB = "mitmproxy-ca*"
# The gateway data-plane image + its Dockerfile. Kept as a local constant
# rather than imported from the backend layer, which would drag
# the whole backend layer into the lean orchestrator (see #359); unify when
# that lands. Env override matches the backend's BOT_BOTTLE_GATEWAY_IMAGE.
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
GATEWAY_DOCKERFILE = "Dockerfile.gateway"
_REPO_ROOT = Path(__file__).resolve().parents[2]
def _host_db_dir() -> str:
"""The host DB directory (created if missing), for the gateway's
supervise-DB bind-mount."""
db_dir = host_db_path().parent
db_dir.mkdir(parents=True, exist_ok=True)
return str(db_dir)
def rotate_gateway_ca(ca_dir: Path | None = None) -> list[Path]:
"""Delete the persisted mitmproxy CA so the next gateway start mints a
fresh one the explicit, deliberate CA-rollover path (issue #450).
Persistence keeps the CA stable across restarts precisely because mitmproxy
reuses the on-disk CA; rotation is therefore just removing that material.
Returns the files removed (empty when there was no CA yet); idempotent.
This only clears the on-disk CA. It does NOT stop the running gateway (whose
mitmproxy still holds the old CA in memory) or re-provision agents the
caller recreates the gateway to mint the new CA and re-attaches bottles.
`rotate-ca` on the orchestrator CLI wires those steps together."""
ca_dir = ca_dir if ca_dir is not None else host_gateway_ca_dir()
removed: list[Path] = []
for path in sorted(ca_dir.glob(GATEWAY_CA_GLOB)):
path.unlink()
removed.append(path)
return removed
class GatewayError(Exception):
"""The shared gateway failed to build/start/stop (non-zero `docker` exit)."""
class Gateway(abc.ABC):
"""Lifecycle of the single per-host gateway. Backend-neutral."""
name: str
def ensure_built(self) -> None:
"""Ensure the gateway's image / rootfs exists, building it if needed.
Default: nothing to build (e.g. a stub or a pre-pulled image)."""
return
@abc.abstractmethod
def ensure_running(self) -> None:
"""Start the gateway if it isn't already up. Idempotent: a no-op
when it's already running (that's the whole point one per host).
Assumes the image exists call `ensure_built()` first."""
@abc.abstractmethod
def is_running(self) -> bool:
"""True iff the gateway instance is currently up."""
@abc.abstractmethod
def stop(self) -> None:
"""Remove the gateway. Idempotent — absent is success."""
from ...gateway import (
Gateway, GatewayTransport, GATEWAY_IMAGE, GATEWAY_NAME, GATEWAY_NETWORK,
GATEWAY_DOCKERFILE, REPO_ROOT, GATEWAY_LABEL, MITMPROXY_HOME,
DEFAULT_CA_TIMEOUT_SECONDS, CA_POLL_SECONDS, GATEWAY_CA_CERT, GatewayError
)
class DockerGateway(Gateway):
"""The consolidated gateway as a single, fixed-name Docker container.
@@ -146,7 +30,7 @@ class DockerGateway(Gateway):
*,
name: str = GATEWAY_NAME,
network: str = GATEWAY_NETWORK,
orchestrator_url: str = "",
control_network: str = "",
build_context: Path | None = None,
dockerfile: str | None = GATEWAY_DOCKERFILE,
host_port_bindings: tuple[int, ...] = (),
@@ -154,14 +38,19 @@ class DockerGateway(Gateway):
self.image_ref = image_ref
self.name = name
self.network = network
# The control-plane URL the gateway's data plane resolves per bottle
# against — reached by container name over docker DNS on the shared
# network (container↔container, no host firewall). Mandatory to *run*
# the gateway (see `ensure_running`); empty is tolerated only for the
# construct-then-read-CA path (`ca_cert_pem` on an already-running
# container), which never launches a container.
self._orchestrator_url = orchestrator_url
self._build_context = build_context or _REPO_ROOT
# When set, the gateway is dual-homed: it also joins this `--internal`
# control network (shared only with the orchestrator) so it can resolve
# `orchestrator_url` by container name over docker DNS. Agents are never
# on it, so they get no route to the control plane (PRD 0070).
self._control_network = control_network
# The orchestrator binding — set by `connect_to_orchestrator` (the
# control-plane URL the data plane resolves against, and the pre-minted
# `gateway` token it presents; the gateway never mints, so it never
# holds the signing key — #469). Empty until connected; `ca_cert_pem` /
# `address` / `stop` work on an already-running gateway without it.
self._orchestrator_url = ""
self._gateway_token = ""
self._build_context = build_context or REPO_ROOT
self._dockerfile = dockerfile
# Ports published on the host (0.0.0.0). Used by the Firecracker
# backend's dev-harness gateway so VMs can reach it via their TAP link;
@@ -225,7 +114,13 @@ class DockerGateway(Gateway):
f"gateway network {self.network} failed to create: {proc.stderr.strip()}"
)
def ensure_running(self) -> None:
def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
# Bind to this orchestrator (PRD 0070): stash the URL its daemons resolve
# policy against + the pre-minted `gateway` token they present, then bring
# the container up. The gateway never mints, so it holds no signing key —
# only this token (#469).
self._orchestrator_url = orchestrator_url
self._gateway_token = gateway_token
# Fail closed on a missing policy source. The data-plane daemons are
# resolver-only now (PRD 0070) — without an orchestrator URL egress
# raises, git-http exits 1, and supervise exits 2 — so launching a
@@ -236,6 +131,11 @@ class DockerGateway(Gateway):
"gateway requires an orchestrator URL to run "
"(resolver-only data plane; no single-tenant fallback)"
)
if not self._gateway_token:
raise GatewayError(
"gateway requires a pre-minted `gateway` token to run "
"(the orchestrator mints it; the gateway never holds the key)"
)
# Recreate when the running container's image is stale (a rebuild),
# so source changes to the gateway's flat daemons take effect — not
# just when the container is absent.
@@ -255,11 +155,10 @@ class DockerGateway(Gateway):
# container recreation AND docker volume pruning (agents trust it)
# — see host_gateway_ca_dir / issue #450.
"--volume", f"{host_gateway_ca_dir()}:{MITMPROXY_HOME}",
# Share the one host DB: the supervise daemon queues proposals
# into the same file the orchestrator (and the operator, over
# HTTP) reads — no second, disconnected DB in the container.
"--volume", f"{_host_db_dir()}:{_SUPERVISE_DB_DIR_IN_CONTAINER}",
"--env", f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
# No DB mount: the data plane (egress / supervise / git-gate) reaches
# the supervise queue over the control-plane RPC and never opens
# bot-bottle.db, so the gateway container gets no file handle on it
# (PRD 0070 / issue #469).
]
for port in self._host_port_bindings:
argv += ["--publish", f"0.0.0.0:{port}:{port}"]
@@ -268,15 +167,42 @@ class DockerGateway(Gateway):
# policy against the control plane per request (guaranteed non-empty by
# the check above).
argv += ["--env", f"BOT_BOTTLE_ORCHESTRATOR_URL={self._orchestrator_url}"]
# ...and present the control-plane secret on those /resolve calls (the
# control plane requires it). Bare `--env NAME` keeps the value off argv
# / `docker inspect`; only the gateway (not the agent) is given it.
argv += ["--env", CONTROL_PLANE_TOKEN_ENV]
run_env[CONTROL_PLANE_TOKEN_ENV] = host_control_plane_token()
# ...presenting a role-scoped `gateway` token (a signed JWT minted from
# the host signing key) on those calls. The gateway never receives the
# signing key — only this pre-minted token, which it cannot rewrite into
# a `cli` token, so a compromised data-plane process can't drive the
# operator routes (issue #469 review). Bare `--env NAME` keeps the value
# off argv / `docker inspect`; only the gateway (not the agent) is given it.
argv += ["--env", ORCHESTRATOR_AUTH_JWT_ENV]
run_env[ORCHESTRATOR_AUTH_JWT_ENV] = self._gateway_token
argv.append(self.image_ref)
proc = run_docker(argv, env=run_env)
if proc.returncode != 0:
raise GatewayError(f"gateway failed to start: {proc.stderr.strip()}")
# Dual-home onto the control network so the daemons resolve the
# orchestrator by name. Docker attaches only one network at `run`, so
# the second is a `network connect` — the daemons tolerate the brief
# pre-connect window (they retry /resolve per request).
if self._control_network:
self._connect_control_network()
def _connect_control_network(self) -> None:
"""Ensure the `--internal` control network exists and attach the gateway
to it (idempotent an already-connected container is a tolerated
no-op)."""
if run_docker(["docker", "network", "inspect", self._control_network]).returncode != 0:
proc = run_docker(["docker", "network", "create", "--internal", self._control_network])
if proc.returncode != 0 and "already exists" not in proc.stderr:
raise GatewayError(
f"control network {self._control_network} failed to create: "
f"{proc.stderr.strip()}"
)
proc = run_docker(["docker", "network", "connect", self._control_network, self.name])
if proc.returncode != 0 and "already exists" not in proc.stderr:
raise GatewayError(
f"gateway failed to join control network {self._control_network}: "
f"{proc.stderr.strip()}"
)
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's CA certificate (PEM) that agents install to trust its
@@ -294,16 +220,33 @@ class DockerGateway(Gateway):
f"gateway CA cert not available after {timeout:g}s: "
f"{proc.stderr.strip() or 'empty'}"
)
time.sleep(_CA_POLL_SECONDS)
time.sleep(CA_POLL_SECONDS)
def stop(self) -> None:
proc = run_docker(["docker", "rm", "--force", self.name])
if proc.returncode != 0 and "No such container" not in proc.stderr:
raise GatewayError(f"gateway failed to stop: {proc.stderr.strip()}")
def address(self) -> str:
"""The gateway's IPv4 on the agent-facing network (`self.network`) — the
proxy target agents dial for egress / git-http / supervise. This is
*not* the control network: agents share only this network with the
gateway, so its address here is also the source IP the gateway
attributes each request by."""
proc = run_docker([
"docker", "inspect", "--format",
f'{{{{(index .NetworkSettings.Networks "{self.network}").IPAddress}}}}',
self.name,
])
ip = proc.stdout.strip()
if proc.returncode != 0 or not ip:
raise GatewayError(
f"gateway {self.name} has no address on {self.network}: "
f"{proc.stderr.strip()}"
)
return ip
__all__ = [
"Gateway", "DockerGateway", "GatewayError", "rotate_gateway_ca",
"GATEWAY_NAME", "GATEWAY_LABEL", "GATEWAY_IMAGE", "GATEWAY_NETWORK",
"GATEWAY_CA_CERT", "GATEWAY_CA_GLOB",
]
def provisioning_transport(self) -> GatewayTransport:
"""The exec/cp transport git-gate provisioning stages per-bottle repos +
deploy keys through (over the docker socket)."""
return DockerGatewayTransport(self.name)
@@ -0,0 +1,35 @@
"""The `GatewayTransport` for the docker gateway container (PRD 0070).
How the launcher stages files + runs commands in the running gateway container:
`docker exec` / `docker cp` over the docker socket. The backend-neutral
provisioning logic that drives it lives in `backend.provision_gateway`.
"""
from __future__ import annotations
from .util import run_docker
from ...gateway import GatewayProvisionError
class DockerGatewayTransport:
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
def __init__(self, gateway: str) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
proc = run_docker(["docker", "exec", self.gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def cp_into(self, src: str, dest: str) -> None:
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
__all__ = ["DockerGatewayTransport"]
+152
View File
@@ -0,0 +1,152 @@
"""The per-host control plane + gateway for the docker backend (PRD 0070).
Runs the orchestrator (control plane) and the gateway (data plane) as **two
separate containers**, split now that #469 got the DB off the data plane:
* `bot-bottle-orchestrator` — the lean control-plane container
(`DockerOrchestrator`). Joins the `bot-bottle-orchestrator` **control
network** (`--internal`) only, plus a host-loopback publish for the CLI.
Sole opener of `bot-bottle.db`; holds the signing key.
* `bot-bottle-gateway` — the data-plane container (`DockerGateway`).
**Dual-homed** on the agent-facing `bot-bottle-gateway` network *and* the
control network, so it reaches the orchestrator by name over docker DNS
(`http://bot-bottle-orchestrator:8099`) while agents — which are never on
the control network — cannot reach the control plane at all (the L3 block
Firecracker's nft already has). Holds the `gateway` JWT + the mitmproxy CA.
`DockerInfraService` composes the two services (`orchestrator()` + `gateway()`)
and brings them up as an idempotent per-host pair. Callers use `ensure_running()`
(returns the host control-plane URL) + `gateway_name` (the container the launch
flow attributes agents against).
"""
from __future__ import annotations
from pathlib import Path
from .util import run_docker
from .gateway import DockerGateway
from .orchestrator import (
DockerOrchestrator,
ORCHESTRATOR_IMAGE,
ORCHESTRATOR_LABEL,
ORCHESTRATOR_NAME,
ORCHESTRATOR_NETWORK,
)
from ...paths import bot_bottle_root
from ...gateway import (
GATEWAY_IMAGE,
GATEWAY_NAME,
GATEWAY_NETWORK,
)
from ..infra_service import InfraService
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
)
# Back-compat aliases: the split retired the combined `bot-bottle-infra`
# container, but the fixed-name constants some callers/tests import still map to
# the pair's public identity.
INFRA_NAME = GATEWAY_NAME # the container agents attribute against is the gateway
_REPO_ROOT = Path(__file__).resolve().parents[3]
class DockerInfraService(InfraService):
"""Composes the per-host control plane + gateway as two containers.
`orchestrator_name` / `gateway_name` let callers run independent pairs on
one host without name collisions (e.g. isolated integration tests that
can't share the production singletons)."""
def __init__(
self,
*,
port: int = DEFAULT_PORT,
network: str = GATEWAY_NETWORK,
control_network: str = ORCHESTRATOR_NETWORK,
orchestrator_image: str = ORCHESTRATOR_IMAGE,
gateway_image: str = GATEWAY_IMAGE,
repo_root: Path = _REPO_ROOT,
host_root: Path | None = None,
orchestrator_name: str = ORCHESTRATOR_NAME,
orchestrator_label: str = ORCHESTRATOR_LABEL,
gateway_name: str = GATEWAY_NAME,
) -> None:
self.port = port
self.network = network
self.control_network = control_network
self.orchestrator_image = orchestrator_image
self.gateway_image = gateway_image
self._repo_root = repo_root
self._host_root = host_root or bot_bottle_root()
self._orchestrator_name = orchestrator_name
self._orchestrator_label = orchestrator_label
self._gateway_name = gateway_name
def orchestrator(self) -> DockerOrchestrator:
"""The control-plane service. Cheap to reconstruct — `ensure_built`
builds its image, `ensure_running` brings it up, and the launch flow
reads its `url()` / `gateway_url()` / `mint_gateway_token()` off it."""
return DockerOrchestrator(
self.orchestrator_image,
name=self._orchestrator_name,
label=self._orchestrator_label,
port=self.port,
control_network=self.control_network,
repo_root=self._repo_root,
host_root=self._host_root,
)
def gateway(self) -> DockerGateway:
"""The data-plane gateway, dual-homed on the agent network + the control
network, resolving policy against the orchestrator by name. Cheap to
reconstruct — the launch flow reads its `address()` / provisioning
transport off it, and `ensure_running` connects it to the control
plane."""
return DockerGateway(
self.gateway_image,
name=self._gateway_name,
network=self.network,
control_network=self.control_network,
build_context=self._repo_root,
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Ensure the orchestrator + gateway containers are up; return the host
control-plane URL. Idempotent — a healthy orchestrator on current source
(and a current-image gateway) is left untouched. Raises
`OrchestratorStartError` on control-plane startup timeout."""
orchestrator = self.orchestrator()
gateway = self.gateway()
# Build both images (cache-aware; a no-op when nothing changed) before
# bringing either plane up.
orchestrator.ensure_built()
gateway.ensure_built()
orchestrator.ensure_running(startup_timeout=startup_timeout)
# Bring up (or refresh) the gateway once the control plane it resolves
# against is healthy. The orchestrator (which holds the signing key)
# mints the role-scoped `gateway` JWT here and hands it to the gateway,
# which never sees the key (#469). `connect_to_orchestrator` is
# idempotent.
gateway.connect_to_orchestrator(
orchestrator.gateway_url(), orchestrator.mint_gateway_token(),
)
return orchestrator.url()
def stop(self) -> None:
"""Remove both containers (idempotent)."""
run_docker(["docker", "rm", "--force", self._gateway_name])
run_docker(["docker", "rm", "--force", self._orchestrator_name])
__all__ = [
"DockerInfraService",
"ORCHESTRATOR_NAME",
"INFRA_NAME",
]
+20 -14
View File
@@ -37,11 +37,8 @@ from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...egress import egress_resolve_token_values
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...egress import Egress
from ...git_gate import GitGate
from ...image_cache import check_stale
from ...log import die, info, warn
from .. import BottleImages
@@ -64,9 +61,10 @@ from .compose import (
write_compose_file,
)
from .consolidated_compose import consolidated_agent_compose
from ...orchestrator.config_store import resolve_teardown_timeout
from .consolidated_launch import launch_consolidated, teardown_consolidated
from ...orchestrator.gateway import DockerGateway
from ...orchestrator.store.config_store import resolve_teardown_timeout
from .consolidated_launch import launch_consolidated, deprovision_consolidated
from .infra import INFRA_NAME
from .gateway import DockerGateway
# Where the repo root lives, for `docker build` context. Computed once.
@@ -125,7 +123,7 @@ def launch(
f"teardown failed for container {plan.container_name}"
f" (compose-down): {exc!r}"
)
revoke_git_gate_provisioned_keys(
GitGate().revoke_provisioned_keys(
_bottle_for_revoke, _git_gate_dir_for_revoke
)
@@ -134,7 +132,7 @@ def launch(
# provisioning the bottle's repos into the shared gateway.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
git_gate_plan = GitGate().provision_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
@@ -145,7 +143,7 @@ def launch(
# handed to the orchestrator (in memory) for the gateway to inject —
# the agent never sees them.
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
token_values = Egress().resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
@@ -153,17 +151,19 @@ def launch(
plan.egress_plan, git_gate_plan, image_ref=plan.image, tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
deprovision_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
# Step 4: install the SHARED gateway CA into the agent (replaces the
# per-bottle CA) — read it out of the running gateway.
# per-bottle CA) — read it out of the running gateway container
# (INFRA_NAME now aliases the gateway container name; the orchestrator,
# split into its own container, holds no CA).
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(DockerGateway(network=ctx.network).ca_cert_pem())
ca_file.write_text(DockerGateway(name=INFRA_NAME).ca_cert_pem())
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=ca_file,
@@ -186,6 +186,7 @@ def launch(
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
identity_token=ctx.identity_token,
env_var_secret=ctx.env_var_secret,
)
# Step 5: render + up the agent-only compose, pinned on the shared
@@ -198,7 +199,12 @@ def launch(
project = compose_project_name(plan.slug)
# Forwarded vars (OAuth token, host interpolations) flow through the
# subprocess env as bare names so values never land in the file.
# ENV_VAR_SECRET follows the same pattern: bare name in the compose
# spec, value only in the subprocess env so it is never written to disk.
compose_env: dict[str, str] = {**os.environ, **plan.forwarded_env}
if plan.env_var_secret:
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
compose_env[ENV_VAR_SECRET_NAME] = plan.env_var_secret
info(
f"docker compose up -d (project {project}, agent on shared "
f"gateway {ctx.gateway_ip}, ip {ctx.source_ip})"
+220
View File
@@ -0,0 +1,220 @@
"""The docker orchestrator (control plane) as a single, fixed-name container
(PRD 0070).
`DockerOrchestrator` is the docker implementation of the backend-neutral
`Orchestrator` service. The lean control-plane container joins the `--internal`
control network only (agents are never on it, so they have no L3 route to it)
plus a host-loopback publish for the CLI. It is the sole opener of
`bot-bottle.db` and the sole holder of the signing key (#469).
"""
from __future__ import annotations
import os
import time
from pathlib import Path
from ... import log
from .util import run_docker
from ...paths import (
ORCHESTRATOR_TOKEN_ENV,
bot_bottle_root,
host_orchestrator_token,
)
from ...gateway import GatewayError
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
Orchestrator,
OrchestratorStartError,
source_hash,
)
# The control plane's own container + the dedicated `--internal` control network
# the gateway reaches it over. The orchestrator + gateway join it, agents never
# do, so agents have no route to the control plane (PRD 0070 "Separating the
# planes"). Same name across docker + macOS.
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
ORCHESTRATOR_LABEL = "bot-bottle-orchestrator=1"
ORCHESTRATOR_NETWORK = "bot-bottle-orchestrator"
ORCHESTRATOR_IMAGE = os.environ.get(
"BOT_BOTTLE_ORCHESTRATOR_IMAGE", "bot-bottle-orchestrator:latest"
)
ORCHESTRATOR_DOCKERFILE = "Dockerfile.orchestrator"
# Baked as a container label so `ensure_running` can detect whether the running
# orchestrator is executing the current bind-mounted source.
ORCHESTRATOR_SOURCE_HASH_LABEL = "bot-bottle-orchestrator-source-hash"
# The bind-mount path for the live control-plane source inside the container.
# PYTHONPATH points here so a code change takes effect on the next launch
# without an image rebuild.
_SRC_IN_CONTAINER = "/bot-bottle-src"
# Bot-bottle host-root bind-mount (DB + state) inside the orchestrator. The
# control plane opens bot-bottle.db under here (via BOT_BOTTLE_ROOT ->
# host_db_path()); it is the ONLY container with a handle on it (issue #469).
_ROOT_IN_CONTAINER = "/bot-bottle-root"
_HEALTH_POLL_SECONDS = 0.25
_REPO_ROOT = Path(__file__).resolve().parents[3]
class DockerOrchestrator(Orchestrator):
"""The control plane as a single fixed-name container. `ensure_built` builds
it from `Dockerfile.orchestrator`; `ensure_running` starts it on the control
network + host loopback and blocks until `/health` answers."""
def __init__(
self,
image_ref: str = ORCHESTRATOR_IMAGE,
*,
name: str = ORCHESTRATOR_NAME,
label: str = ORCHESTRATOR_LABEL,
port: int = DEFAULT_PORT,
control_network: str = ORCHESTRATOR_NETWORK,
repo_root: Path = _REPO_ROOT,
host_root: Path | None = None,
dockerfile: str | None = ORCHESTRATOR_DOCKERFILE,
) -> None:
self.image_ref = image_ref
self.name = name
self.label = label
self.port = port
self.control_network = control_network
self._repo_root = repo_root
self._host_root = host_root or bot_bottle_root()
self._dockerfile = dockerfile
def url(self) -> str:
"""Host-side control-plane URL — the orchestrator's published loopback,
which the CLI reaches."""
return f"http://127.0.0.1:{self.port}"
def gateway_url(self) -> str:
"""The URL the gateway's data plane resolves policy against — the
orchestrator container by name on the control network (docker DNS,
container↔container, no host firewall)."""
return f"http://{self.name}:{DEFAULT_PORT}"
def ensure_built(self) -> None:
"""Build the control-plane image from its Dockerfile, cache-aware (a
no-op when nothing changed). No-op when no dockerfile is configured (a
pre-pulled image). BOT_BOTTLE_NO_CACHE forces a full rebuild."""
if self._dockerfile is None:
return
argv = ["docker", "build", "-t", self.image_ref,
"-f", str(self._repo_root / self._dockerfile),
str(self._repo_root)]
if os.environ.get("BOT_BOTTLE_NO_CACHE"):
argv.insert(2, "--no-cache")
proc = run_docker(argv)
if proc.returncode != 0:
raise GatewayError(
f"{self._dockerfile} build failed: {proc.stderr.strip()}")
def is_running(self) -> bool:
proc = run_docker([
"docker", "ps", "--filter", f"name=^/{self.name}$", "--format", "{{.Names}}",
])
return self.name in proc.stdout.split()
def _source_current(self, current_hash: str) -> bool:
"""True iff the running orchestrator was started from the current
bind-mounted source."""
if not self.is_running():
return False
proc = run_docker([
"docker", "inspect", "--format",
"{{ index .Config.Labels \"" + ORCHESTRATOR_SOURCE_HASH_LABEL + "\" }}",
self.name,
])
if proc.returncode != 0:
return True # can't compare → don't churn a working container
return proc.stdout.strip() == current_hash
def _ensure_control_network(self) -> None:
if run_docker(["docker", "network", "inspect", self.control_network]).returncode == 0:
return
proc = run_docker(["docker", "network", "create", "--internal", self.control_network])
if proc.returncode != 0 and "already exists" not in proc.stderr:
raise GatewayError(
f"control network {self.control_network} failed to create: "
f"{proc.stderr.strip()}"
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> None:
"""Ensure the control-plane container is up on current source; block
until healthy. Idempotent — a healthy orchestrator on current source is
left untouched (its in-memory egress tokens survive). Raises
`OrchestratorStartError` on startup timeout."""
current_hash = source_hash(self._repo_root)
if self.is_healthy() and self._source_current(current_hash):
return
log.info("starting orchestrator container", context={"name": self.name})
self._run_container(current_hash)
deadline = time.monotonic() + startup_timeout
while time.monotonic() < deadline:
if self.is_healthy():
log.info("orchestrator healthy", context={"url": self.url()})
return
time.sleep(_HEALTH_POLL_SECONDS)
raise OrchestratorStartError(
f"orchestrator at {self.url()} did not become healthy "
f"within {startup_timeout:g}s"
)
def _run_container(self, current_hash: str) -> None:
"""Start the lean control-plane container on the control network only,
published to host loopback for the CLI. Idempotent (clears a stale
fixed-name container first)."""
self._ensure_control_network()
run_docker(["docker", "rm", "--force", self.name])
_signing_key = host_orchestrator_token()
proc = run_docker([
"docker", "run", "--detach",
"--name", self.name,
"--label", self.label,
"--label", f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={current_hash}",
# Control network only — agents are never on it, so they have no
# route to the control plane (the L3 block, not just the JWT).
"--network", self.control_network,
# Host CLI reaches the control plane here (loopback only). The
# orchestrator listens on the fixed DEFAULT_PORT inside the
# container; self.port is the host-side published port.
"--publish", f"127.0.0.1:{self.port}:{DEFAULT_PORT}",
# Live control-plane source (code changes without an image rebuild).
"--volume", f"{self._repo_root}:{_SRC_IN_CONTAINER}:ro",
"--env", f"PYTHONPATH={_SRC_IN_CONTAINER}",
# Orchestrator registry DB on the host (sole writer: control plane).
"--volume", f"{self._host_root}:{_ROOT_IN_CONTAINER}",
"--env", f"BOT_BOTTLE_ROOT={_ROOT_IN_CONTAINER}",
# The signing key — held ONLY by the orchestrator (it verifies
# tokens); the gateway gets the pre-minted `gateway` JWT, never the
# key (issue #469). Bare `--env NAME` keeps the value off argv.
"--env", ORCHESTRATOR_TOKEN_ENV,
self.image_ref,
# Dockerfile.orchestrator ENTRYPOINT is `python3 -m bot_bottle.orchestrator`;
# these are its args.
"--host", "0.0.0.0", "--port", str(DEFAULT_PORT), "--broker", "stub",
], env={**os.environ, ORCHESTRATOR_TOKEN_ENV: _signing_key})
if proc.returncode != 0:
raise OrchestratorStartError(
f"orchestrator container failed to start: {proc.stderr.strip()}"
)
def stop(self) -> None:
"""Remove the control-plane container (idempotent)."""
run_docker(["docker", "rm", "--force", self.name])
__all__ = [
"DockerOrchestrator",
"ORCHESTRATOR_NAME",
"ORCHESTRATOR_LABEL",
"ORCHESTRATOR_NETWORK",
"ORCHESTRATOR_IMAGE",
"ORCHESTRATOR_DOCKERFILE",
"ORCHESTRATOR_SOURCE_HASH_LABEL",
]
@@ -1,12 +0,0 @@
"""Backend-infrastructure provisioners for the Docker backend.
Per PRD 0050 the per-provider provisioning steps (prompt, skills,
declarative provision-plan apply, supervise MCP registration) live on
the `AgentProvider` plugin under `bot_bottle/contrib/`. CA and git
provisioning also moved to the AgentProvider ABC (with Debian/node
defaults); user plugins override them for non-standard images.
No modules remain in this subpackage — the directory is kept so that
existing imports of `from .provision import ...` don't need updating
if new backend-specific provisioners are added later.
"""
+1 -1
View File
@@ -19,7 +19,7 @@ from ...env import ResolvedEnv
from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from ...supervisor.plan import SupervisePlan
from ...git_gate import GitGatePlan
def preflight() -> None:
+20 -5
View File
@@ -1,6 +1,6 @@
"""Docker host-side primitives used by DockerBottleBackend: probing
for docker on PATH, slugifying agent names, checking image/container
existence, and building images."""
"""Docker host-side primitives used by DockerBottleBackend: the lean
`run_docker` subprocess wrapper, probing for docker on PATH, slugifying agent
names, checking image/container existence, and building images."""
from __future__ import annotations
@@ -11,9 +11,24 @@ import shutil
import subprocess
from typing import Iterator
from ...docker_cmd import run_docker
from ...log import die, info
# from ...workspace import WorkspacePlan
def run_docker(
argv: list[str], *, env: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
"""Run a `docker` command, capturing stdout/stderr as text. Never raises on
a non-zero exit — callers inspect `returncode` / `stderr` so they can stay
fail-closed or tolerate idempotent no-ops (e.g. removing an already-absent
container).
`env` sets the child process environment — used to hand a secret to a bare
`--env NAME` flag (docker inherits its value from this process) so the value
never lands on argv or in `docker inspect`'s recorded command line."""
return subprocess.run(
argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
check=False, env=env,
)
# Cap on the suffix the container-name conflict logic will try before
+1 -1
View File
@@ -11,7 +11,7 @@ from pathlib import Path
from ..bottle_state import egress_state_dir
from ..egress import EGRESS_ROUTES_FILENAME
from ..egress_addon_core import LOG_OFF, load_config
from ..gateway.egress.addon_core import LOG_OFF, load_config
class EgressApplyError(RuntimeError):
+20 -2
View File
@@ -1,7 +1,25 @@
"""Firecracker backend: Linux KVM microVM isolation (issue #342)."""
"""Firecracker backend: Linux KVM microVM isolation (issue #342).
Thin by design: `FirecrackerBottleBackend` is re-exported lazily via
`__getattr__`, so importing a leaf module under this package doesn't drag the
backend (and the framework it pulls) into memory.
"""
from __future__ import annotations
from .backend import FirecrackerBottleBackend
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .backend import FirecrackerBottleBackend
def __getattr__(name: str) -> Any:
if name == "FirecrackerBottleBackend":
from .backend import FirecrackerBottleBackend
globals()[name] = FirecrackerBottleBackend
return FirecrackerBottleBackend
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
__all__ = ["FirecrackerBottleBackend"]
+9 -5
View File
@@ -8,7 +8,8 @@ fail-closed nftables egress boundary. Selected by
from __future__ import annotations
from contextlib import contextmanager
import io
from contextlib import contextmanager, redirect_stderr
from pathlib import Path
from typing import Generator, Sequence
@@ -17,7 +18,7 @@ from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from ...supervisor.plan import SupervisePlan
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from . import cleanup as _cleanup
from . import enumerate as _enumerate
@@ -52,8 +53,11 @@ class FirecrackerBottleBackend(
return _setup.setup()
@classmethod
def status(cls) -> int:
def status(cls, *, quiet: bool = False) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -119,5 +123,5 @@ class FirecrackerBottleBackend(
return plan.agent_supervise_url
def ensure_orchestrator(self) -> str:
from . import infra_vm
return infra_vm.ensure_running().control_plane_url
from .infra import FirecrackerInfraService
return FirecrackerInfraService().ensure_running()
@@ -22,6 +22,9 @@ class FirecrackerBottlePlan(BottlePlan):
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
# Applied to every agent SSH exec and mirrored into /run inside the VM so
# the host can recover it after the infra VM restarts.
env_var_secret: str = ""
@property
def container_name(self) -> str:
@@ -88,6 +88,12 @@ def _scan_processes(run_root: Path) -> tuple[set[str], list[int]]:
return live, orphan_pids
def live_run_dirs() -> tuple[Path, ...]:
"""Run directories backed by currently running agent microVMs."""
live, _ = _scan_processes(_run_root())
return tuple(Path(path) for path in sorted(live))
def _orphan_run_dirs(run_root: Path, live: set[str]) -> list[str]:
"""Run dirs with no live VM behind them — the leaked ones to remove."""
if not run_root.is_dir():
@@ -1,22 +1,23 @@
"""Consolidated bottle launch sequence for the Firecracker backend
(PRD 0070, Stage B).
The shared gateway + orchestrator control plane run in a single persistent
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
the gateway's egress / supervise / git-http ports at the infra VM via a
PREROUTING DNAT on their own host-side TAP IP (see
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
over HTTP at the infra VM's guest IP.
The orchestrator control plane and the shared gateway run in **two** persistent
per-host **infra VMs** (`infra_vm.py`), split per PRD 0070 — not Docker
containers. Agent VMs reach the gateway's egress / supervise / git-http ports
via a PREROUTING DNAT on their own host-side TAP IP that redirects to the
**gateway** VM (see `scripts/firecracker-netpool.sh`) — never the orchestrator,
so a breached agent has no L3 route to the control plane. The host CLI reaches
the control plane over HTTP at the orchestrator VM's guest IP.
Attribution is by the agent VM's guest IP, unspoofable by construction: the
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
expected VM can source-IP that address.
Sequence:
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
prior launcher may already have booted it);
2. register the bottle by its guest IP (attribution key) → bottle id +
identity token;
1. ensure the infra VMs (orchestrator control plane + gateway data plane)
are up (a singleton pair — a prior launcher may already have booted them);
2. register the bottle on the orchestrator by its guest IP (attribution key)
→ bottle id + identity token;
3. provision its git-gate repos/creds into the gateway VM (over SSH);
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
@@ -25,16 +26,26 @@ The TAP slot allocation, rootfs build, and VM boot are the caller's job.
from __future__ import annotations
import json
import subprocess
from dataclasses import dataclass
from pathlib import Path
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...log import info
from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
from ...orchestrator.lifecycle import (
OrchestratorStartError, # re-exported so callers can catch it
)
from ..consolidated_util import provision_bottle, teardown_consolidated as _teardown_util
from . import infra_vm
from ...orchestrator.reprovision import reprovision_bottles
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ..provision_bottle import deprovision_bottle, provision_bottle
from . import cleanup, util
from .gateway import FirecrackerGateway
from .infra import FirecrackerInfraService
_ENV_VAR_SECRET_PATH = "/run/bot-bottle/env-var-secret"
class ConsolidatedLaunchError(RuntimeError):
@@ -50,6 +61,55 @@ class LaunchContext:
source_ip: str # the VM's guest IP — the attribution key
gateway_ca_pem: str # the shared gateway CA the provisioner installs
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def _guest_ip_from_config(config_path: Path) -> str:
"""Read the kernel's configured guest IP from a Firecracker config."""
try:
config = json.loads(config_path.read_text())
args = config["boot-source"]["boot_args"]
ip_arg = next(part for part in args.split() if part.startswith("ip="))
return ip_arg.removeprefix("ip=").split(":", 1)[0]
except (OSError, ValueError, KeyError, TypeError, StopIteration):
return ""
def persist_env_var_secret(private_key: Path, guest_ip: str, secret: str) -> None:
"""Mirror the exec-time key into guest tmpfs for restart recovery."""
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip)
+ [f"umask 077; mkdir -p /run/bot-bottle; cat > {_ENV_VAR_SECRET_PATH}"],
input=secret, capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
raise ConsolidatedLaunchError(
f"failed to persist {ENV_VAR_SECRET_NAME} in agent VM: "
f"{proc.stderr.strip() or '<no stderr>'}"
)
def _reprovision_running_bottles(client: OrchestratorClient) -> None:
"""Read keys from live agent VMs and restore the restarted gateway."""
try:
secrets_by_ip: dict[str, str] = {}
for run_dir in cleanup.live_run_dirs():
guest_ip = _guest_ip_from_config(run_dir / "config.json")
private_key = run_dir / "bottle_id_ed25519"
if not guest_ip or not private_key.is_file():
continue
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip)
+ [f"cat {_ENV_VAR_SECRET_PATH}"],
capture_output=True, text=True, check=False,
)
if proc.returncode == 0 and proc.stdout.strip():
secrets_by_ip[guest_ip] = proc.stdout.strip()
count = reprovision_bottles(client, secrets_by_ip)
if count:
info(f"reprovisioned egress tokens for {count} Firecracker bottle(s)")
except (OSError, OrchestratorClientError) as exc:
info(f"egress token reprovision skipped: {exc}")
def launch_consolidated(
@@ -63,41 +123,45 @@ def launch_consolidated(
"""Ensure the infra VM is up, register the bottle by its guest IP, and
provision its git-gate state into the gateway VM. Returns the context the
agent-VM launch needs. Raises on failure — the caller tears down."""
infra = infra_vm.ensure_running()
url = infra.control_plane_url
service = FirecrackerInfraService()
url = service.ensure_running()
client = OrchestratorClient(url)
_reprovision_running_bottles(client)
transport = infra_vm.gateway_transport()
# Read the gateway's provisioning transport + CA off the Gateway service.
gateway = service.gateway()
transport = gateway.provisioning_transport()
reg = provision_bottle(
client, guest_ip, egress_plan, git_gate_plan, transport,
image_ref=image_ref, tokens=tokens,
)
# The shared gateway CA every agent on this host trusts for TLS
# interception — fetched from the infra VM over SSH.
# interception — fetched from the gateway VM over SSH.
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=guest_ip,
gateway_ca_pem=infra.gateway_ca_pem(),
gateway_ca_pem=gateway.ca_cert_pem(),
orchestrator_url=url,
env_var_secret=reg.env_var_secret,
)
def teardown_consolidated(
def deprovision_consolidated(
bottle_id: str, *, orchestrator_url: str, timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
NOT stop the infra VM — it's a persistent per-host singleton shared by
NOT stop the infra VMs — they're persistent per-host singletons shared by
every bottle."""
_teardown_util(bottle_id, infra_vm.gateway_transport(),
deprovision_bottle(bottle_id, FirecrackerGateway().provisioning_transport(),
orchestrator_url=orchestrator_url, timeout=timeout)
__all__ = [
"LaunchContext",
"launch_consolidated",
"teardown_consolidated",
"deprovision_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
]
@@ -60,12 +60,18 @@ class VmHandle:
self.process.wait(timeout=5)
def _boot_args(guest_ip: str, host_ip: str, pubkey: str) -> str:
def _boot_args(
guest_ip: str, host_ip: str, pubkey: str, extra: str = "",
) -> str:
# ip=<client>::<gw>:<netmask>::<dev>:off — /31 point-to-point link,
# so netmask is 255.255.255.254 and the gateway is the host TAP IP.
ip_arg = f"ip={guest_ip}::{host_ip}:255.255.255.254::eth0:off"
pub_b64 = base64.b64encode(pubkey.encode()).decode()
return f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
args = f"{_BASE_BOOT_ARGS} {ip_arg} bb_pubkey={pub_b64}"
# `extra` carries caller-supplied cmdline params the guest init reads
# (e.g. the gateway VM's `bb_orch=<orchestrator guest IP>`). Agent VMs pass
# nothing.
return f"{args} {extra}".rstrip() if extra else args
def _config(
@@ -79,6 +85,7 @@ def _config(
mem_mib: int,
guest_mac: str,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> dict[str, object]:
drives: list[dict[str, object]] = [
{
@@ -101,7 +108,7 @@ def _config(
return {
"boot-source": {
"kernel_image_path": str(util.kernel_path()),
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
"boot_args": _boot_args(guest_ip, host_ip, pubkey, extra_boot_args),
},
"drives": drives,
"network-interfaces": [
@@ -132,6 +139,7 @@ def boot(
guest_mac: str = "06:00:AC:10:00:02",
detached: bool = False,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> VmHandle:
"""Write the config and launch the VMM. Returns once the process is
spawned; callers wait for SSH readiness separately.
@@ -147,7 +155,7 @@ def boot(
_config(
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
data_drive=data_drive,
data_drive=data_drive, extra_boot_args=extra_boot_args,
),
indent=2,
))
+156
View File
@@ -0,0 +1,156 @@
"""The Firecracker gateway data plane as a microVM (PRD 0070).
`FirecrackerGateway` is the Firecracker implementation of the backend-neutral
`Gateway` service, and owns the gateway's host-side logic directly: booting the
gateway microVM on its NAT'd link, seeding the pre-minted `gateway` token,
fetching the mitmproxy CA, and the SSH exec/cp provisioning transport. The
gateway VM never opens `bot-bottle.db` (#469), so it holds no signing key —
only the token the host hands it via `connect_to_orchestrator`.
The plane-agnostic VM substrate the orchestrator VM also uses stays in
`infra_vm` — booting a VM from a per-plane rootfs (`boot_vm`), the stable SSH
keypair, the secret-push retry loop, the PID lifecycle, and the
adoption/version helpers. The guest-side gateway daemon startup lives in the
gateway rootfs's guest init (`_gateway_init` in `infra_vm`), so it can't live in
a host-side method either. The pair coordinator (orchestrator-first, under a
singleton flock) is `FirecrackerInfraService` (`infra.py`).
"""
from __future__ import annotations
import subprocess
from urllib.parse import urlparse
from ...gateway import (
DEFAULT_CA_TIMEOUT_SECONDS,
Gateway,
GatewayError,
GatewayTransport,
)
from .. import util as backend_util
from . import infra_vm, netpool, util
from .gateway_transport import FirecrackerGatewayTransport
# The gateway microVM's name (its run dir). Fixed per host — one gateway VM,
# shared by every agent VM.
GATEWAY_NAME = "bot-bottle-gateway"
# The gateway VM's slim memory ceiling — the data plane carries no build
# tooling (buildah lives only on the orchestrator rootfs) — PRD 0070
# "Memory: fixed ceilings".
_GW_MEM_MIB = 2048
# The pre-minted `gateway` JWT path in the guest. The gateway init (in
# `infra_vm`) waits for it before starting the data plane, so its canonical
# definition lives with that init; imported here for the push so the
# load-bearing path isn't duplicated.
_GUEST_GATEWAY_JWT_PATH = infra_vm._GUEST_GATEWAY_JWT_PATH
# mitmproxy writes its CA here a beat after start; agents install it to trust
# the gateway's TLS interception. Host-side only (SSH cat), so it lives here.
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
_CA_FETCH_TIMEOUT_SECONDS = 15.0
class FirecrackerGateway(Gateway):
"""The consolidated gateway as a Firecracker microVM on the gateway link.
The gateway rootfs is built/downloaded by `infra_vm.ensure_built` (the ABC's
`ensure_built` no-op here); `connect_to_orchestrator` boots the VM resolving
policy against the orchestrator and seeds the pre-minted `gateway` token."""
name = GATEWAY_NAME
def __init__(self, vm: infra_vm.InfraVm | None = None) -> None:
# The live VM handle when this process booted it; None when adapting an
# already-running gateway (CA fetch / provisioning go through the stable
# key + fixed link, so no live handle is needed).
self._vm = vm
self._orchestrator_url = ""
self._gateway_token = ""
def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
"""Boot the gateway VM on its link resolving policy against
`orchestrator_url`, then seed `gateway_token` over SSH. The orchestrator's
guest IP is parsed from the URL and passed on the cmdline (`bb_orch=`), so
the baked init stays IP-independent. Boot the orchestrator first — the
gateway daemons reach it at startup. The gateway never mints, so it holds
no signing key, only this token (#469)."""
self._orchestrator_url = orchestrator_url
self._gateway_token = gateway_token
if not self._orchestrator_url:
raise GatewayError(
"gateway requires an orchestrator URL to run "
"(resolver-only data plane; no single-tenant fallback)"
)
if not self._gateway_token:
raise GatewayError(
"gateway requires a pre-minted `gateway` token to run "
"(the orchestrator mints it; the gateway never holds the key)"
)
orchestrator_guest_ip = urlparse(self._orchestrator_url).hostname or ""
if not orchestrator_guest_ip:
raise GatewayError(
f"cannot resolve orchestrator guest IP from {self._orchestrator_url!r}"
)
# Boot on the gateway link from the gateway rootfs, then push the token
# the init waits for before starting the data plane.
vm = infra_vm.boot_vm(
name=GATEWAY_NAME, slot=netpool.gw_slot(), run_dir=infra_vm._gw_dir(),
role="gateway", mem_mib=_GW_MEM_MIB,
extra_boot_args=f"bb_orch={orchestrator_guest_ip}",
)
infra_vm.push_secret(
vm, self._gateway_token, _GUEST_GATEWAY_JWT_PATH,
"the gateway JWT to the gateway VM (its data plane will not start)",
)
self._vm = vm
def is_running(self) -> bool:
return infra_vm._pidfile_alive(infra_vm._gw_dir())
def stop(self) -> None:
"""Stop only the gateway VM (idempotent — absent is success). A dead
gateway already fails `infra_vm.adoptable`, so no version marker to
clear here."""
infra_vm._kill_pidfile(infra_vm._gw_dir())
infra_vm._pid_file(infra_vm._gw_dir()).unlink(missing_ok=True)
def address(self) -> str:
"""The gateway VM's guest IP — the agent-facing target agent VMs'
gateway-port traffic is DNAT'd to."""
return netpool.gw_slot().guest_ip
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception. Fetched from the gateway VM over SSH; polls because
mitmproxy writes it a beat after boot. Works from any later launcher —
falls back to the stable key + fixed link when this process didn't boot
the VM."""
key = self._vm.private_key if self._vm else infra_vm._infra_dir() / "id_ed25519"
ip = self._vm.guest_ip if self._vm else netpool.gw_slot().guest_ip
def _fetch() -> str | None:
proc = subprocess.run(
util.ssh_base_argv(key, ip) + [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True,
timeout=_CA_FETCH_TIMEOUT_SECONDS, check=False,
)
ok = proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout
return proc.stdout if ok else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
raise GatewayError(str(exc)) from exc
def provisioning_transport(self) -> GatewayTransport:
"""The exec/cp transport git-gate provisioning stages per-bottle repos +
deploy keys through (over SSH to the gateway VM). Needs no live handle —
built from the stable key + the gateway link's guest IP, so teardown can
use it too."""
return FirecrackerGatewayTransport(
infra_vm._infra_dir() / "id_ed25519", netpool.gw_slot().guest_ip)
__all__ = ["FirecrackerGateway", "FirecrackerGatewayTransport", "GATEWAY_NAME"]
@@ -0,0 +1,56 @@
"""The `GatewayTransport` for the gateway running in the gateway microVM
(PRD 0070).
How the launcher stages files + runs commands in the running gateway: the
docker exec/cp equivalents over SSH (dropbear + the stable infra key). The
backend-neutral provisioning logic that drives it lives in
`backend.provision_gateway`.
"""
from __future__ import annotations
import os
import shlex
import stat
import subprocess
from pathlib import Path
from ...gateway import GatewayProvisionError
from . import util
class FirecrackerGatewayTransport:
"""`GatewayTransport` for the gateway running in the gateway VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
# Preserve the source mode (docker cp does): the access-hook is staged
# 0700 and git-http execs it directly — a plain `cat >` would land it
# 0644 and the exec fails with EACCES; keys stay 0600.
mode = stat.S_IMODE(os.stat(src).st_mode)
q = shlex.quote(dest)
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip)
+ [f"cat > {q} && chmod {mode:o} {q}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
__all__ = ["FirecrackerGatewayTransport"]
+17 -14
View File
@@ -1,17 +1,17 @@
"""Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
Agent Dockerfiles build **inside the persistent per-host infra VM**
Agent Dockerfiles build **inside the persistent per-host orchestrator VM**
(`infra_vm.py`), which carries buildah (rootless, daemonless): no host Docker
daemon, no root-equivalent `docker` group. The build runs over SSH against the
infra VM and its rootfs streams back to the host, where the existing
orchestrator VM and its rootfs streams back to the host, where the existing
`mke2fs -d` path (`util.build_rootfs_ext4`) turns it into a bootable ext4.
Building in the infra VM — rather than a throwaway builder VM — means there is
one buildah image (`bot-bottle-infra`) and no contention for the orchestrator
TAP. Tradeoff: an untrusted Dockerfile's `RUN` steps share the VM with the
control plane + gateway (buildah `--isolation chroot` isn't a hard boundary) —
the accepted single-VM blast-radius tradeoff, re-splittable into a disposable
builder (booted from this same image on its own TAP) later.
Builds run on the control-plane (orchestrator) side — not the data-plane
gateway — per PRD 0070 v1 ("builds stay in the orchestrator"): it owns
launches and already carries buildah. Tradeoff: an untrusted Dockerfile's
`RUN` steps share the VM with the control plane (buildah `--isolation chroot`
isn't a hard boundary) — the accepted blast-radius tradeoff, re-splittable
into a disposable builder (booted from this same image on its own TAP) later.
"""
from __future__ import annotations
@@ -26,7 +26,8 @@ from pathlib import Path
from typing import Generator
from ...log import die, info
from . import infra_vm, util
from . import util
from .infra import FirecrackerInfraService
# vfs + chroot: buildah works as root in the microVM (no fuse-overlayfs /
# overlay module / subuid maps). `--isolation` is a build/run-only flag;
@@ -121,11 +122,13 @@ def _build_lock() -> Generator[None, None, None]:
def _build_in_infra(
dockerfile: Path, base: Path, smoke_test: tuple[str, ...], digest: str,
) -> None:
"""Ensure the infra VM is up, `buildah build` the Dockerfile in it, smoke
test the image, and stream its rootfs into `base`. The infra VM persists;
only the per-build container/image/context are cleaned up."""
infra = infra_vm.ensure_running()
key, ip = infra.private_key, infra.guest_ip
"""Ensure the infra VMs are up, `buildah build` the Dockerfile in the
orchestrator VM (the build-capable control plane), smoke test the image,
and stream its rootfs into `base`. The infra VMs persist; only the
per-build container/image/context are cleaned up."""
service = FirecrackerInfraService()
service.ensure_running()
key, ip = service.orchestrator().ssh_target()
tag = f"bot-bottle-agent-build-{digest}"
ctx = f"/tmp/agent-build-{digest}"
smoke_ctr, export_ctr = f"{tag}-smoke", f"{tag}-export"
+80
View File
@@ -0,0 +1,80 @@
"""The per-host infra service for the Firecracker backend (PRD 0070).
`FirecrackerInfraService` is the Firecracker `InfraService`: it composes the
orchestrator + gateway microVMs as an idempotent per-host singleton pair over
the shared `infra_vm` substrate (boot primitives, the singleton flock, the
adopt/version machinery). Unlike the docker/macOS services it takes no
per-instance config — the firecracker pair is a **hard** per-host singleton,
pinned to the fixed netpool links (`orch_slot()` / `gw_slot()`), host cache
paths, and a `booted-version` marker, so two pairs can't coexist on one host.
"""
from __future__ import annotations
from ...log import info
from ..infra_service import InfraService
from ...orchestrator.lifecycle import DEFAULT_STARTUP_TIMEOUT_SECONDS
from . import infra_vm
from .gateway import FirecrackerGateway
from .orchestrator import FirecrackerOrchestrator
class FirecrackerInfraService(InfraService):
"""Bring up the orchestrator + gateway microVM pair as a per-host singleton."""
def orchestrator(self) -> FirecrackerOrchestrator:
"""The control-plane service (adopts the running orchestrator VM by its
fixed link; no live handle needed for URL / SSH / health)."""
return FirecrackerOrchestrator()
def gateway(self) -> FirecrackerGateway:
"""The data-plane service (adopts the running gateway VM by its fixed
link; CA fetch / provisioning go through the stable key)."""
return FirecrackerGateway()
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Idempotent per-host singleton pair. Adopt the running VMs if the
orchestrator's control plane is healthy AND the gateway VM is alive AND
both booted the CURRENT code version (a prior launcher booted them — they
outlive short-lived `start` processes); otherwise clear any stale VMs and
boot a fresh pair (orchestrator first, then the gateway that resolves
policy against it). Returns the host control-plane URL.
Concurrency-safe: the cold stop/build/boot path is serialized by a host
flock, so two simultaneous first launches don't both boot on the same
links/PIDs. The healthy fast-path takes no lock."""
key = infra_vm._infra_dir() / "id_ed25519"
orchestrator = self.orchestrator()
url = orchestrator.url()
want = infra_vm.expected_version()
if infra_vm.adoptable(key, url, want):
info(f"adopting running infra VMs (orchestrator at {url})")
return url
with infra_vm.singleton_lock():
# Re-check under the lock: another launcher may have booted them while
# we waited (double-checked, so we adopt not re-boot).
if infra_vm.adoptable(key, url, want):
info(f"adopting running infra VMs (orchestrator at {url})")
return url
# Clear stale/hung/OUTDATED VMs holding either link before booting fresh.
infra_vm.stop()
infra_vm.ensure_built()
# Orchestrator first — the gateway daemons reach the control plane at
# startup. Each service owns its own boot; the orchestrator (which
# holds the signing key) mints the role-scoped `gateway` JWT for the
# gateway, which never sees the key (#469).
orchestrator.ensure_running(startup_timeout=startup_timeout)
self.gateway().connect_to_orchestrator(
orchestrator.gateway_url(), orchestrator.mint_gateway_token())
infra_vm.record_booted_version(want)
return url
def stop(self) -> None:
"""Stop both infra VMs (idempotent) and drop the version marker."""
infra_vm.stop()
__all__ = ["FirecrackerInfraService"]
@@ -1,22 +1,26 @@
"""Prebuilt infra-VM rootfs, pulled as an artifact (PRD 0069 Stage 2).
"""Prebuilt infra-VM rootfs images, pulled as artifacts (PRD 0069 Stage 2).
The Firecracker infra VM boots a fixed rootfs (orchestrator control plane +
gateway + buildah, control-plane init as PID 1) that does not vary per launch —
the per-boot bits (authorized_keys, guest IP) ride the kernel cmdline, so one
rootfs boots on any host. Instead of building that rootfs on the launch host
with Docker, we build it **off-host** and publish it as a versioned, ready-to-
boot ext4 (gzip-compressed) to a Gitea **generic package**; the launch host
downloads + verifies + boots it. No Docker, no image tooling on the launch
host — just an HTTP fetch and gunzip.
The two Firecracker infra VMs each boot a fixed per-plane rootfs that does not
vary per launch — the per-boot bits (authorized_keys, guest IP) ride the kernel
cmdline, so one rootfs boots on any host:
* `orchestrator` — the control plane + buildah (in-VM agent-image builds);
* `gateway` — the slim data plane (no build tooling on the exposed VM).
Instead of building them on the launch host with Docker, we build them
**off-host** and publish each as a versioned, ready-to-boot ext4 (gzip-
compressed) to a per-role Gitea **generic package**; the launch host downloads +
verifies + boots them. No Docker, no image tooling on the launch host — just an
HTTP fetch and gunzip.
publish (off-host, see publish_infra.py):
docker build -> rootfs dir -> mke2fs -> gzip -> PUT generic package
pull (this module, launch host):
GET .../rootfs.ext4.gz (+ .sha256) -> verify -> gunzip -> boot
The artifact **version** is a content hash of everything baked into the rootfs
(the shipped bot_bottle package, the three Dockerfiles, and the init), so a
launch host always pulls the artifact matching its code and a content change
Each artifact **version** is a content hash of everything baked into that rootfs
(the shipped bot_bottle package, the role's Dockerfiles, and the role init), so
a launch host always pulls the artifact matching its code and a content change
can't silently boot a stale rootfs. A checksum mismatch fails closed.
Set `BOT_BOTTLE_INFRA_BUILD=local` to skip the pull and build the rootfs
@@ -41,11 +45,23 @@ from . import util
_ARTIFACT_FORMAT = "1"
_REPO_ROOT = Path(__file__).resolve().parents[3]
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra", "Dockerfile.infra.fc")
# The two per-plane infra VM roles. Each publishes/pulls its own rootfs artifact
# from its own generic package; the Dockerfiles baked into each differ (only the
# orchestrator rootfs carries buildah), so the versions are hashed separately.
ROLES = ("orchestrator", "gateway")
_DOCKERFILES = {
"orchestrator": ("Dockerfile.orchestrator", "Dockerfile.orchestrator.fc"),
"gateway": ("Dockerfile.gateway",),
}
_DEFAULT_BASE = "https://gitea.dideric.is"
_DEFAULT_OWNER = "didericis"
_PACKAGE = "bot-bottle-firecracker-infra"
def _package(role: str) -> str:
return f"bot-bottle-firecracker-{role}"
# Streaming copy chunk for the (hundreds-of-MB) download.
_CHUNK = 1 << 20
@@ -57,21 +73,24 @@ def local_build_requested() -> bool:
return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"
def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) -> str:
"""Content hash (16 hex) of everything baked into the infra rootfs: the
whole shipped `bot_bottle` package, the three fixed Dockerfiles, and the
guest init. Deterministic across the publish host and the launch host when
both run the same checkout, so the tag the launch host pulls is exactly the
tag publish produced.
def infra_artifact_version(
init_script: str, role: str, *, repo_root: Path = _REPO_ROOT,
) -> str:
"""Content hash (16 hex) of everything baked into `role`'s infra rootfs: the
whole shipped `bot_bottle` package, that role's Dockerfiles, and its guest
init. Deterministic across the publish host and the launch host when both run
the same checkout, so the tag the launch host pulls is exactly the tag
publish produced.
The package is `COPY bot_bottle /app/bot_bottle`'d wholesale into the image,
so hash *every* regular file under it — not just `*.py`. Non-Python inputs
(e.g. `egress_entrypoint.sh`, `netpool.defaults.env`) are baked in too, and
a change to one must bump the version or a launch host could boot a stale
rootfs whose code differs from its checkout. `__pycache__`/`.pyc` are the
only exclusions — build artifacts, never copied."""
The package is baked into both images wholesale (orchestrator `COPY`s it,
gateway `pip install`s it), so hash *every* regular file under it — not just
`*.py`. Non-Python inputs (e.g. `gateway/egress/entrypoint.sh`,
`netpool.defaults.env`) are baked in too, and a change to one must bump the
version or a launch host could boot a stale rootfs whose code differs from
its checkout. `__pycache__`/`.pyc` are the only exclusions — build artifacts,
never copied."""
h = hashlib.sha256()
h.update(f"format={_ARTIFACT_FORMAT}\n".encode())
h.update(f"format={_ARTIFACT_FORMAT}\nrole={role}\n".encode())
pkg = repo_root / "bot_bottle"
for path in sorted(pkg.rglob("*")):
if not path.is_file():
@@ -81,7 +100,7 @@ def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) ->
h.update(str(path.relative_to(repo_root)).encode())
h.update(b"\0")
h.update(path.read_bytes())
for name in _DOCKERFILES:
for name in _DOCKERFILES[role]:
h.update(name.encode())
h.update(b"\0")
h.update((repo_root / name).read_bytes())
@@ -107,11 +126,11 @@ def _config() -> tuple[str, str, str]:
return base, owner, token
def artifact_url(version: str, filename: str) -> str:
"""The generic-package download URL for one file of this version's
artifact (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
def artifact_url(version: str, filename: str, *, role: str) -> str:
"""The generic-package download URL for one file of `role`'s artifact at
`version` (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
base, owner, _ = _config()
return f"{base}/api/packages/{owner}/generic/{_PACKAGE}/{version}/{filename}"
return f"{base}/api/packages/{owner}/generic/{_package(role)}/{version}/{filename}"
_GZ_NAME = "rootfs.ext4.gz"
@@ -119,8 +138,8 @@ _SHA_NAME = "rootfs.ext4.gz.sha256"
_CANDIDATE_DIR_ENV = "BOT_BOTTLE_INFRA_ARTIFACT_DIR"
def _cache_root(version: str) -> Path:
return util.cache_dir() / "infra-artifact" / version
def _cache_root(version: str, role: str) -> Path:
return util.cache_dir() / "infra-artifact" / role / version
def _open(url: str) -> urllib.request.Request:
@@ -162,13 +181,17 @@ def _sha256_file(path: Path) -> str:
return h.hexdigest()
def ensure_artifact_gz(version: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `version` — downloading it (and
its `.sha256`) once, then reusing it. Fail-closed on a checksum mismatch:
the partial is removed and we die rather than boot an unverified rootfs."""
def ensure_artifact_gz(version: str, *, role: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `role` at `version` —
downloading it (and its `.sha256`) once, then reusing it. Fail-closed on a
checksum mismatch: the partial is removed and we die rather than boot an
unverified rootfs.
A pre-staged candidate bundle (`BOT_BOTTLE_INFRA_ARTIFACT_DIR`) holds each
role under its own `<dir>/<role>/` subdir."""
candidate_dir = os.environ.get(_CANDIDATE_DIR_ENV, "").strip()
if candidate_dir:
root = Path(candidate_dir)
root = Path(candidate_dir) / role
version_file = root / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
# FileNotFoundError.
@@ -177,7 +200,7 @@ def ensure_artifact_gz(version: str) -> Path:
declared = version_file.read_text(encoding="utf-8").strip()
if declared != version:
die(
f"infra candidate version mismatch: expected {version}, "
f"infra candidate version mismatch ({role}): expected {version}, "
f"bundle contains {declared or '<empty>'}"
)
gz = root / _GZ_NAME
@@ -188,22 +211,22 @@ def ensure_artifact_gz(version: str) -> Path:
actual = _sha256_file(gz)
if actual != expected:
die(
f"infra candidate checksum mismatch for {version}:\n"
f"infra candidate checksum mismatch ({role}) for {version}:\n"
f" expected {expected}\n actual {actual}"
)
return gz
root = _cache_root(version)
root = _cache_root(version, role)
root.mkdir(parents=True, exist_ok=True)
gz = root / _GZ_NAME
ok = root / ".verified"
if gz.is_file() and ok.is_file():
return gz
info(f"pulling infra rootfs artifact {_PACKAGE}/{version}")
_download(artifact_url(version, _GZ_NAME), gz)
info(f"pulling infra rootfs artifact {_package(role)}/{version}")
_download(artifact_url(version, _GZ_NAME, role=role), gz)
sha = root / _SHA_NAME
_download(artifact_url(version, _SHA_NAME), sha)
_download(artifact_url(version, _SHA_NAME, role=role), sha)
expected = sha.read_text().split()[0].strip().lower()
actual = _sha256_file(gz)
@@ -211,7 +234,7 @@ def ensure_artifact_gz(version: str) -> Path:
gz.unlink(missing_ok=True)
sha.unlink(missing_ok=True)
die(
f"infra artifact checksum mismatch for {version}:\n"
f"infra artifact checksum mismatch ({role}) for {version}:\n"
f" expected {expected}\n"
f" actual {actual}\n"
f" refusing to boot an unverified rootfs."
@@ -220,13 +243,13 @@ def ensure_artifact_gz(version: str) -> Path:
return gz
def materialize_ext4(version: str, dest: Path) -> None:
"""Ensure the verified artifact is cached, then gunzip it to `dest` — a
fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
def materialize_ext4(version: str, dest: Path, *, role: str) -> None:
"""Ensure the verified `role` artifact is cached, then gunzip it to `dest` —
a fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
pristine). Atomic via a `.part` sibling."""
gz = ensure_artifact_gz(version)
gz = ensure_artifact_gz(version, role=role)
tmp = dest.with_suffix(dest.suffix + ".part")
info(f"expanding infra rootfs -> {dest}")
info(f"expanding {role} infra rootfs -> {dest}")
with gzip.open(gz, "rb") as src, open(tmp, "wb") as out:
shutil.copyfileobj(src, out, _CHUNK)
tmp.replace(dest)
+278 -286
View File
@@ -1,18 +1,30 @@
"""The per-host infra VM for the Firecracker backend (PRD 0070 Stage B).
"""The per-host infra VMs for the Firecracker backend (PRD 0070).
A single persistent microVM that runs the orchestrator **control plane** (and,
in a following step, the gateway **data plane**) — the trusted per-host service
the docker backend runs as containers. It boots on the NAT'd orchestrator link
(`netpool.orch_slot()`): the host CLI reaches its control plane over HTTP at the
guest IP, and agent VMs reach its gateway ports over VM-to-VM routing.
Two persistent microVMs, split now that #469 got the DB off the data plane
(PRD 0070 "Separating the planes"):
Build-from-source (the default while the design churns): the rootfs is exported
from the locally built orchestrator image, which bakes the stdlib-only
control-plane source. A pull-from-registry mode (Gitea's OCI registry) becomes
the default later.
* **orchestrator VM** — the control plane. Boots on the NAT'd orchestrator
link (`netpool.orch_slot()`); the host CLI reaches its `/health` +
operator routes over HTTP at the guest IP. Sole opener of `bot-bottle.db`
(on its persistent /dev/vdb registry volume); holds the host-canonical
signing key (pushed post-boot). Also carries buildah, so in-VM agent-image
builds run here (PRD 0070 v1: builds stay with the control plane).
* **gateway VM** — the data plane. Boots on its own NAT'd link
(`netpool.gw_slot()`); runs the egress / git-http / supervise daemons that
agent VMs reach (their gateway-port traffic is DNAT'd here — never to the
orchestrator, so a breached agent has no L3 route to the control plane).
Holds the mitmproxy CA + a pre-minted `gateway` JWT (never the signing
key); reaches the orchestrator's control plane at `orch_guest:8099` over
the one nft forward rule that link allows.
SSH is left enabled for debugging; the control plane is the load-bearing
surface.
Each VM boots its **own** per-plane rootfs — the orchestrator rootfs carries the
control plane + buildah (in-VM agent builds), the gateway rootfs is the slim
data plane with no build tooling on the exposed VM. Two artifacts, built/pulled
per role (`infra_artifact`); each rootfs bakes only its own role init as PID 1.
The gateway VM also runs a slimmer memory ceiling.
SSH is left enabled for debugging + provisioning; the control plane is the
load-bearing surface.
"""
from __future__ import annotations
@@ -20,9 +32,7 @@ from __future__ import annotations
import fcntl
import hashlib
import os
import shlex
import signal
import stat
import subprocess
import time
import urllib.error
@@ -33,43 +43,53 @@ from pathlib import Path
from typing import Generator
from ...log import die, info
from .. import util as backend_util
from ..docker import util as docker_mod
from ..docker.gateway_provision import GatewayProvisionError
from . import firecracker_vm, infra_artifact, netpool, util
# The single infra-VM image: gateway data plane + baked control-plane source
# (Dockerfile.infra FROM the gateway image). Built from source by default;
# a pull-from-registry mode lands later.
_INFRA_IMAGE = "bot-bottle-infra:latest"
# The orchestrator VM's signing-key path on its persistent /dev/vdb volume
# (mounted at BOT_BOTTLE_ROOT=/var/lib/bot-bottle). The launcher seeds this
# file with the host-canonical key AFTER boot (over SSH), so the VM verifies
# tokens with the same key the host CLI signs with — the host token file stays
# the single source of truth, never clobbered per-backend (issue #469 review).
_GUEST_SIGNING_KEY_PATH = "/var/lib/bot-bottle/orchestrator-token"
# The gateway VM's pre-minted `gateway` JWT path (rootfs, not /dev/vdb — the
# data plane has no registry volume and never opens the DB). Pushed post-boot;
# the gateway daemons present it to the orchestrator, and never see the key.
_GUEST_GATEWAY_JWT_PATH = "/var/lib/bot-bottle/gateway-jwt"
# The two per-plane rootfs source images. The orchestrator VM boots a control
# plane + buildah rootfs (Dockerfile.orchestrator.fc, FROM orchestrator); the
# gateway VM boots the slim data-plane image directly (no build tooling on the
# exposed VM). Built from source by default; the launch host pulls prebuilt
# artifacts instead (`infra_artifact`).
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
_ORCHESTRATOR_FC_IMAGE = "bot-bottle-orchestrator-fc:latest"
_REPO_ROOT = Path(__file__).resolve().parents[3]
CONTROL_PLANE_PORT = 8099
# Gateway data-plane ports (agent-facing): egress proxy, supervise MCP,
# git-http. Reached by agent VMs over VM-to-VM routing (added next).
EGRESS_PORT = 9099
SUPERVISE_PORT = 9100
GIT_HTTP_PORT = 9420
# mitmproxy writes its CA here a beat after start; agents install it to trust
# the gateway's TLS interception.
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
# Per-role rootfs source image + the extra free space `mke2fs` leaves for the
# guest to grow into. The orchestrator keeps buildah's large build slack; the
# gateway carries no build tooling, so its rootfs is much smaller.
_ROOTFS_IMAGE = {"orchestrator": _ORCHESTRATOR_FC_IMAGE, "gateway": _GATEWAY_IMAGE}
_ROOTFS_SLACK_MIB = {"orchestrator": 8192, "gateway": 1024}
# The infra VM makes direct upstream connections (gateway egress, and buildah
ORCHESTRATOR_PORT = 8099
# The infra VMs make direct upstream connections (gateway egress, and buildah
# during builds), and the kernel `ip=` cmdline sets no resolver. Public for
# now; routing DNS through a filtered path is a later refinement.
_INFRA_RESOLVER = "1.1.1.1"
_HEALTH_TIMEOUT_SECONDS = 45.0
_HEALTH_POLL_SECONDS = 0.5
_CA_TIMEOUT_SECONDS = 30.0
# How long the launcher retries pushing a secret while the guest's SSH comes
# up. Below the init's own wait window, so a failed push dies here first.
_SECRET_PUSH_TIMEOUT_SECONDS = 30.0
_SECRET_PUSH_POLL_SECONDS = 0.5
@dataclass
class InfraVm:
"""A handle to the per-host infra VM: its guest IP and the stable SSH key
used to fetch the gateway CA / provision git-gate. `vm` is the live VMM
"""A handle to one infra VM: its guest IP and the stable SSH key used to
seed secrets / fetch the CA / provision git-gate. `vm` is the live VMM
handle when this process booted it, and None when adopting a singleton a
prior launcher started (teardown then goes through the PID file)."""
@@ -77,116 +97,61 @@ class InfraVm:
private_key: Path
vm: firecracker_vm.VmHandle | None = None
@property
def control_plane_url(self) -> str:
return f"http://{self.guest_ip}:{CONTROL_PLANE_PORT}"
def terminate(self) -> None:
"""Stop the infra VM — via the live handle if we booted it, else the
PID file (adopting-process case)."""
if self.vm is not None:
self.vm.terminate()
else:
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def role_init(role: str) -> str:
"""The guest PID-1 init for `role` (each per-plane rootfs bakes only its
own — no `bb_role` branch, since the rootfs *is* the role)."""
return _orchestrator_init() if role == "orchestrator" else _gateway_init()
def gateway_ca_pem(self, *, timeout: float = _CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) that agents install to trust its
TLS interception. Generated a moment after boot, so this polls over
SSH until it appears (mirrors DockerGateway.ca_cert_pem)."""
def _fetch() -> str | None:
proc = subprocess.run(
util.ssh_base_argv(self.private_key, self.guest_ip)
+ [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True, timeout=15, check=False,
)
ok = proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout
return proc.stdout if ok else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
die(str(exc))
def _role_version(role: str) -> str:
return infra_artifact.infra_artifact_version(role_init(role), role)
def ensure_built() -> None:
"""Ensure the infra rootfs is available before boot.
"""Ensure both infra rootfs artifacts are available before boot.
Default (docker-free, PRD 0069 Stage 2): download + verify the prebuilt
rootfs artifact matching this code version (see `infra_artifact`); the
launch host needs no Docker. `BOT_BOTTLE_INFRA_BUILD=local` instead builds
the three fixed images from source with host Docker — the infra image
`COPY --from`s the orchestrator image and is `FROM` the gateway image, so
both must exist first — for iterating on the Dockerfiles."""
orchestrator + gateway rootfs artifacts matching this code version (see
`infra_artifact`); the launch host needs no Docker.
`BOT_BOTTLE_INFRA_BUILD=local` instead builds the images from source with
host Docker (the orchestrator-fc image is `FROM` the orchestrator image, so
it must exist first) — for iterating on the Dockerfiles."""
if infra_artifact.local_build_requested():
build_infra_images_with_docker()
return
infra_artifact.ensure_artifact_gz(
infra_artifact.infra_artifact_version(_infra_init()))
for role in infra_artifact.ROLES:
infra_artifact.ensure_artifact_gz(_role_version(role), role=role)
def build_infra_images_with_docker() -> None:
"""Build the four fixed images from source with host Docker: orchestrator,
gateway, the shared infra base (Dockerfile.infra), then the Firecracker
infra image (Dockerfile.infra.fc: FROM infra + buildah). The launch host
uses this only in `BOT_BOTTLE_INFRA_BUILD=local` mode; `publish_infra`
uses it off-host to produce the published artifact."""
"""Build the fixed images from source with host Docker: orchestrator,
gateway, then the orchestrator-fc image (Dockerfile.orchestrator.fc: FROM
orchestrator + buildah). The gateway VM boots the gateway image directly.
The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local` mode;
`publish_infra` uses it off-host to produce the published artifacts."""
docker_mod.build_image(
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
docker_mod.build_image(
"bot-bottle-infra:latest", str(_REPO_ROOT), dockerfile="Dockerfile.infra")
docker_mod.build_image(
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra.fc")
_ORCHESTRATOR_FC_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator.fc")
def build_infra_rootfs_dir() -> Path:
"""The infra VM's base rootfs: the infra image prepared with the
control-plane + gateway init as PID 1. The init's content is folded into
the cache key so an init change rebuilds the rootfs (the base image digest
alone wouldn't catch it)."""
init = _infra_init()
def build_rootfs_dir(role: str) -> Path:
"""`role`'s base rootfs dir: its source image prepared with the role init as
PID 1. The init's content is folded into the cache key so an init change
rebuilds the rootfs (the base image digest alone wouldn't catch it)."""
init = role_init(role)
tag = hashlib.sha256(init.encode()).hexdigest()[:8]
return util.build_base_rootfs_dir(
_INFRA_IMAGE, variant=f"-infra-{tag}", init_script=init,
_ROOTFS_IMAGE[role], variant=f"-{role}-{tag}", init_script=init,
)
def ensure_running() -> InfraVm:
"""Idempotent per-host singleton. Adopt the infra VM if its control plane
is already healthy (a prior launcher booted it — it outlives short-lived
`start` processes); otherwise clear any stale VM and boot a fresh one.
Returns a handle usable for CA fetch / git-gate provisioning.
Concurrency-safe: the cold stop/build/boot path is serialized by a host
flock, so two simultaneous first launches don't both boot on the same
rootfs/PID. The healthy fast-path takes no lock."""
slot = netpool.orch_slot()
url = f"http://{slot.guest_ip}:{CONTROL_PLANE_PORT}"
key = _infra_dir() / "id_ed25519"
want = _expected_version()
if _adoptable(key, url, want):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
with _singleton_lock():
# Re-check under the lock: another launcher may have booted it while
# we waited for the lock (double-checked, so we adopt not re-boot).
if _adoptable(key, url, want):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
# Clear a stale/hung/OUTDATED VM holding the link before booting fresh.
stop()
ensure_built()
infra = boot()
wait_for_health(infra)
_record_booted_version(want)
return infra
@contextmanager
def _singleton_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing the infra VM's cold create path
def singleton_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing the infra pair's cold create path
(`stop`/`ensure_built`/`boot`). flock auto-releases if the launcher
crashes, so the lock is never leaked."""
lock_path = _infra_dir() / "singleton.lock"
@@ -199,44 +164,62 @@ def _singleton_lock() -> Generator[None, None, None]:
def stop() -> None:
"""Stop the infra VM singleton (idempotent — absent is success). Reaps the
recorded VMM AND any orphaned firecracker still bound to the infra config —
the PID file drifts after crashes / out-of-band kills, and a survivor would
hold the orchestrator TAP so the next boot dies with "tap … Resource busy".
Drops the version marker so a stopped VM is never treated as adoptable."""
_kill_pidfile()
"""Stop BOTH infra VMs (idempotent — absent is success). Reaps the
recorded VMMs AND any orphaned firecracker still bound to either infra
config — the PID files drift after crashes / out-of-band kills, and a
survivor would hold a link's TAP so the next boot dies with "tap …
Resource busy". Also reaps a surviving *legacy* single combined-VM (the
pre-split layout booted from `<infra>/config.json` on the orchestrator
link): the two-VM `stop` otherwise wouldn't know about it, and it would
hold the orchestrator TAP so the first split boot fails — so the cutover
is self-healing, no manual host teardown. Drops the version marker so a
stopped pair is never treated as adoptable."""
_kill_pidfile(_orch_dir())
_kill_pidfile(_gw_dir())
_kill_pidfile(_infra_dir()) # legacy pre-split combined VM (migration)
_kill_infra_firecrackers()
_pid_file().unlink(missing_ok=True)
_pid_file(_orch_dir()).unlink(missing_ok=True)
_pid_file(_gw_dir()).unlink(missing_ok=True)
_pid_file(_infra_dir()).unlink(missing_ok=True) # legacy
_version_file().unlink(missing_ok=True)
def boot() -> InfraVm:
"""Boot the infra VM (detached, so it outlives the launcher) on the
orchestrator link, recording its PID. Prefer `ensure_running`."""
slot = netpool.orch_slot()
def boot_vm(
*,
name: str,
slot: netpool.Slot,
run_dir: Path,
role: str,
mem_mib: int,
data_drive: Path | None = None,
extra_boot_args: str = "",
) -> InfraVm:
"""Boot the `role` infra VM from its per-plane rootfs on `slot`'s link.
Records the PID."""
if not netpool.tap_present(slot.iface):
die(f"orchestrator link {slot.iface} not present.\n"
die(f"infra link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
run_dir = _infra_dir()
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
if infra_artifact.local_build_requested():
util.build_rootfs_ext4(build_infra_rootfs_dir(), rootfs, slack_mib=8192)
util.build_rootfs_ext4(
build_rootfs_dir(role), rootfs, slack_mib=_ROOTFS_SLACK_MIB[role])
else:
# Prebuilt artifact already carries the buildah build slack; expand it
# to a fresh writable rootfs for this boot.
infra_artifact.materialize_ext4(
infra_artifact.infra_artifact_version(_infra_init()), rootfs)
# Prebuilt artifact already carries the role's build slack; expand it to
# a fresh writable rootfs for this boot.
infra_artifact.materialize_ext4(_role_version(role), rootfs, role=role)
private_key, pubkey = _stable_keypair()
info(f"booting infra VM on {slot.iface} (guest {slot.guest_ip})")
info(f"booting {role} VM on {slot.iface} (guest {slot.guest_ip})")
boot_args = extra_boot_args
vm = firecracker_vm.boot(
name="bot-bottle-infra", rootfs=rootfs, tap=slot.iface,
name=name, rootfs=rootfs, tap=slot.iface,
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
run_dir=run_dir, mem_mib=4096, detached=True,
data_drive=_ensure_registry_volume(),
run_dir=run_dir, mem_mib=mem_mib, detached=True,
data_drive=data_drive, extra_boot_args=boot_args,
)
_pid_file().write_text(str(vm.process.pid))
_pid_file(run_dir).write_text(str(vm.process.pid))
return InfraVm(guest_ip=slot.guest_ip, private_key=private_key, vm=vm)
@@ -246,73 +229,85 @@ def _infra_dir() -> Path:
return d
def _pid_file() -> Path:
return _infra_dir() / "vm.pid"
def _orch_dir() -> Path:
d = _infra_dir() / "orchestrator"
d.mkdir(parents=True, exist_ok=True)
return d
def _gw_dir() -> Path:
d = _infra_dir() / "gateway"
d.mkdir(parents=True, exist_ok=True)
return d
def _pid_file(run_dir: Path) -> Path:
return run_dir / "vm.pid"
def _version_file() -> Path:
"""Records the infra-artifact version the *running* VM booted from, so a
later launcher can tell whether the singleton it found is the current code.
Without it, a healthy VM built from an older image gets adopted forever and
the new code never boots — every infra change would need an out-of-band
kill to dislodge the stale VM (and races whatever launched next)."""
"""Records the infra-artifact version the *running* pair booted from, so a
later launcher can tell whether the singletons it found are the current
code. Without it, a healthy pair built from an older image gets adopted
forever and the new code never boots — every infra change would need an
out-of-band kill to dislodge the stale VMs (and races whatever launched
next). Both VMs boot the same artifact, so one marker covers the pair."""
return _infra_dir() / "booted-version"
def _expected_version() -> str:
return infra_artifact.infra_artifact_version(_infra_init())
def expected_version() -> str:
"""The combined marker for the running pair: both per-plane artifact
versions, so a change to either rootfs dislodges the adopted pair."""
return " ".join(f"{role}={_role_version(role)}" for role in infra_artifact.ROLES)
def _adoptable(key: Path, url: str, want: str) -> bool:
"""Adopt a running infra VM only if it booted from the CURRENT version and
its control plane is healthy. A missing/mismatched marker means a prior
launcher booted an older infra image — reboot rather than reuse stale code."""
def adoptable(key: Path, url: str, want: str) -> bool:
"""Adopt the running pair only if it booted from the CURRENT version, the
orchestrator's control plane is healthy, and the gateway VM is still alive.
A missing/mismatched marker means a prior launcher booted an older infra
image; a dead gateway means the pair is half-down — reboot both rather than
reuse stale or partial state."""
if not key.exists():
return False
try:
booted = _version_file().read_text(encoding="utf-8").strip()
except OSError:
return False
return booted == want and _health_ok(url)
if booted != want:
return False
if not _health_ok(url):
return False
return _pidfile_alive(_gw_dir())
def _record_booted_version(version: str) -> None:
def record_booted_version(version: str) -> None:
_version_file().write_text(version + "\n", encoding="utf-8")
# The registry "volume": a host-side ext4 file attached to the infra VM as a
# second virtio-block device (guest /dev/vdb), mounted at the control plane's
# DB dir. It outlives the ephemeral rootfs, so the bottle registry survives an
# infra-VM restart — the firecracker analogue of a docker volume. It is a
# plain ext4 file: `sudo mount -o loop <path>` on the host (with the VM
# stopped) to inspect bot-bottle.db directly.
_REGISTRY_SIZE = "512M"
def registry_volume_path() -> Path:
return _infra_dir() / "registry.ext4"
def _ensure_registry_volume() -> Path:
"""Create the empty ext4 registry volume on first use; reuse it after."""
vol = registry_volume_path()
if vol.exists():
return vol
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
proc = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
vol.unlink(missing_ok=True)
die(f"creating registry volume failed: {proc.stderr.strip()}")
return vol
def push_secret(infra: InfraVm, secret: str, dest: str, what: str) -> None:
"""Pipe `secret` to an atomic write of `dest` in the guest over SSH,
retrying while the guest's SSH comes up; die (naming `what`) if it never
succeeds. Bare-pipe input keeps the value off argv."""
push = f"umask 077; cat > {dest}.tmp && mv {dest}.tmp {dest}"
deadline = time.monotonic() + _SECRET_PUSH_TIMEOUT_SECONDS
last = ""
while time.monotonic() < deadline:
proc = subprocess.run(
util.ssh_base_argv(infra.private_key, infra.guest_ip) + [push],
input=secret, capture_output=True, text=True, check=False,
)
if proc.returncode == 0:
return
last = proc.stderr.strip()
time.sleep(_SECRET_PUSH_POLL_SECONDS)
die(f"could not push {what}: {last or '<no stderr>'}")
def _stable_keypair() -> tuple[Path, str]:
"""The infra VM's SSH keypair — generated once and reused, so any later
launcher can SSH in (fetch CA / provision) even though a different process
booted the VM. The pubkey is re-injected on every boot via the cmdline."""
"""The infra VMs' shared SSH keypair — generated once and reused, so any
later launcher can SSH in (seed secrets / fetch CA / provision) even though
a different process booted the VMs. Both VMs get the same pubkey re-injected
on every boot via the cmdline."""
d = _infra_dir()
key, pub = d / "id_ed25519", d / "id_ed25519.pub"
if key.exists() and pub.exists():
@@ -327,11 +322,24 @@ def _stable_keypair() -> tuple[Path, str]:
return key, pub.read_text().strip()
def _kill_pidfile() -> None:
"""SIGTERM (then SIGKILL) the recorded infra VMM, if it's still ours.
def _pidfile_alive(run_dir: Path) -> bool:
"""True iff `run_dir`'s recorded VMM is still a live firecracker (guards a
recycled PID by checking `comm`)."""
try:
pid = int(_pid_file(run_dir).read_text().strip())
except (OSError, ValueError):
return False
try:
return Path(f"/proc/{pid}/comm").read_text().strip() == "firecracker"
except OSError:
return False
def _kill_pidfile(run_dir: Path) -> None:
"""SIGTERM (then SIGKILL) the VMM recorded in `run_dir`, if it's still ours.
Guards against a recycled PID by checking the process is firecracker."""
try:
pid = int(_pid_file().read_text().strip())
pid = int(_pid_file(run_dir).read_text().strip())
except (OSError, ValueError):
return
try:
@@ -352,11 +360,18 @@ def _kill_pidfile() -> None:
def _kill_infra_firecrackers(proc_root: Path = Path("/proc")) -> None:
"""SIGKILL any firecracker VMM whose `--config-file` is this host's infra
config, independent of the PID file — reaps orphans it lost track of so the
orchestrator TAP is free to rebind. Scoped to the infra config path, so the
interactive pool's agent/infra VMs (other config paths) are untouched."""
cfg = str(_infra_dir() / "config.json")
"""SIGKILL any firecracker VMM whose `--config-file` is one of this host's
infra configs (orchestrator or gateway), independent of the PID files —
reaps orphans it lost track of so both links' TAPs are free to rebind.
Also matches the *legacy* pre-split combined-VM config (`<infra>/config.json`)
so a surviving old singleton is cleared off the orchestrator link during the
cutover. Scoped to these infra config paths, so the pool's agent VMs (other
config paths) are untouched."""
cfgs = {
str(_orch_dir() / "config.json"),
str(_gw_dir() / "config.json"),
str(_infra_dir() / "config.json"), # legacy pre-split combined VM
}
for entry in proc_root.iterdir():
if not entry.name.isdigit():
continue
@@ -366,7 +381,7 @@ def _kill_infra_firecrackers(proc_root: Path = Path("/proc")) -> None:
args = (entry / "cmdline").read_bytes().split(b"\0")
except OSError:
continue # process vanished / not ours
if any(a.decode("utf-8", "replace") == cfg for a in args):
if any(a.decode("utf-8", "replace") in cfgs for a in args):
try:
os.kill(int(entry.name), signal.SIGKILL)
except (OSError, ValueError):
@@ -381,77 +396,12 @@ def _health_ok(url: str) -> bool:
return False
class SshGatewayTransport:
"""`GatewayTransport` for the gateway running in the infra VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
# Preserve the source mode (docker cp does): the access-hook is staged
# 0700 and git-http execs it directly — a plain `cat >` would land it
# 0644 and the exec fails with EACCES; keys stay 0600.
mode = stat.S_IMODE(os.stat(src).st_mode)
q = shlex.quote(dest)
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip)
+ [f"cat > {q} && chmod {mode:o} {q}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def gateway_transport() -> SshGatewayTransport:
"""git-gate provisioning transport for the gateway in the infra VM, built
from the stable key + the orchestrator link's guest IP. Needs no live VM
handle, so teardown can use it too."""
return SshGatewayTransport(
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
def wait_for_health(
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
) -> None:
"""Poll the control plane's /health until it answers 200 or the deadline
passes. Dies (with the console tail) if the VMM exits early."""
url = f"{infra.control_plane_url}/health"
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if infra.vm is not None and not infra.vm.is_alive():
die(f"infra VM exited during boot (rc={infra.vm.process.returncode}).\n"
f"{firecracker_vm._console_tail(infra.vm.console_log)}")
try:
with urllib.request.urlopen(url, timeout=1.0) as resp:
if resp.status == 200:
info(f"infra control plane healthy at {infra.control_plane_url}")
return
except (urllib.error.URLError, TimeoutError, OSError):
pass
time.sleep(_HEALTH_POLL_SECONDS)
tail = (firecracker_vm._console_tail(infra.vm.console_log)
if infra.vm is not None else "")
die(f"infra control plane at {url} did not become healthy within "
f"{timeout:.0f}s.\n{tail}")
def _infra_init() -> str:
"""PID-1 init for the infra VM: mount the pseudo-filesystems, wire a
resolver, start dropbear (debug SSH), then launch the control plane and
the gateway data plane (multi-tenant against the local control plane)."""
def _init_head() -> str:
"""The shared PID-1 preamble both role inits open with: mount the pseudo-
filesystems, export a real PATH (a bare-init shell's built-in exec path
isn't in the *environment*, so backgrounded `python3 ...` children would
find no PATH), set the direct upstream resolver, install the per-boot SSH
pubkey from the cmdline, and start dropbear for debug/provisioning."""
return f"""#!/bin/sh
# bot-bottle Firecracker infra VM init (PID 1).
mount -t proc proc /proc 2>/dev/null
@@ -460,10 +410,6 @@ mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Export a real PATH: a bare-init shell resolves its own execs via a
# built-in default path, but that isn't in the *environment*, so
# gateway_init's subprocess daemons (spawned as `python3 ...`) would
# inherit no PATH and fail to find python3. Export it for all children.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Direct upstream resolver (control-plane / gateway egress + buildah).
@@ -479,26 +425,72 @@ fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run /var/lib/bot-bottle
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted
# at the control plane's DB dir, so bot-bottle.db survives infra-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
/bb-dropbear -R -E -p 22 &
# Control plane. Source is baked at /app; the package is stdlib-only.
cd /app
BOT_BOTTLE_ROOT=/var/lib/bot-bottle python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {CONTROL_PLANE_PORT} --broker stub &
"""
# Gateway data plane, multi-tenant: each request resolves source-IP ->
# policy against the local control plane. The VM backend reaches git over
# git-http (9420), so the git:// daemon (git-gate, needs a per-bottle
# entrypoint the consolidated model doesn't use) is left out.
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{CONTROL_PLANE_PORT} \\
SUPERVISE_DB_PATH=/var/lib/bot-bottle/db/bot-bottle.db \\
python3 -m bot_bottle.gateway_init &
_INIT_TAIL = """
# Reap as PID 1; children are backgrounded, so `wait` blocks.
while : ; do wait ; done
"""
def _gateway_init() -> str:
"""PID-1 init for the gateway (data-plane) VM. Waits for the host-seeded
`gateway` JWT, then starts ONLY the data-plane daemons, multi-tenant against
the orchestrator at the `bb_orch` cmdline address. The VM backend reaches git
over git-http (9420), so the git:// daemon (a per-bottle entrypoint the
consolidated model doesn't use) is left out. No SUPERVISE_DB_PATH: the data
plane reaches the supervise queue over the control-plane RPC and never opens
bot-bottle.db (PRD 0070 / #469). If the JWT never arrives, REFUSE to start
rather than run without auth."""
return _init_head() + f"""
ORCH=$(sed -n 's/.*bb_orch=\\([^ ]*\\).*/\\1/p' /proc/cmdline)
GW_JWT=""
i=0
while [ "$i" -lt 600 ]; do
GW_JWT=$(cat {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null)
[ -n "$GW_JWT" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$GW_JWT" ]; then
echo "infra gateway: gateway JWT never arrived; refusing to start the data plane" >&2
else
chmod 600 {_GUEST_GATEWAY_JWT_PATH} 2>/dev/null || true
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://$ORCH:{ORCHESTRATOR_PORT} \\
BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT="$GW_JWT" \\
python3 -m bot_bottle.gateway.bootstrap &
fi
""" + _INIT_TAIL
def _orchestrator_init() -> str:
"""PID-1 init for the orchestrator (control-plane) VM. Mounts the persistent
registry volume (/dev/vdb — bot-bottle.db survives a VM restart), waits for
the host-seeded signing key, then starts ONLY the control plane. If the key
never arrives, REFUSE to start rather than run OPEN — open mode would grant
every unauthenticated caller the `cli` role (#469)."""
return _init_head() + f"""
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted at
# the DB dir, so bot-bottle.db survives orchestrator-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
CP_KEY=""
i=0
while [ "$i" -lt 600 ]; do
CP_KEY=$(cat {_GUEST_SIGNING_KEY_PATH} 2>/dev/null)
[ -n "$CP_KEY" ] && break
i=$((i + 1))
sleep 0.1
done
if [ -z "$CP_KEY" ]; then
echo "infra: control-plane signing key never arrived; refusing to start the control plane (would run OPEN)" >&2
else
chmod 600 {_GUEST_SIGNING_KEY_PATH} 2>/dev/null || true
BOT_BOTTLE_ROOT=/var/lib/bot-bottle BOT_BOTTLE_ORCHESTRATOR_TOKEN="$CP_KEY" python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {ORCHESTRATOR_PORT} --broker stub &
fi
""" + _INIT_TAIL
+25 -24
View File
@@ -11,15 +11,16 @@ Per bottle:
6. provision (shared gateway CA, prompt, skills, workspace, git, supervise)
over SSH.
The per-bottle Docker sidecar bundle is gone. The shared gateway handles
egress / git-gate / supervise for every VM; Docker's PREROUTING DNAT routes
the VMs' traffic to it, and the nft table's `ct status dnat accept` rule
in the forward chain lets it pass. The VM still sends to `host_tap_ip:PORT`
— the address its world is, by nft design, limited to.
The per-bottle Docker sidecar bundle is gone. The shared gateway (its own
data-plane VM) handles egress / git-gate / supervise for every VM; the nft
netpool's PREROUTING DNAT routes the VMs' gateway-port traffic to that gateway
VM, and the nft table's `ct status dnat accept` rule in the forward chain lets
it pass. The VM still sends to `host_tap_ip:PORT` — the address its world is,
by nft design, limited to.
Isolation is enforced by the operator-provisioned nft table (checked
fail-closed in preflight): a VM reaches only the sidecar (DNAT'd from
the host TAP IP) and nothing else.
fail-closed in preflight): a VM reaches only the gateway (DNAT'd from the host
TAP IP) and nothing else — not even the orchestrator control plane.
"""
from __future__ import annotations
@@ -38,26 +39,22 @@ from ...bottle_state import (
git_gate_state_dir,
read_committed_image,
)
from ...egress import (
egress_agent_env_entries,
egress_resolve_token_values,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...egress import Egress
from ...git_gate import GitGate
from ...image_cache import check_stale_path
from ...log import die, info, warn
from ...supervise import SUPERVISE_PORT
from ...supervisor.types import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
from ...orchestrator.config_store import resolve_teardown_timeout
from ...orchestrator.store.config_store import resolve_teardown_timeout
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from .consolidated_launch import (
launch_consolidated,
teardown_consolidated,
persist_env_var_secret,
deprovision_consolidated,
)
@@ -84,7 +81,7 @@ def launch(
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"firecracker teardown failed: {exc!r}")
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
GitGate().revoke_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
@@ -95,7 +92,7 @@ def launch(
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
git_gate_plan = GitGate().provision_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
@@ -111,7 +108,7 @@ def launch(
# (in memory) for the gateway to inject — the agent never sees them.
# Attribution is by the VM's guest IP (unspoofable via /31 TAP + nft).
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
token_values = Egress().resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
@@ -122,7 +119,7 @@ def launch(
tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
deprovision_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
@@ -140,7 +137,7 @@ def launch(
)
# Point the agent's git-gate insteadOf rewrites and supervise MCP URL
# at the shared gateway (reached at the slot's host TAP IP — the VM
# sends there and Docker DNAT routes to the gateway container).
# sends there and the nft netpool DNAT routes to the gateway VM).
git_gate_url = (
f"http://{slot.host_ip}:{_GIT_HTTP_PORT}" if git_gate_plan.upstreams else ""
)
@@ -153,6 +150,7 @@ def launch(
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
identity_token=ctx.identity_token,
env_var_secret=ctx.env_var_secret,
# Deliver the identity token as egress proxy credentials — clients
# honor `HTTPS_PROXY=http://id:token@gw` without app changes; the
# gateway reads Proxy-Authorization, validates the (source_ip,
@@ -187,6 +185,7 @@ def launch(
)
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
persist_env_var_secret(private_key, slot.guest_ip, ctx.env_var_secret)
# Authoritative fail-closed egress-boundary check, before the agent
# runs: prove the VM cannot reach the host directly.
@@ -281,7 +280,9 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
env["GIT_GATE_URL"] = plan.agent_git_gate_url
if plan.agent_supervise_url:
env["MCP_SUPERVISE_URL"] = plan.agent_supervise_url
for entry in egress_agent_env_entries(plan.egress_plan):
if plan.env_var_secret:
env[ENV_VAR_SECRET_NAME] = plan.env_var_secret
for entry in Egress().agent_env_entries(plan.egress_plan):
key, _, value = entry.partition("=")
env[key] = value
env.update(plan.agent_provision.guest_env)
@@ -15,11 +15,19 @@ BOT_BOTTLE_FC_POOL_SIZE=8
BOT_BOTTLE_FC_IP_BASE=10.243.0.0
BOT_BOTTLE_FC_IFACE_PREFIX=bbfc
BOT_BOTTLE_FC_NFT_TABLE=bot_bottle_fc
# The orchestrator/gateway VM's own TAP — a dedicated link OUTSIDE the
# bbfc* agent pool. Unlike agent VMs (which reach only their gateway),
# the orchestrator is trusted infra that needs real NAT'd internet
# egress: to FROM-pull + apt/npm during in-VM agent-image builds
# (buildah) and to forward agent egress upstream (Stage B gateway). Its
# The orchestrator VM's own TAP — a dedicated link OUTSIDE the bbfc*
# agent pool. Unlike agent VMs (which reach only their gateway), the
# orchestrator is trusted infra that needs real NAT'd internet egress:
# to FROM-pull + apt/npm during in-VM agent-image builds (buildah). Its
# /31 is the top of the IP_BASE /16 (host x.y.255.0, guest x.y.255.1),
# clear of the pool near the bottom of the block.
BOT_BOTTLE_FC_ORCH_IFACE=bborch0
# The gateway (data-plane) VM's own TAP — the second infra link, split
# from the orchestrator now that #469 got the DB off the data plane (PRD
# 0070 "Separating the planes"). It mirrors the orchestrator link: NAT'd
# internet egress (the gateway forwards agent egress upstream) plus a
# forward path to reach the orchestrator's control plane. Its /31 is the
# next one above the orchestrator link (host x.y.255.2, guest x.y.255.3).
# Agents DNAT their gateway-port traffic here (not to the orchestrator),
# so a breached agent has no L3 route to the control plane.
BOT_BOTTLE_FC_GW_IFACE=bbgw0
+35 -9
View File
@@ -79,12 +79,19 @@ def _cfg(key: str) -> str:
IFACE_PREFIX = _cfg("BOT_BOTTLE_FC_IFACE_PREFIX")
NFT_TABLE = _cfg("BOT_BOTTLE_FC_NFT_TABLE")
# The orchestrator/gateway VM's dedicated TAP — outside the bbfc* agent
# pool and, unlike it, NAT'd to the internet (see `orch_slot`). The
# The orchestrator VM's dedicated TAP — outside the bbfc* agent pool
# and, unlike it, NAT'd to the internet (see `orch_slot`). The
# orchestrator is trusted infra: it builds agent images in-VM (buildah
# needs to FROM-pull + apt/npm) and forwards agent egress upstream.
# needs to FROM-pull + apt/npm).
ORCH_IFACE = _cfg("BOT_BOTTLE_FC_ORCH_IFACE")
# The gateway (data-plane) VM's dedicated TAP — the second infra link
# (see `gw_slot`), split from the orchestrator per PRD 0070. Like the
# orchestrator link it is NAT'd to the internet (the gateway forwards
# agent egress upstream); unlike it, agents DNAT here, never to the
# orchestrator, so a breached agent has no L3 route to the control plane.
GW_IFACE = _cfg("BOT_BOTTLE_FC_GW_IFACE")
def pool_size() -> int:
return int(_cfg("BOT_BOTTLE_FC_POOL_SIZE"))
@@ -130,12 +137,12 @@ def all_slots() -> list[Slot]:
def orch_slot() -> Slot:
"""The orchestrator/gateway VM's dedicated link — its own TAP
(`ORCH_IFACE`) on a /31 at the TOP of the IP_BASE /16 (host
x.y.255.0, guest x.y.255.1), well clear of the agent pool near the
bottom of the block. Unlike a pool `Slot`, this link is NAT'd out to
the internet by the setup (the orchestrator is trusted infra), so it
is deliberately *not* one of the isolated `bbfc*` slots.
"""The orchestrator VM's dedicated link — its own TAP (`ORCH_IFACE`)
on a /31 at the TOP of the IP_BASE /16 (host x.y.255.0, guest
x.y.255.1), well clear of the agent pool near the bottom of the
block. Unlike a pool `Slot`, this link is NAT'd out to the internet
by the setup (the orchestrator is trusted infra), so it is
deliberately *not* one of the isolated `bbfc*` slots.
`index` is -1 (sentinel: not a pool index)."""
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
@@ -148,6 +155,25 @@ def orch_slot() -> Slot:
)
def gw_slot() -> Slot:
"""The gateway (data-plane) VM's dedicated link — its own TAP
(`GW_IFACE`) on the /31 immediately above the orchestrator link (host
x.y.255.2, guest x.y.255.3), still clear of the agent pool at the
bottom of the block. Mirrors `orch_slot`: NAT'd to the internet (the
gateway forwards agent egress upstream), not one of the isolated
`bbfc*` slots. Agents DNAT their gateway-port traffic to this guest.
`index` is -2 (sentinel: not a pool index)."""
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
host = base16 + 0xFF02
return Slot(
index=-2,
iface=GW_IFACE,
host_ip=str(ipaddress.IPv4Address(host)),
guest_ip=str(ipaddress.IPv4Address(host + 1)),
)
# --- fail-closed verification ---------------------------------------
def _run_ok(argv: list[str]) -> bool:
@@ -0,0 +1,158 @@
"""The Firecracker orchestrator (control plane) as a microVM (PRD 0070).
`FirecrackerOrchestrator` is the Firecracker implementation of the backend-neutral
`Orchestrator` service, and owns the control plane's host-side logic directly:
booting the orchestrator microVM on its NAT'd link with the persistent registry
volume (/dev/vdb the sole opener of `bot-bottle.db`), seeding the host-canonical
signing key over SSH, and waiting for `/health`. The host CLI reaches it at its
guest IP, and so does the gateway (`bb_orch` cmdline), so `url()` == `gateway_url()`.
The plane-agnostic VM substrate the gateway VM also uses stays in `infra_vm`
booting a VM from a per-plane rootfs (`boot_vm`), the stable SSH keypair, the
secret-push retry, the PID lifecycle, the adoption/version helpers, and the
per-plane guest inits (`role_init`). The pair coordinator (adopt-or-boot-both
under a singleton flock) is `FirecrackerInfraService` (`infra.py`).
"""
from __future__ import annotations
import subprocess
import time
from pathlib import Path
from ...log import die, info
from ...paths import host_orchestrator_token
from ...orchestrator.lifecycle import (
DEFAULT_STARTUP_TIMEOUT_SECONDS,
Orchestrator,
)
from . import firecracker_vm, infra_vm, netpool
from .infra_vm import ORCHESTRATOR_PORT
# The orchestrator microVM's name (its run dir). Fixed per host — one
# control-plane VM.
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
# Memory ceiling (fixed at boot, demand-paged). The orchestrator keeps the build
# headroom (buildah's 2-4 GB working set during in-VM agent builds) — PRD 0070
# "Memory: fixed ceilings".
_ORCH_MEM_MIB = 4096
_HEALTH_POLL_SECONDS = 0.5
# The registry "volume": a host-side ext4 file attached to the orchestrator VM as
# a second virtio-block device (guest /dev/vdb), mounted at the control plane's DB
# dir. It outlives the ephemeral rootfs, so the bottle registry survives an
# orchestrator-VM restart — the firecracker analogue of a docker volume. A plain
# ext4 file: `sudo mount -o loop <path>` on the host (VM stopped) to inspect
# bot-bottle.db. The gateway VM has no such volume — the data plane never opens
# the DB (#469).
_REGISTRY_SIZE = "512M"
def registry_volume_path() -> Path:
return infra_vm._infra_dir() / "registry.ext4"
class FirecrackerOrchestrator(Orchestrator):
"""The control plane as a Firecracker microVM on the orchestrator link.
The orchestrator rootfs is built/downloaded by `infra_vm.ensure_built` (the
ABC's `ensure_built` no-op here); `ensure_running` boots the VM, seeds the
signing key, and blocks until `/health` answers."""
name = ORCHESTRATOR_NAME
def __init__(self, vm: infra_vm.InfraVm | None = None) -> None:
# The live VM handle when this process booted it; None when adapting an
# already-running orchestrator (health/URL go through the fixed link).
self._vm = vm
def url(self) -> str:
"""The orchestrator VM's guest IP URL — where the host CLI reaches the
control plane."""
return f"http://{netpool.orch_slot().guest_ip}:{ORCHESTRATOR_PORT}"
def gateway_url(self) -> str:
"""Same guest IP the host uses — the gateway resolves the orchestrator at
`bb_orch=<guest_ip>` off its cmdline."""
return self.url()
def ssh_target(self) -> tuple[Path, str]:
"""(private key, guest IP) for SSHing into the orchestrator VM. In-VM
agent-image builds (buildah) run here the build host drives them over
SSH with the stable infra key."""
return infra_vm._infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip
def is_running(self) -> bool:
return infra_vm._pidfile_alive(infra_vm._orch_dir())
def stop(self) -> None:
"""Stop only the orchestrator VM (idempotent). A dead orchestrator fails
`infra_vm.adoptable`'s health check, so no version marker to clear."""
infra_vm._kill_pidfile(infra_vm._orch_dir())
infra_vm._pid_file(infra_vm._orch_dir()).unlink(missing_ok=True)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> None:
"""Boot the orchestrator VM on its link with the persistent registry
volume, seed the signing key over SSH, and block until `/health` answers.
Idempotent a live, healthy control plane is left alone (the pair
coordinator otherwise `stop()`s it before calling, so this boots fresh)."""
if self.is_running() and self.is_healthy():
return
vm = infra_vm.boot_vm(
name=ORCHESTRATOR_NAME, slot=netpool.orch_slot(),
run_dir=infra_vm._orch_dir(), role="orchestrator", mem_mib=_ORCH_MEM_MIB,
data_drive=self._ensure_registry_volume(),
)
# Push the host-canonical signing key (the init waits for it before
# starting the control plane). The host token file stays the single
# source of truth, so a co-running docker/macOS control plane keeps
# working; the guest verifies tokens with the same key the CLI signs from.
infra_vm.push_secret(
vm, host_orchestrator_token(), infra_vm._GUEST_SIGNING_KEY_PATH,
"the control-plane signing key to the orchestrator VM "
"(its control plane will not start)",
)
self._vm = vm
self._wait_for_health(vm, timeout=startup_timeout)
def _wait_for_health(
self, vm: infra_vm.InfraVm, *, timeout: float,
) -> None:
"""Poll `/health` until it answers 200 or the deadline passes. Dies (with
the console tail) if the VMM exits early."""
url = f"{self.url()}/health"
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if vm.vm is not None and not vm.vm.is_alive():
die(f"orchestrator VM exited during boot (rc={vm.vm.process.returncode}).\n"
f"{firecracker_vm._console_tail(vm.vm.console_log)}")
if self.is_healthy():
info(f"orchestrator control plane healthy at {self.url()}")
return
time.sleep(_HEALTH_POLL_SECONDS)
tail = (firecracker_vm._console_tail(vm.vm.console_log)
if vm.vm is not None else "")
die(f"orchestrator control plane at {url} did not become healthy within "
f"{timeout:.0f}s.\n{tail}")
def _ensure_registry_volume(self) -> Path:
"""Create the empty ext4 registry volume on first use; reuse it after."""
vol = registry_volume_path()
if vol.exists():
return vol
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
proc = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
vol.unlink(missing_ok=True)
die(f"creating registry volume failed: {proc.stderr.strip()}")
return vol
__all__ = ["FirecrackerOrchestrator", "ORCHESTRATOR_NAME", "registry_volume_path"]
+96 -78
View File
@@ -1,20 +1,22 @@
"""Build the infra rootfs and publish it as a Gitea generic package.
"""Build the infra rootfs artifacts and publish them as Gitea generic packages.
The off-host (build / CI) half of PRD 0069 Stage 2: this DOES use Docker, but
never on the launch host. It runs the same pipeline the launch host used to run
locally `docker build` the three fixed images, export to a rootfs dir, inject
the guest boot, `mke2fs` to an ext4 with the buildah build slack then gzips
the ext4 and PUTs it (plus a `.sha256`) to
`/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<version>/`.
locally `docker build` the fixed images, export each per-plane rootfs, inject
the guest boot, `mke2fs` to an ext4 then gzips each and PUTs it (plus a
`.sha256`) to `/api/packages/<owner>/generic/bot-bottle-firecracker-<role>/<version>/`.
The `<version>` is `infra_artifact.infra_artifact_version(...)`, the content
hash of the rootfs inputs, so a launch host at the same code checkout resolves
the exact artifact this produced.
There are two artifacts, one per plane (`orchestrator`, `gateway`); the
orchestrator rootfs carries buildah, the gateway rootfs is slim. Each
`<version>` is `infra_artifact.infra_artifact_version(...)`, the content hash of
that rootfs's inputs, so a launch host at the same code checkout resolves the
exact artifacts this produced.
python3 -m bot_bottle.backend.firecracker.publish_infra --output DIR
python3 -m bot_bottle.backend.firecracker.publish_infra --publish-dir DIR
Auth: a token with `write:package` on the target owner, from
A candidate bundle holds each role under its own `DIR/<role>/` subdir. Auth: a
token with `write:package` on the target owner, from
`BOT_BOTTLE_INFRA_ARTIFACT_TOKEN`.
"""
@@ -33,17 +35,27 @@ from . import infra_artifact, infra_vm, util
_CHUNK = 1 << 20
# A human-readable description shipped alongside the artifact — generic packages
_GZ_NAME = "rootfs.ext4.gz"
_SHA_NAME = "rootfs.ext4.gz.sha256"
# A human-readable description shipped alongside each artifact — generic packages
# have no description field, so this file *is* the description on the package
# page. Uploaded on every publish so it never goes stale.
_ABOUT_NAME = "about.txt"
_ABOUT_TEXT = (
"bot-bottle infra rootfs for the Firecracker backend (PRD 0069 Stage 2, "
"#348): the per-host infra VM (orchestrator control plane + gateway + "
"buildah). Prebuilt off-host, gzip ext4; the launch host downloads + "
"sha256-verifies + boots it, no host Docker. The version tag is a content "
"hash of the rootfs inputs. Files: rootfs.ext4.gz + rootfs.ext4.gz.sha256.\n"
)
def _about_text(role: str) -> str:
return (
f"bot-bottle firecracker {role} rootfs (PRD 0069 Stage 2 / PRD 0070): "
f"the per-host {role} infra VM. Prebuilt off-host, gzip ext4; the launch "
f"host downloads + sha256-verifies + boots it, no host Docker. The "
f"version tag is a content hash of the rootfs inputs. Files: "
f"{_GZ_NAME} + {_SHA_NAME}.\n"
)
def _role_version(role: str) -> str:
return infra_artifact.infra_artifact_version(infra_vm.role_init(role), role)
def _gzip(src: Path, dest: Path) -> None:
@@ -109,34 +121,35 @@ def _delete(url: str, token: str) -> None:
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
def build_artifact(out_dir: Path) -> tuple[str, Path, Path]:
"""Build the infra rootfs ext4, gzip it, and write the checksum. Returns
`(version, gz_path, sha_path)`. Uses host Docker (off-host / CI)."""
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
print(f"building infra rootfs artifact {version} (docker)")
infra_vm.build_infra_images_with_docker()
base = infra_vm.build_infra_rootfs_dir()
def build_role_artifact(role: str, role_dir: Path) -> str:
"""Build `role`'s rootfs ext4, gzip it, and write the checksum + version into
`role_dir`. Returns the version. Assumes the docker images are already
built (`infra_vm.build_infra_images_with_docker`). Uses host Docker."""
version = _role_version(role)
print(f"building {role} rootfs artifact {version} (docker)")
base = infra_vm.build_rootfs_dir(role)
ext4 = out_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, ext4, slack_mib=8192)
gz = out_dir / "rootfs.ext4.gz"
print("compressing rootfs")
ext4 = role_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, ext4, slack_mib=infra_vm._ROOTFS_SLACK_MIB[role])
gz = role_dir / _GZ_NAME
print(f"compressing {role} rootfs")
_gzip(ext4, gz)
ext4.unlink(missing_ok=True)
sha = out_dir / "rootfs.ext4.gz.sha256"
sha = role_dir / _SHA_NAME
digest = _sha256(gz)
sha.write_text(f"{digest} rootfs.ext4.gz\n")
print(f" {gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
return version, gz, sha
sha.write_text(f"{digest} {_GZ_NAME}\n")
(role_dir / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f" {role}/{gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
return version
def _try_download_published(out_dir: Path) -> tuple[str, Path, Path] | None:
"""If this version's artifact is already in the registry, download the gz
and sha to out_dir and return (version, gz_path, sha_path). Returns None
when not yet published."""
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
sha_url = infra_artifact.artifact_url(version, "rootfs.ext4.gz.sha256")
def _try_download_published(role: str, role_dir: Path) -> str | None:
"""If `role`'s artifact for this version is already in the registry, download
the gz + sha into `role_dir` and return the version. None when not yet
published."""
version = _role_version(role)
sha_url = infra_artifact.artifact_url(version, _SHA_NAME, role=role)
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)):
pass
@@ -146,70 +159,70 @@ def _try_download_published(out_dir: Path) -> tuple[str, Path, Path] | None:
raise SystemExit(f"registry check failed (HTTP {e.code}): {sha_url}")
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {sha_url} ({e.reason})")
print(f"infra rootfs {version} already published — downloading instead of building")
gz = out_dir / "rootfs.ext4.gz"
sha = out_dir / "rootfs.ext4.gz.sha256"
infra_artifact._download(infra_artifact.artifact_url(version, "rootfs.ext4.gz"), gz)
infra_artifact._download(infra_artifact.artifact_url(version, "rootfs.ext4.gz.sha256"), sha)
return version, gz, sha
print(f"{role} rootfs {version} already published — downloading instead of building")
infra_artifact._download(
infra_artifact.artifact_url(version, _GZ_NAME, role=role), role_dir / _GZ_NAME)
infra_artifact._download(sha_url, role_dir / _SHA_NAME)
(role_dir / "version.txt").write_text(version + "\n", encoding="utf-8")
return version
def _publish_bundle(root: Path, token: str) -> str:
version_file = root / "version.txt"
def _publish_bundle(role: str, role_dir: Path, token: str) -> str:
version_file = role_dir / "version.txt"
# Guard the read so a missing version.txt is a clean error, not a raw
# FileNotFoundError.
if not version_file.is_file():
raise SystemExit(f"incomplete artifact bundle: {root}")
raise SystemExit(f"incomplete {role} artifact bundle: {role_dir}")
version = version_file.read_text(encoding="utf-8").strip()
expected = infra_artifact.infra_artifact_version(infra_vm._infra_init())
expected = _role_version(role)
if version != expected:
raise SystemExit(
f"artifact bundle version {version!r} does not match checkout {expected!r}"
f"{role} artifact bundle version {version!r} does not match checkout {expected!r}"
)
gz = root / "rootfs.ext4.gz"
sha = root / "rootfs.ext4.gz.sha256"
gz = role_dir / _GZ_NAME
sha = role_dir / _SHA_NAME
if not gz.is_file() or not sha.is_file():
raise SystemExit(f"incomplete artifact bundle: {root}")
raise SystemExit(f"incomplete {role} artifact bundle: {role_dir}")
expected_sha = sha.read_text().split()[0].strip().lower()
if _sha256(gz) != expected_sha:
raise SystemExit("artifact bundle checksum mismatch")
raise SystemExit(f"{role} artifact bundle checksum mismatch")
gz_url = infra_artifact.artifact_url(version, gz.name)
sha_url = infra_artifact.artifact_url(version, sha.name)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME)
gz_url = infra_artifact.artifact_url(version, _GZ_NAME, role=role)
sha_url = infra_artifact.artifact_url(version, _SHA_NAME, role=role)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME, role=role)
# Publishing is idempotent. If this exact complete artifact is already
# present, a test-only main commit is a no-op. Otherwise clear any partial
# upload left by an interrupted prior attempt and upload the complete set.
# present, a re-publish is a no-op. Otherwise clear any partial upload left
# by an interrupted prior attempt and upload the complete set.
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)) as resp:
remote_sha = resp.read().decode("utf-8").split()[0].strip().lower()
except urllib.error.HTTPError as e:
if e.code != 404:
raise SystemExit(f"checking existing artifact failed (HTTP {e.code})")
raise SystemExit(f"checking existing {role} artifact failed (HTTP {e.code})")
remote_sha = ""
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {sha_url} ({e.reason})")
if remote_sha == expected_sha:
print(f"infra rootfs {version} already published")
print(f"{role} rootfs {version} already published")
return version
for url in (gz_url, sha_url, about_url):
_delete(url, token)
_put(gz_url, gz, token)
_put(sha_url, sha.read_bytes(), token)
_put(about_url, _ABOUT_TEXT.encode(), token)
_put(about_url, _about_text(role).encode(), token)
return version
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="publish_infra", description="Build + publish the infra rootfs artifact.")
prog="publish_infra", description="Build + publish the infra rootfs artifacts.")
mode = parser.add_mutually_exclusive_group(required=True)
mode.add_argument("--output", type=Path,
help="build a candidate bundle in DIR without publishing")
help="build candidate bundles in DIR/<role>/ without publishing")
mode.add_argument("--publish-dir", type=Path,
help="publish an already-built and tested candidate bundle")
help="publish already-built + tested candidate bundles under DIR")
parser.add_argument("--reuse-published", action="store_true",
help="with --output: download from registry if already published instead of building")
args = parser.parse_args(argv)
@@ -221,23 +234,28 @@ def main(argv: list[str] | None = None) -> int:
"with write:package")
if args.output is not None:
args.output.mkdir(parents=True, exist_ok=True)
reused = None
if args.reuse_published:
reused = _try_download_published(args.output)
if reused is not None:
version, _, _ = reused
(args.output / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f"reused published infra rootfs candidate {version}")
return 0
version, _gz, _sha = build_artifact(args.output)
(args.output / "version.txt").write_text(version + "\n", encoding="utf-8")
print(f"built infra rootfs candidate {version}")
# Build (or reuse) all roles. Images are built once, up front, only when
# something actually needs building.
pending = []
for role in infra_artifact.ROLES:
role_dir = args.output / role
role_dir.mkdir(parents=True, exist_ok=True)
if args.reuse_published and _try_download_published(role, role_dir):
print(f"reused published {role} rootfs candidate")
continue
pending.append(role)
if pending:
print("building infra images (docker)")
infra_vm.build_infra_images_with_docker()
for role in pending:
build_role_artifact(role, args.output / role)
print(f"built {role} rootfs candidate")
return 0
assert args.publish_dir is not None
version = _publish_bundle(args.publish_dir, token)
print(f"published infra rootfs {version}")
for role in infra_artifact.ROLES:
version = _publish_bundle(role, args.publish_dir / role, token)
print(f"published {role} rootfs {version}")
return 0
@@ -9,7 +9,7 @@ from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...manifest import Manifest
from ...supervise import SupervisePlan
from ...supervisor.plan import SupervisePlan
from .. import BottleSpec
from . import util
from .bottle_plan import FirecrackerBottlePlan
+86 -6
View File
@@ -13,6 +13,7 @@ generic `./cli.py backend {setup,status}` command dispatches to.
from __future__ import annotations
import fcntl
import os
import shutil
import subprocess
@@ -22,6 +23,9 @@ from pathlib import Path
from . import netpool
from . import util
# KVM_GET_API_VERSION = _IO(KVMIO=0xAE, 0x00): cheapest proof of KVM access.
_KVM_GET_API_VERSION = 0xAE00
_FC_RELEASES = "https://github.com/firecracker-microvm/firecracker/releases"
_UNIT_PATH = Path("/etc/systemd/system") / netpool.SYSTEMD_UNIT
@@ -219,14 +223,90 @@ def teardown() -> int:
return 0
def _firecracker_binary_ok() -> bool:
"""True iff the firecracker binary is on PATH and `--version` exits 0."""
if shutil.which("firecracker") is None:
return False
try:
return subprocess.run(
["firecracker", "--version"],
capture_output=True, check=False, timeout=5,
).returncode == 0
except (OSError, subprocess.TimeoutExpired):
return False
def _kvm_accessible() -> bool:
"""True iff /dev/kvm can be opened read-write and responds to KVM_GET_API_VERSION.
VM creation requires write access; opening read-only may satisfy the
ioctl but fails at boot time, so O_RDWR is the permission check."""
if not os.path.exists(util._KVM_DEVICE):
return False
try:
fd = os.open(util._KVM_DEVICE, os.O_RDWR | os.O_CLOEXEC)
try:
fcntl.ioctl(fd, _KVM_GET_API_VERSION)
finally:
os.close(fd)
return True
except OSError:
return False
def status() -> int:
# Readiness == what the launch preflight hard-requires: the TAP pool
# present (unprivileged, authoritative) and no range overlap. Listing
# the nft table usually needs root, so — like the preflight — an
# unconfirmable table is reported but NOT treated as not-ready; the
# post-boot isolation probe is the authoritative check. This keeps an
# unprivileged `backend status` usable as a launch gate.
# Readiness == what the launch preflight hard-requires: the binary
# executable, /dev/kvm accessible, the TAP pool present, and no range
# overlap. Listing the nft table usually needs root, so — like the
# preflight — an unconfirmable table is reported but NOT treated as
# not-ready; the post-boot isolation probe is the authoritative check.
# This keeps an unprivileged `backend status` usable as a launch gate.
ok = True
if _firecracker_binary_ok():
sys.stderr.write(f"firecracker binary: ok ({shutil.which('firecracker')})\n")
else:
fc_path = shutil.which("firecracker")
if fc_path is None:
sys.stderr.write("firecracker binary: NOT found on PATH\n")
else:
sys.stderr.write(
f"firecracker binary: found ({fc_path}) but `--version` failed\n"
)
ok = False
if _kvm_accessible():
sys.stderr.write(f"KVM: {util._KVM_DEVICE} accessible\n")
else:
if not os.path.exists(util._KVM_DEVICE):
sys.stderr.write(f"KVM: {util._KVM_DEVICE} not present\n")
else:
sys.stderr.write(
f"KVM: {util._KVM_DEVICE} not accessible (open/ioctl failed)\n"
)
ok = False
kernel = util.kernel_path()
if kernel.is_file():
sys.stderr.write(f"guest kernel: {kernel}\n")
else:
sys.stderr.write(
f"guest kernel: NOT found at {kernel} "
f"(set BOT_BOTTLE_FC_KERNEL or cache a vmlinux there)\n"
)
ok = False
dropbear = util.dropbear_path()
if dropbear.is_file():
sys.stderr.write(f"dropbear: {dropbear}\n")
else:
sys.stderr.write(
f"dropbear: NOT found at {dropbear} "
f"(set BOT_BOTTLE_FC_DROPBEAR or cache a static binary)\n"
)
ok = False
mke2fs = shutil.which("mke2fs")
if mke2fs is not None:
sys.stderr.write(f"mke2fs: {mke2fs}\n")
else:
sys.stderr.write("mke2fs: NOT found on PATH (install e2fsprogs)\n")
ok = False
missing = netpool.missing_taps()
total = netpool.pool_size()
if missing:
+5 -2
View File
@@ -258,8 +258,8 @@ def build_committed_rootfs_dir(tar_path: Path) -> Path:
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs.
`init_script` defaults to the SSH-only agent init; the infra VM
passes its own (control plane + gateway) init.
`init_script` defaults to the SSH-only agent init; each infra VM
passes its own per-plane init (orchestrator or gateway).
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
may already exist as symlinks aimed at a host file (e.g. bb-init ->
@@ -399,6 +399,9 @@ fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run
# Keep restart-recovery key material memory-backed, separate from both the
# agent rootfs and the infra VM's persistent registry volume.
mount -t tmpfs -o mode=0755 tmpfs /run 2>/dev/null || true
# -R: generate host keys on demand. -E: log auth failures to stderr,
# captured in the host-side console.log for debugging.
/bb-dropbear -R -E -p 22 &
+58
View File
@@ -0,0 +1,58 @@
"""The per-host infra service contract (PRD 0070).
`InfraService` is the backend-neutral composer of the per-host **pair**: the
`Orchestrator` (control plane) + `Gateway` (data plane) services, brought up as
an idempotent per-host singleton. One concrete impl per backend
(`backend/*/infra.py`), mirroring how `Orchestrator` and `Gateway` each have a
per-backend impl.
The two backend-specific accessors (`orchestrator()` / `gateway()`) are the
source of truth for how to reach each plane; the convenience `url()` /
`is_healthy()` just delegate to the orchestrator.
"""
from __future__ import annotations
import abc
from ..gateway import Gateway
from ..orchestrator.lifecycle import DEFAULT_STARTUP_TIMEOUT_SECONDS, Orchestrator
class InfraService(abc.ABC):
"""Compose + bring up the per-host orchestrator + gateway pair.
Backend-neutral: docker/macOS run the pair as two containers, firecracker as
two microVMs. Callers read reach-info off `orchestrator()` / `gateway()`."""
@abc.abstractmethod
def orchestrator(self) -> Orchestrator:
"""The control-plane service. Cheap to reconstruct."""
@abc.abstractmethod
def gateway(self) -> Gateway:
"""The data-plane service. Cheap to reconstruct."""
@abc.abstractmethod
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Bring the orchestrator + gateway pair up (idempotent per-host
singleton a healthy, current pair is left running). Returns the host
control-plane URL. Raises `OrchestratorStartError` on control-plane
startup timeout."""
@abc.abstractmethod
def stop(self) -> None:
"""Remove both the orchestrator and the gateway. Idempotent."""
def url(self) -> str:
"""The host-facing control-plane URL the CLI reaches (the orchestrator's)."""
return self.orchestrator().url()
def is_healthy(self) -> bool:
"""True iff the control plane answers `/health`."""
return self.orchestrator().is_healthy()
__all__ = ["InfraService"]
+23 -4
View File
@@ -1,10 +1,29 @@
"""macOS Apple Container backend.
Selectable via `BOT_BOTTLE_BACKEND=macos-container`. This package owns
the Apple `container` CLI integration; launch remains gated until the
gateway network enforcement shape is implemented.
Selectable via `BOT_BOTTLE_BACKEND=macos-container`. This package owns the Apple
`container` CLI integration; launch remains gated until the gateway network
enforcement shape is implemented.
Thin by design: `MacosContainerBottleBackend` is re-exported lazily via
`__getattr__`, so importing a leaf module under this package doesn't drag the
backend (and the framework it pulls) into memory.
"""
from .backend import MacosContainerBottleBackend
from __future__ import annotations
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .backend import MacosContainerBottleBackend
def __getattr__(name: str) -> Any:
if name == "MacosContainerBottleBackend":
from .backend import MacosContainerBottleBackend
globals()[name] = MacosContainerBottleBackend
return MacosContainerBottleBackend
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
__all__ = ["MacosContainerBottleBackend"]
+11 -6
View File
@@ -2,7 +2,8 @@
from __future__ import annotations
from contextlib import contextmanager
import io
from contextlib import contextmanager, redirect_stderr
from pathlib import Path
from typing import Generator, Sequence
@@ -10,7 +11,7 @@ from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervise import SupervisePlan
from ...supervisor.plan import SupervisePlan
from ...manifest import Manifest
from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
from . import cleanup as _cleanup
@@ -31,6 +32,7 @@ class MacosContainerBottleBackend(
`--backend=macos-container`."""
name = "macos-container"
supports_nested_containers = True
@classmethod
def is_available(cls) -> bool:
@@ -42,8 +44,11 @@ class MacosContainerBottleBackend(
return _setup.setup()
@classmethod
def status(cls) -> int:
def status(cls, *, quiet: bool = False) -> int:
from . import setup as _setup
if quiet:
with redirect_stderr(io.StringIO()):
return _setup.status()
return _setup.status()
@classmethod
@@ -96,12 +101,12 @@ class MacosContainerBottleBackend(
yield bottle
def ensure_orchestrator(self) -> str:
"""Bring up the per-host infra container (control plane + gateway) and
"""Bring up the per-host pair (orchestrator + gateway containers) and
return its control-plane URL the on-demand entry point operator tools
(`supervise`) call when no control plane is running yet. Mirrors
firecracker's infra-VM bring-up."""
firecracker's infra bring-up."""
from .infra import MacosInfraService
return MacosInfraService().ensure_running().control_plane_url
return MacosInfraService().ensure_running()
def prepare_cleanup(self) -> MacosContainerBottleCleanupPlan:
return _cleanup.prepare_cleanup()
@@ -20,6 +20,12 @@ class MacosContainerBottlePlan(BottlePlan):
# bottle is registered. See launch.py's stamp for why it lives here and not
# only in the exec-time proxy env.
identity_token: str = ""
# Guest-local container engine (issue #392). Gates the derived image, the
# device-mode relaxation, and the resident podman service.
nested_containers: bool = False
# Generated before `container run` so it becomes part of the container's
# configured environment and can be read back after an infra restart.
env_var_secret: str = ""
@property
def container_name(self) -> str:
@@ -15,8 +15,10 @@ caller has to start the agent in between. `ensure_gateway` runs first because
the agent's proxy env needs the gateway's address at `container run` time; the
agent's *own* address (the attribution key) only exists afterwards.
The control plane and the gateway are one **infra container** here (see
`infra`), so `gateway_ip` and the control-plane host are the same address.
The control plane and the gateway are **separate containers** here (see
`infra`): the orchestrator on the host-only control network, the gateway on the
agent network `gateway_ip` is the gateway container's agent-network address,
distinct from the orchestrator's control-network host.
The consequence for the identity token: it is minted by registration, i.e.
*after* the agent container exists, so it cannot be baked into the run-time
@@ -38,11 +40,13 @@ from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...log import info
from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
from ..consolidated_util import provision_bottle, teardown_consolidated as _teardown_util
from ...orchestrator.reprovision import reprovision_bottles
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
from ..provision_bottle import deprovision_bottle, provision_bottle
from . import util as container_mod
from .enumerate import CONTAINER_NAME_PREFIX, EnumerationError, enumerate_active
from .gateway import GATEWAY_NETWORK
from .gateway_provision import AppleGatewayTransport
from .gateway_transport import MacosGatewayTransport
from .infra import MacosInfraService, OrchestratorStartError
@@ -52,9 +56,9 @@ class ConsolidatedLaunchError(RuntimeError):
@dataclass(frozen=True)
class GatewayEndpoint:
"""What the agent `container run` needs to reach the shared gateway (the
infra container). `gateway_ip` is that container's host-only address, the
same host the control-plane URL points at."""
"""What the agent `container run` needs to reach the shared gateway.
`gateway_ip` is the gateway container's agent-network address (the agent's
proxy target); `orchestrator_url` points at the separate control plane."""
orchestrator_url: str
gateway_ip: str # the gateway's address — the agent's proxy target
@@ -72,23 +76,47 @@ class LaunchContext:
gateway_ip: str
network: str
orchestrator_url: str
env_var_secret: str = "" # encryption key injected into the agent's env
def ensure_gateway(
*, service: MacosInfraService | None = None,
) -> GatewayEndpoint:
"""Ensure the per-host infra container (control plane + gateway) is up and
report how to reach it. Idempotent one singleton, so N bottle launches
share it. Call before starting the agent container: the agent's proxy env
needs `gateway_ip` at run time."""
"""Ensure the per-host pair (orchestrator + gateway containers) is up and
report how to reach the gateway. Idempotent one singleton pair, so N bottle
launches share it. Call before starting the agent container: the agent's
proxy env needs `gateway_ip` at run time."""
service = service or MacosInfraService()
infra = service.ensure_running()
return GatewayEndpoint(
orchestrator_url=infra.control_plane_url,
gateway_ip=infra.gateway_ip,
orchestrator_url = service.ensure_running()
endpoint = GatewayEndpoint(
orchestrator_url=orchestrator_url,
gateway_ip=service.gateway().address(),
gateway_ca_pem=service.ca_cert_pem(),
network=service.network,
)
_reprovision_running_bottles(endpoint)
return endpoint
def _reprovision_running_bottles(endpoint: GatewayEndpoint) -> None:
"""Recover keys from live Apple containers and restore gateway tokens."""
try:
secrets_by_ip: dict[str, str] = {}
for agent in enumerate_active():
name = f"{CONTAINER_NAME_PREFIX}{agent.slug}"
source_ip = container_mod.inspect_container_network_ip(name, endpoint.network)
if not source_ip:
continue
secret = container_mod.read_container_env(name, ENV_VAR_SECRET_NAME)
if secret:
secrets_by_ip[source_ip] = secret
count = reprovision_bottles(
OrchestratorClient(endpoint.orchestrator_url), secrets_by_ip,
)
if count:
info(f"reprovisioned egress tokens for {count} macOS bottle(s)")
except (OrchestratorClientError, EnumerationError, OSError) as exc:
info(f"egress token reprovision skipped: {exc}")
def live_source_ips(network: str) -> list[str]:
@@ -125,6 +153,7 @@ def register_agent(
endpoint: GatewayEndpoint,
image_ref: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str | None = None,
) -> LaunchContext:
"""Register the (already running) agent by its address and provision its
git-gate state into the gateway. `source_ip` must be read from the live
@@ -142,8 +171,9 @@ def register_agent(
except (OrchestratorClientError, EnumerationError) as e:
info(f"registry reconciliation skipped: {e}")
reg = provision_bottle(
client, source_ip, egress_plan, git_gate_plan, AppleGatewayTransport(),
client, source_ip, egress_plan, git_gate_plan, MacosGatewayTransport(),
image_ref=image_ref, tokens=tokens,
env_var_secret=env_var_secret,
)
return LaunchContext(
bottle_id=reg.bottle_id,
@@ -152,16 +182,17 @@ def register_agent(
gateway_ip=endpoint.gateway_ip,
network=endpoint.network,
orchestrator_url=endpoint.orchestrator_url,
env_var_secret=reg.env_var_secret,
)
def teardown_consolidated(
def deprovision_consolidated(
bottle_id: str, *, orchestrator_url: str, timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap. Does NOT
stop the gateway it's a persistent per-host singleton."""
_teardown_util(bottle_id, AppleGatewayTransport(),
deprovision_bottle(bottle_id, MacosGatewayTransport(),
orchestrator_url=orchestrator_url, timeout=timeout)
@@ -171,7 +202,7 @@ __all__ = [
"ensure_gateway",
"live_source_ips",
"register_agent",
"teardown_consolidated",
"deprovision_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
"GATEWAY_NETWORK",
@@ -6,17 +6,18 @@ import subprocess
from ...bottle_state import read_metadata
from .. import ActiveAgent
from .infra import INFRA_NAME
from .infra import INFRA_NAME, ORCHESTRATOR_NAME
# The name every agent container carries: `bot-bottle-<slug>`. Exported
# because callers that act on a running bottle (gateway-host rewrites,
# registry reconciliation) have to map an enumerated slug back to a
# container name.
CONTAINER_NAME_PREFIX = "bot-bottle-"
# The shared per-host infra container carries the same prefix as agent
# containers but is infrastructure, not a bottle — one control plane + gateway
# serves every agent, so listing it as an agent would invent one per host.
_INFRA_NAMES = frozenset({INFRA_NAME})
# The two shared per-host infra containers (orchestrator + gateway) carry the
# same `bot-bottle-` prefix as agent containers but are infrastructure, not
# bottles — one pair serves every agent, so enumerating either as an agent would
# invent a phantom bottle per host.
_INFRA_NAMES = frozenset({INFRA_NAME, ORCHESTRATOR_NAME})
class EnumerationError(RuntimeError):
+180 -14
View File
@@ -1,46 +1,212 @@
"""Shared network/image constants for the macOS consolidated infra container.
"""The macOS gateway data plane as an Apple container (PRD 0070).
The gateway data plane no longer runs as its own Apple container it shares a
single per-host **infra container** with the control plane (see `infra`),
because two Apple-Container guests writing one `bot-bottle.db` over virtiofs
would race incoherent `fcntl` locks. This module holds the pieces both the
infra service and the launch/provision glue need: the network names, the
gateway image, and the network-creation helper.
`MacosGateway` is the Apple-Container implementation of the backend-neutral
`Gateway` service. It runs the gateway daemons (egress / git-http / supervise)
in a single triple-homed container and never opens `bot-bottle.db` it reaches
the supervise queue over the control-plane RPC (#469), so it holds no signing
key, only the pre-minted `gateway` token the orchestrator hands it via
`connect_to_orchestrator`.
This module also holds the pieces the infra service + launch/provision glue
share: the network names, the gateway image, and the network-creation helper.
"""
from __future__ import annotations
import os
from pathlib import Path
from ...orchestrator.gateway import GatewayError
from ...gateway import (
DEFAULT_CA_TIMEOUT_SECONDS,
GATEWAY_CA_CERT,
MITMPROXY_HOME,
Gateway,
GatewayError,
GatewayTransport,
)
from ...paths import (
ORCHESTRATOR_AUTH_JWT_ENV,
host_gateway_ca_dir,
)
from .. import util as backend_util
from . import util as container_mod
# The shared host-only network the infra container and every agent bottle sit
# The shared host-only network the gateway container and every agent bottle sit
# on. The agent's address here is the attribution key. Distinct from the docker
# names so both backends can coexist on one host.
GATEWAY_NETWORK = "bot-bottle-mac-gateway"
# The NAT network that gives the infra container (and only it) a route out.
# The NAT network that gives the gateway (and only it) a route out.
GATEWAY_EGRESS_NETWORK = "bot-bottle-mac-egress"
# The control network the gateway reaches the orchestrator over (host-only).
# Only the orchestrator + gateway join it; agents never do, so agents have no
# route to the control plane (PRD 0070 "Separating the planes").
CONTROL_NETWORK = "bot-bottle-mac-control"
# The gateway (data plane) container. The name predates the split — it was the
# combined "infra" container — and is kept so callers importing it still resolve
# the gateway (probe / reprovision attribute against it).
GATEWAY_NAME = "bot-bottle-mac-infra"
GATEWAY_LABEL = "bot-bottle-mac-infra=1"
# The gateway subset the consolidated model runs (no per-bottle git:// daemon).
GATEWAY_DAEMONS = "egress,git-http,supervise"
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
DEFAULT_CA_TIMEOUT_SECONDS = 30.0
_REPO_ROOT = Path(__file__).resolve().parents[3]
def ensure_networks(
network: str = GATEWAY_NETWORK, egress_network: str = GATEWAY_EGRESS_NETWORK,
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
control_network: str = CONTROL_NETWORK,
) -> None:
"""Create the shared host-only network + the NAT egress network. Idempotent
`create_network` tolerates 'already exists'."""
"""Create the shared host-only agent network, the NAT egress network, and
the host-only control network. Idempotent `create_network` tolerates
'already exists'."""
container_mod.create_network(egress_network)
container_mod.create_network(network, internal=True)
container_mod.create_network(control_network, internal=True)
class MacosGateway(Gateway):
"""The consolidated gateway as a single Apple container, triple-homed on the
NAT egress, host-only agent, and control networks.
`ensure_built` builds `Dockerfile.gateway`; the networks are ensured by the
composer. `connect_to_orchestrator` runs the container carrying the mitmproxy
CA + the pre-minted `gateway` token."""
def __init__(
self,
image_ref: str = GATEWAY_IMAGE,
*,
name: str = GATEWAY_NAME,
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
control_network: str = CONTROL_NETWORK,
repo_root: Path = _REPO_ROOT,
) -> None:
self.image_ref = image_ref
self.name = name
self.network = network
self.egress_network = egress_network
self.control_network = control_network
self._repo_root = repo_root
# Set by `connect_to_orchestrator`: the URL the daemons resolve policy
# against + the pre-minted `gateway` token they present. The gateway
# never mints, so it never holds the signing key (#469).
self._orchestrator_url = ""
self._gateway_token = ""
def ensure_built(self) -> None:
"""Build the data-plane image from `Dockerfile.gateway`."""
container_mod.build_image(
self.image_ref, str(self._repo_root), dockerfile="Dockerfile.gateway")
def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
"""Bind the gateway to this orchestrator and (re)start it, dual-homed on
the agent + control networks, resolving policy against `orchestrator_url`
(the orchestrator's control-network address — Apple has no container
DNS) and presenting `gateway_token`."""
self._orchestrator_url = orchestrator_url
self._gateway_token = gateway_token
# Fail closed on a missing policy source or token: the resolver-only data
# plane (PRD 0070) would only crash-loop its daemons without an
# orchestrator URL, and it cannot mint the token it presents (#469).
if not self._orchestrator_url:
raise GatewayError(
"gateway requires an orchestrator URL to run "
"(resolver-only data plane; no single-tenant fallback)"
)
if not self._gateway_token:
raise GatewayError(
"gateway requires a pre-minted `gateway` token to run "
"(the orchestrator mints it; the gateway never holds the key)"
)
container_mod.force_remove_container(self.name)
argv = [
"container", "run", "--detach",
"--name", self.name,
"--label", "bot-bottle.backend=macos-container",
"--label", GATEWAY_LABEL,
# NAT egress FIRST (default route out); the host-only agent network
# is where agents reach the gateway; the control network reaches the
# orchestrator.
"--network", self.egress_network,
"--network", self.network,
"--network", self.control_network,
"--dns", container_mod.dns_server(),
# The mitmproxy CA on a host bind-mount (survives recreation +
# volume pruning — issue #450). No DB mount: the data plane never
# opens bot-bottle.db (#469).
"--mount",
container_mod.bind_mount_spec(str(host_gateway_ca_dir()), MITMPROXY_HOME),
"--env", f"BOT_BOTTLE_GATEWAY_DAEMONS={GATEWAY_DAEMONS}",
"--env", f"BOT_BOTTLE_ORCHESTRATOR_URL={self._orchestrator_url}",
# The pre-minted `gateway` JWT (never the signing key). Bare
# `--env NAME` inherits the value from run_env below.
"--env", ORCHESTRATOR_AUTH_JWT_ENV,
self.image_ref,
]
run_env = {**os.environ, ORCHESTRATOR_AUTH_JWT_ENV: self._gateway_token}
result = container_mod.run_container_argv(argv, env=run_env)
if result.returncode != 0:
raise GatewayError(
f"gateway container failed to start: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def is_running(self) -> bool:
return container_mod.container_is_running(self.name)
def stop(self) -> None:
"""Remove the gateway container (idempotent)."""
container_mod.force_remove_container(self.name)
def address(self) -> str:
"""The gateway's agent-network address — the proxy / git-http / supervise
target agents dial (also the source IP the gateway attributes by)."""
ip = container_mod.try_container_ipv4_on_network(self.name, self.network)
if not ip:
raise GatewayError(
f"gateway {self.name} has no address on {self.network}"
)
return ip
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception. Read from the gateway container; polls because mitmproxy
writes it a beat after start."""
def _fetch() -> str | None:
result = container_mod.run_container_argv(
["container", "exec", self.name, "cat", GATEWAY_CA_CERT])
return result.stdout if result.returncode == 0 and result.stdout.strip() else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
raise GatewayError(
f"gateway CA not available in {self.name} after {timeout:g}s"
) from exc
def provisioning_transport(self) -> GatewayTransport:
"""The exec/cp transport git-gate provisioning stages per-bottle repos +
deploy keys through (over the `container` CLI)."""
# Local import: gateway_transport imports GATEWAY_NAME from this module,
# so importing MacosGatewayTransport at module scope would cycle.
from .gateway_transport import MacosGatewayTransport
return MacosGatewayTransport(self.name)
__all__ = [
"GATEWAY_NETWORK",
"GATEWAY_EGRESS_NETWORK",
"CONTROL_NETWORK",
"GATEWAY_NAME",
"GATEWAY_LABEL",
"GATEWAY_DAEMONS",
"GATEWAY_IMAGE",
"GatewayError",
"DEFAULT_CA_TIMEOUT_SECONDS",
"ensure_networks",
"MacosGateway",
]
@@ -1,7 +1,7 @@
"""Stable gateway name for macOS agents, via each bottle's `/etc/hosts`.
The shared gateway's address is assigned by vmnet's DHCP and changes whenever
the infra container is recreated a source-hash bump, an image upgrade, a
the gateway container is recreated a source-hash bump, an image upgrade, a
crash. Every agent-facing URL (egress proxy, git-http, supervise) embeds that
address, and the proxy URL reaches the agent as **process environment** at
`container exec` time. A running process's `environ` cannot be rewritten from
@@ -1,23 +1,22 @@
"""`GatewayTransport` for the Apple infra container (PRD 0070).
"""The `GatewayTransport` for the Apple gateway container (PRD 0070).
The provisioning *logic* (per-bottle creds dirs, namespaced repo init) is
backend-neutral and lives in `backend.docker.gateway_provision`; this is only
the transport how files and commands reach the running gateway. Docker uses
`docker exec`/`docker cp` and Firecracker uses SSH; Apple uses the `container`
CLI's equivalents against the infra container that hosts the gateway daemons.
How the launcher stages files + runs commands in the running gateway container:
the `container` CLI's exec/cp equivalents. The backend-neutral provisioning
logic that drives it lives in `backend.provision_gateway`; Docker uses
`docker exec`/`docker cp` and Firecracker uses SSH.
"""
from __future__ import annotations
from ..docker.gateway_provision import GatewayProvisionError
from ...gateway import GatewayProvisionError
from . import util as container_mod
from .infra import INFRA_NAME
from .gateway import GATEWAY_NAME
class AppleGatewayTransport:
"""`GatewayTransport` for the gateway daemons in the Apple infra container."""
class MacosGatewayTransport:
"""`GatewayTransport` for the gateway daemons in the Apple gateway container."""
def __init__(self, gateway: str = INFRA_NAME) -> None:
def __init__(self, gateway: str = GATEWAY_NAME) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
@@ -41,4 +40,4 @@ class AppleGatewayTransport:
)
__all__ = ["AppleGatewayTransport", "GatewayProvisionError"]
__all__ = ["MacosGatewayTransport"]
+100 -252
View File
@@ -1,125 +1,65 @@
"""The per-host infra container for the macOS backend (PRD 0070).
"""The per-host control plane + gateway for the macOS backend (PRD 0070).
A single persistent Apple container that runs BOTH the orchestrator control
plane and the gateway data plane the macOS analogue of the Firecracker infra
VM (`backend/firecracker/infra_vm.py`), not the docker backend's two separate
containers.
Two Apple containers the orchestrator (control plane) and the gateway (data
plane) split now that #469 got the DB off the data plane. The single-container
model existed only because two Apple-Container guests writing one `bot-bottle.db`
over virtiofs would race incoherent `fcntl` locks; with the data plane no longer
opening the DB at all, only the orchestrator does, so the split is safe.
Why one container, not two: Apple Containers are lightweight VMs, each with its
own kernel. The docker backend runs the orchestrator and gateway as two
containers safely because they share the host kernel, so their concurrent
writes to the one `bot-bottle.db` (the orchestrator's registry + the gateway
supervise daemon's queue) are serialized by coherent `fcntl` locks. Across two
*guest* kernels sharing a virtiofs-mounted DB those locks are not coherent, and
concurrent writers can corrupt the file. Firecracker solved this by putting
both services in one guest with the DB on a device only that guest mounts; this
does the same with Apple primitives.
* `bot-bottle-mac-orchestrator` the lean control plane (`MacosOrchestrator`).
Joins the host-only **control network** (`bot-bottle-mac-control`) only. Sole
opener of the container-only DB volume; holds the signing key. The host CLI
reaches it at its control-network address; the gateway reaches it there too.
* `bot-bottle-mac-infra` the gateway data plane (`MacosGateway`). Triple-homed:
the NAT egress network (route out), the host-only agent network (agents + CLI
reach the gateway), and the control network (reach the orchestrator by IP
Apple has no container DNS). Holds the mitmproxy CA + the `gateway` JWT.
Two consequences fall out of the single container, both simplifications:
- **No DNS dance.** The control plane and the gateway daemons reach each other
over `127.0.0.1`, so nothing depends on Apple's (absent) container DNS and
there is no orchestrator-before-gateway ordering to get right.
- **The DB is never host-shared.** It lives on a container-only volume, so no
host process opens the live file. The host CLI reaches registry + supervise
state through the control-plane HTTP surface (`cli/supervise.py` already uses
`OrchestratorClient`), exactly as it does for firecracker.
The control-plane source is bind-mounted (like the docker orchestrator), so a
code change takes effect on the next launch without an image rebuild; the
gateway daemons are baked in the gateway image and rebuild through its own
digest check.
`MacosInfraService` composes the two services and brings them up as an idempotent
per-host pair. Agents sit on the agent network only, never the control network,
so they have no route to the control plane (the L3 block, not just the JWT).
"""
from __future__ import annotations
import os
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from ... import log
from ...orchestrator.gateway import GATEWAY_CA_CERT, MITMPROXY_HOME
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
OrchestratorStartError,
source_hash,
)
from ...paths import (
CONTROL_PLANE_TOKEN_ENV,
HOST_DB_FILENAME,
host_control_plane_token,
host_gateway_ca_dir,
)
from .. import util as backend_util
from ..infra_service import InfraService
from . import util as container_mod
from .gateway import (
CONTROL_NETWORK,
DEFAULT_CA_TIMEOUT_SECONDS,
GATEWAY_EGRESS_NETWORK,
GATEWAY_IMAGE,
GATEWAY_NAME,
GATEWAY_NETWORK,
GatewayError,
MacosGateway,
ensure_networks,
)
from .orchestrator import (
ORCHESTRATOR_DB_VOLUME,
ORCHESTRATOR_IMAGE,
ORCHESTRATOR_NAME,
MacosOrchestrator,
probe_orchestrator_url,
)
# The one per-host infra container: control plane + gateway data plane.
INFRA_NAME = "bot-bottle-mac-infra"
INFRA_LABEL = "bot-bottle-mac-infra=1"
# Container-only volume holding bot-bottle.db. No host bind-mount, so the DB is
# written by exactly one kernel (this container's). Survives recreation.
INFRA_DB_VOLUME = "bot-bottle-mac-db"
# BOT_BOTTLE_ROOT inside the container; host_db_path() resolves the DB to
# <root>/db/<filename> and the supervise daemon writes the same file.
_DB_ROOT_IN_CONTAINER = "/var/lib/bot-bottle"
_DB_PATH_IN_CONTAINER = f"{_DB_ROOT_IN_CONTAINER}/db/{HOST_DB_FILENAME}"
_SRC_IN_CONTAINER = "/bot-bottle-src"
# `INFRA_NAME` is kept — now aliasing the gateway container — for callers that
# still import it (probe / reprovision attribute against the gateway).
INFRA_NAME = GATEWAY_NAME
_REPO_ROOT = Path(__file__).resolve().parents[3]
_HEALTH_POLL_SECONDS = 0.25
_HEALTH_REQUEST_TIMEOUT_SECONDS = 1.0
_CA_POLL_SECONDS = 0.5
# The gateway subset the consolidated model runs (no per-bottle git:// daemon).
_GATEWAY_DAEMONS = "egress,git-http,supervise"
def _init_script(port: int) -> str:
"""PID-1 init: start the control plane and the gateway daemons, both in
this container, reaching each other over loopback. Backgrounded so `wait`
reaps as PID 1. No `set -e` a transient daemon failure must not kill the
whole container (gateway_init applies the same 'stay up' policy)."""
return (
"export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n"
f"mkdir -p $(dirname {_DB_PATH_IN_CONTAINER})\n"
# Control plane, from the bind-mounted source (stdlib-only package).
f"( cd {_SRC_IN_CONTAINER} && BOT_BOTTLE_ROOT={_DB_ROOT_IN_CONTAINER} "
f"python3 -m bot_bottle.orchestrator --host 0.0.0.0 --port {port} "
"--broker stub ) &\n"
# Gateway data plane, multi-tenant against the local control plane.
f"( cd /app && BOT_BOTTLE_GATEWAY_DAEMONS={_GATEWAY_DAEMONS} "
f"BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{port} "
f"SUPERVISE_DB_PATH={_DB_PATH_IN_CONTAINER} python3 -m bot_bottle.gateway_init ) &\n"
"while : ; do wait ; done\n"
)
@dataclass(frozen=True)
class InfraEndpoint:
"""How to reach the running infra container. The control plane and the
gateway are the same container, so one address serves both."""
control_plane_url: str # http://<infra ip>:8099 — host CLI + registration
gateway_ip: str # same container; agents' proxy / git-http / MCP target
class MacosInfraService:
"""Manages the single per-host infra container. Callers use
`ensure_running()` (returns the endpoint) and `ca_cert_pem()`."""
class MacosInfraService(InfraService):
"""Composes the per-host orchestrator + gateway containers. Callers use
`ensure_running()` (returns the control-plane URL), the `orchestrator()` /
`gateway()` accessors, and `ca_cert_pem()`."""
def __init__(
self,
@@ -127,187 +67,95 @@ class MacosInfraService:
port: int = DEFAULT_PORT,
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
image: str = GATEWAY_IMAGE,
control_network: str = CONTROL_NETWORK,
gateway_image: str = GATEWAY_IMAGE,
orchestrator_image: str = ORCHESTRATOR_IMAGE,
repo_root: Path = _REPO_ROOT,
name: str = INFRA_NAME,
db_volume: str = INFRA_DB_VOLUME,
orchestrator_name: str = ORCHESTRATOR_NAME,
gateway_name: str = INFRA_NAME,
db_volume: str = ORCHESTRATOR_DB_VOLUME,
) -> None:
self.port = port
self.network = network
self.egress_network = egress_network
self.image = image
self.control_network = control_network
self.gateway_image = gateway_image
self.orchestrator_image = orchestrator_image
self._repo_root = repo_root
self._name = name
self._orchestrator_name = orchestrator_name
self._gateway_name = gateway_name
self._db_volume = db_volume
def _resolve_url(self) -> str:
"""The control-plane URL, or "" while the container has no address."""
ip = container_mod.try_container_ipv4_on_network(self._name, self.network)
return f"http://{ip}:{self.port}" if ip else ""
def orchestrator(self) -> MacosOrchestrator:
"""The control-plane service on the host-only control network. Cheap to
reconstruct the launch flow reads its `url()` / `gateway_url()` /
`mint_gateway_token()` off it."""
return MacosOrchestrator(
self.orchestrator_image,
name=self._orchestrator_name,
port=self.port,
control_network=self.control_network,
repo_root=self._repo_root,
db_volume=self._db_volume,
)
def is_healthy(
self, url: str, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS,
) -> bool:
if not url:
return False
try:
with urllib.request.urlopen(f"{url}/health", timeout=timeout) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
def _source_current(self, current_hash: str) -> bool:
"""True iff the running infra container was created from the current
bind-mounted control-plane source. The control-plane process loads that
code at startup and won't reload it, so a stale container keeps serving
OLD code."""
if not container_mod.container_is_running(self._name):
return False
env = container_mod.container_env(self._name)
if not env:
return True # can't compare → don't churn a working container
return env.get("BOT_BOTTLE_SOURCE_HASH") == current_hash
def _running_healthy_endpoint(self, current_hash: str) -> InfraEndpoint | None:
"""The endpoint if the running container is BOTH source-current and
answering /health, else None ( recreate). Health, not just the source
label, is what lets a wedged-but-current container self-heal instead of
being polled to death forever."""
if not self._source_current(current_hash):
return None
url = self._resolve_url()
if url and self.is_healthy(url):
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
return None
def ensure_built(self) -> None:
"""Ensure the gateway data-plane image exists. The control-plane source
is bind-mounted, not baked, so only the gateway image needs building."""
container_mod.build_image(
self.image, str(self._repo_root), dockerfile="Dockerfile.gateway",
def gateway(self) -> MacosGateway:
"""The data-plane gateway service, triple-homed on the egress + agent +
control networks. Cheap to reconstruct the launch flow reads its
`address()` / CA / provisioning transport off it, and `ensure_running`
connects it to the control plane."""
return MacosGateway(
self.gateway_image,
name=self._gateway_name,
network=self.network,
egress_network=self.egress_network,
control_network=self.control_network,
repo_root=self._repo_root,
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> InfraEndpoint:
"""Ensure the single infra container is up; return how to reach it.
Idempotent per-host singleton a healthy container on current source
is left untouched, so N launches share the one control plane + gateway.
Raises `OrchestratorStartError` on startup timeout."""
current_hash = source_hash(self._repo_root)
endpoint = self._running_healthy_endpoint(current_hash)
if endpoint is not None:
return endpoint
self.ensure_built()
log.info("starting infra container", context={"name": self._name})
self._run_container(current_hash)
return self._wait_healthy(startup_timeout)
) -> str:
"""Ensure the orchestrator + gateway containers are up; return the host
control-plane URL. Idempotent per-host singleton a healthy orchestrator
on current source is left untouched. Raises `OrchestratorStartError` on
control-plane startup timeout."""
# The networks (host-only agent + NAT egress + host-only control) are a
# shared concern — create them before either plane comes up.
ensure_networks(self.network, self.egress_network, self.control_network)
def _run_container(self, current_hash: str) -> None:
ensure_networks(self.network, self.egress_network)
container_mod.force_remove_container(self._name)
argv = [
"container", "run", "--detach",
"--name", self._name,
"--label", "bot-bottle.backend=macos-container",
"--label", INFRA_LABEL,
# NAT network FIRST so the gateway's egress has a default route;
# the host-only network is where agents (and the host CLI) reach it.
"--network", self.egress_network,
"--network", self.network,
"--dns", container_mod.dns_server(),
# Container-only DB volume: one kernel writes bot-bottle.db, never
# shared with the host or another guest.
"--volume", f"{self._db_volume}:{_DB_ROOT_IN_CONTAINER}",
# The DB needs a container-only ext4 volume for coherent SQLite
# locking, but the CA has no such constraint. Keep it in the host
# app-data root so infra-container recreation and Apple Container
# volume pruning cannot silently rotate every bottle's trust
# anchor (issue #450).
"--mount",
container_mod.bind_mount_spec(
str(host_gateway_ca_dir()), MITMPROXY_HOME),
# Bind-mount the control-plane source (read-only); a code change
# takes effect on relaunch with no image rebuild.
"--mount",
container_mod.bind_mount_spec(
str(self._repo_root), _SRC_IN_CONTAINER, readonly=True),
# Baked onto the container so `_source_current` can detect a real
# control-plane code change and recreate.
"--env", f"BOT_BOTTLE_SOURCE_HASH={current_hash}",
# The control-plane secret, for BOTH the control plane (to require
# it) and the gateway's PolicyResolver (to present it) — they share
# this one container. Bare `--env NAME` inherits the value from the
# run process below, so the secret never lands on argv or in
# `container inspect`'s command line. The agent runs in a SEPARATE
# container that is never given this var, which is the whole point.
"--env", CONTROL_PLANE_TOKEN_ENV,
"--entrypoint", "sh",
self.image,
"-c", _init_script(self.port),
]
run_env = {**os.environ, CONTROL_PLANE_TOKEN_ENV: host_control_plane_token()}
result = container_mod.run_container_argv(argv, env=run_env)
if result.returncode != 0:
raise OrchestratorStartError(
f"infra container failed to start: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
orchestrator = self.orchestrator()
gateway = self.gateway()
orchestrator.ensure_built()
gateway.ensure_built()
def _wait_healthy(self, startup_timeout: float) -> InfraEndpoint:
deadline = time.monotonic() + startup_timeout
while True:
url = self._resolve_url()
if url and self.is_healthy(url):
log.info("infra container healthy", context={"url": url})
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
if time.monotonic() >= deadline:
raise OrchestratorStartError(
f"infra container did not become healthy within "
f"{startup_timeout:g}s"
)
time.sleep(_HEALTH_POLL_SECONDS)
orchestrator.ensure_running(startup_timeout=startup_timeout)
# (Re)ensure the gateway once the control plane it resolves against is
# healthy — it needs the orchestrator's control-network address. The
# orchestrator (which holds the signing key) mints the role-scoped
# `gateway` JWT and hands it to the gateway, which never sees the key
# (#469).
url = orchestrator.url()
gateway.connect_to_orchestrator(url, orchestrator.mint_gateway_token())
return url
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception. Read through the container path backed by the persistent
host CA directory; polls because mitmproxy writes it a beat after
start."""
def _fetch() -> str | None:
result = container_mod.run_container_argv(
["container", "exec", self._name, "cat", GATEWAY_CA_CERT])
return result.stdout if result.returncode == 0 and result.stdout.strip() else None
try:
return backend_util.poll_ca_cert(_fetch, timeout=timeout)
except TimeoutError as exc:
raise GatewayError(
f"gateway CA not available in {self._name} after {timeout:g}s"
) from exc
interception delegated to the gateway service (reads it out of the
gateway container, polling until mitmproxy writes it)."""
return self.gateway().ca_cert_pem(timeout=timeout)
def stop(self) -> None:
"""Remove the infra container (idempotent). The DB volume persists."""
container_mod.force_remove_container(self._name)
def _ip_of(url: str) -> str:
"""The host from an http://host:port URL."""
return url.split("://", 1)[-1].rsplit(":", 1)[0]
def probe_control_plane_url(port: int = DEFAULT_PORT) -> str:
"""The running infra container's control-plane URL, or "" if it isn't up.
Used by host-side control-plane discovery (`discover_orchestrator_url`);
safe to call on any host returns "" when the container or the `container`
CLI isn't present."""
ip = container_mod.try_container_ipv4_on_network(INFRA_NAME, GATEWAY_NETWORK)
return f"http://{ip}:{port}" if ip else ""
"""Remove both containers (idempotent). The DB volume persists."""
container_mod.force_remove_container(self._gateway_name)
container_mod.force_remove_container(self._orchestrator_name)
__all__ = [
"MacosInfraService",
"InfraEndpoint",
"OrchestratorStartError",
"GatewayError",
"ORCHESTRATOR_NAME",
"INFRA_NAME",
"INFRA_DB_VOLUME",
"probe_orchestrator_url",
]
+57 -21
View File
@@ -44,19 +44,13 @@ from ...bottle_state import (
git_gate_state_dir,
read_committed_image,
)
from ...egress import (
egress_agent_env_entries,
egress_resolve_token_values,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...git_http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
from ...egress import Egress
from ...git_gate import GitGate
from ...gateway.git_gate.http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
from ...image_cache import check_stale
from ...log import die, info, warn
from .. import BottleImages
from ...supervise import SUPERVISE_PORT
from ...supervisor.types import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import util as container_mod
@@ -66,13 +60,15 @@ from .gateway_hosts import (
refresh_gateway_host,
set_gateway_host,
)
from . import nested_containers as nested_containers_mod
from .bottle_plan import MacosContainerBottlePlan
from ...orchestrator.config_store import resolve_teardown_timeout
from ...orchestrator.store.config_store import resolve_teardown_timeout
from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME, new_env_var_secret
from .consolidated_launch import (
GatewayEndpoint,
ensure_gateway,
register_agent,
teardown_consolidated,
deprovision_consolidated,
)
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
@@ -82,10 +78,14 @@ _AGENT_SLEEP_SECONDS = "2147483647"
def build_or_load_images(plan: MacosContainerBottlePlan) -> BottleImages:
"""Resolve the agent image ref for this plan. The gateway's own image is
built by `ensure_gateway` it belongs to the shared singleton."""
return BottleImages(agent=_layer_nested_containers(plan, _agent_image(plan)))
def _agent_image(plan: MacosContainerBottlePlan) -> str:
committed = read_committed_image(plan.slug)
if committed and container_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return BottleImages(agent=committed)
return committed
if plan.spec.image_policy == "cached":
if not container_mod.image_exists(plan.image):
die(
@@ -93,9 +93,31 @@ def build_or_load_images(plan: MacosContainerBottlePlan) -> BottleImages:
"run without --cached-images to build it"
)
info(f"using cached agent image {plan.image!r}")
return BottleImages(agent=plan.image)
return plan.image
container_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
return BottleImages(agent=plan.image)
return plan.image
def _layer_nested_containers(
plan: MacosContainerBottlePlan, agent_image: str,
) -> str:
"""Add the guest-local container tooling on top of the agent image.
A separate derived tag, not the provider Dockerfile, so bottles that never
ask for nested containers carry none of its weight.
"""
if not plan.nested_containers:
return agent_image
derived = f"{agent_image}{nested_containers_mod.IMAGE_SUFFIX}"
if plan.spec.image_policy == "cached":
if not container_mod.image_exists(derived):
die(
f"cached nested-container image {derived!r} not found; "
"run without --cached-images to build it"
)
info(f"using cached nested-container image {derived!r}")
return derived
return nested_containers_mod.build_image(agent_image, container_mod.build_image)
@contextmanager
@@ -123,7 +145,7 @@ def launch(
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"macos-container teardown failed: {exc!r}")
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
GitGate().revoke_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
@@ -142,6 +164,7 @@ def launch(
plan = _provision_git_gate_keys(plan)
plan = _install_gateway_ca(plan, endpoint)
plan = _stamp_agent_urls(plan, endpoint)
plan = dataclasses.replace(plan, env_var_secret=new_env_var_secret())
# Step 3: run the agent. It has no identity token yet — registration
# needs the address this run assigns.
@@ -165,7 +188,7 @@ def launch(
f"{endpoint.network}"
)
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
token_values = Egress().resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
teardown_timeout = resolve_teardown_timeout()
@@ -176,9 +199,10 @@ def launch(
endpoint=endpoint,
image_ref=plan.image,
tokens=token_values,
env_var_secret=plan.env_var_secret,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
deprovision_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
timeout=teardown_timeout,
)
@@ -196,6 +220,10 @@ def launch(
# token above, so — unlike the run-time env — the plan CAN carry it.
plan = dataclasses.replace(plan, identity_token=ctx.identity_token)
exec_env = {
**_identity_proxy_env(endpoint, ctx.identity_token),
**nested_containers_mod.guest_env(plan.nested_containers),
}
bottle = MacosContainerBottle(
plan.container_name,
teardown,
@@ -209,10 +237,16 @@ def launch(
),
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
exec_env=_identity_proxy_env(endpoint, ctx.identity_token),
exec_env=exec_env,
)
bottle.prompt_path = provision(plan, bottle)
if plan.nested_containers:
nested_containers_mod.prepare_guest_devices(
plan.container_name, container_mod.exec_container_as_root,
)
nested_containers_mod.start(bottle)
yield bottle
finally:
teardown()
@@ -242,7 +276,7 @@ def _provision_git_gate_keys(
) -> MacosContainerBottlePlan:
if not plan.git_gate_plan.upstreams:
return plan
git_gate_plan = provision_git_gate_dynamic_keys(
git_gate_plan = GitGate().provision_dynamic_keys(
plan.manifest.bottle,
plan.git_gate_plan,
git_gate_state_dir(plan.slug),
@@ -406,13 +440,15 @@ def _agent_env_entries(
env.append(f"GIT_GATE_URL={plan.agent_git_gate_url}")
if plan.agent_supervise_url:
env.append(f"MCP_SUPERVISE_URL={plan.agent_supervise_url}")
if getattr(plan, "env_var_secret", ""):
env.append(f"{ENV_VAR_SECRET_NAME}={plan.env_var_secret}")
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars: bare name → inherits from the `container run` process env
# so the secret value never lands on argv.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
env.extend(Egress().agent_env_entries(plan.egress_plan))
return tuple(env)
@@ -0,0 +1,210 @@
#!/bin/sh
set -eu
uid="$(id -u)"
if [ "$uid" -eq 0 ]; then
echo "refusing to run the guest container engine as root" >&2
exit 1
fi
# Every piece of podman 5's networking stack is checked here, because each
# one fails at a different and misleading layer if it is absent: no pasta and
# nothing starts at all; no nft and netavark cannot build the bridge every
# compose file expects; no aardvark-dns and DNS inside nested containers fails
# while everything else looks healthy.
for command in podman docker fuse-overlayfs pasta nft slirp4netns; do
command -v "$command" >/dev/null 2>&1 || {
echo "missing nested-container prerequisite: $command" >&2
exit 1
}
done
# The inverse of the rootless-Docker check, and the whole point of the podman
# variant: a subordinate range would push podman onto newuidmap, which cannot
# write a multi-range uid_map without CAP_SYS_ADMIN in this guest. An empty
# range keeps it on the single-UID self-mapping an unprivileged process may
# write itself.
if grep -q "^$(id -un):" /etc/subuid 2>/dev/null; then
echo "unexpected subordinate UID range for $(id -un): podman would" >&2
echo "require CAP_SYS_ADMIN via newuidmap in this guest" >&2
exit 1
fi
for device in /dev/fuse /dev/net/tun; do
[ -r "$device" ] && [ -w "$device" ] || {
echo "device $device is not readable/writable by $(id -un)" >&2
exit 1
}
done
# Short by necessity, not by accident: conmon's attach socket lives under
# this directory and must fit in a 108-byte sun_path. See nested_containers.py.
# Must stay in step with AGENT_CA_BUNDLE in bot_bottle/backend/util.py; a unit
# test pins the two together.
CA_BUNDLE="/etc/ssl/certs/ca-certificates.crt"
[ -r "$CA_BUNDLE" ] || {
echo "gateway CA bundle $CA_BUNDLE is missing or unreadable" >&2
exit 1
}
# The proxy URL the agent inherits names `bot-bottle-gateway`, which resolves
# only through this bottle's /etc/hosts. A nested container gets its own hosts
# file, so it cannot resolve the name and dies at "Could not resolve proxy".
#
# podman's containers.conf `hosts_file` would fix that, except the
# Docker-compatible API ignores it — it only takes effect for native
# `podman run`, and the agent types `docker`. So the name is resolved *here*
# and the address, not the name, goes into the proxy URL the nested container
# receives. Verified on macOS 26 / podman 5.4.2: with the address in place,
# https://quay.io returns 200 and a non-allowlisted host still gets 403, so
# the egress boundary applies inside nested containers too.
GATEWAY_NAME="bot-bottle-gateway"
gateway_ip="$(
awk -v name="$GATEWAY_NAME" '$2 == name { print $1; exit }' /etc/hosts
)"
[ -n "$gateway_ip" ] || {
echo "no /etc/hosts entry for $GATEWAY_NAME; the gateway address is" >&2
echo "needed so nested containers can reach the egress proxy" >&2
exit 1
}
export XDG_RUNTIME_DIR="${XDG_RUNTIME_DIR:-/tmp/bbp}"
config="$HOME/.config/containers"
mkdir -p "$XDG_RUNTIME_DIR" "$config"
chmod 700 "$XDG_RUNTIME_DIR"
# ignore_chown_errors is required, not incidental: with a single-UID mapping
# there is no second UID for image layers to be chowned to, so layers that
# record other owners would otherwise fail to extract.
cat > "$config/storage.conf" <<'CONF'
[storage]
driver="overlay"
[storage.options.overlay]
mount_program="/usr/bin/fuse-overlayfs"
ignore_chown_errors="true"
CONF
# No cgroup delegation reaches this guest, so asking podman to manage cgroups
# fails; events_logger=file avoids the journald socket that is equally absent.
#
# The rest of this config is what lets a nested container reach the network:
#
# hosts_file only takes effect for native `podman run` — the
# Docker-compatible API ignores it, and the agent types
# `docker`. Kept anyway because it costs nothing and makes
# podman-native use behave; the compat path is covered by the
# address-bearing proxy URL below.
# volumes/env the gateway TLS-intercepts, so a container that does not
# trust the bottle's CA bundle gets "unable to get local issuer
# certificate". Mounting the bundle read-only and pointing the
# usual env vars at it covers curl, wget, python, and node
# without distro-specific trust commands.
#
# The proxy URL carries the bottle's identity token. podman already forwards
# that same URL into every nested container from the agent's own environment,
# so writing it to a 0600 file inside this disposable VM hands it to nobody
# new. It is never echoed.
CA_BUNDLE="$CA_BUNDLE" GATEWAY_NAME="$GATEWAY_NAME" GATEWAY_IP="$gateway_ip" \
CONTAINERS_CONF="$config/containers.conf" python3 - <<'PY'
import os
from pathlib import Path
ca = os.environ["CA_BUNDLE"]
name = os.environ["GATEWAY_NAME"]
ip = os.environ["GATEWAY_IP"]
entries = [
f"SSL_CERT_FILE={ca}",
f"CURL_CA_BUNDLE={ca}",
f"REQUESTS_CA_BUNDLE={ca}",
f"NODE_EXTRA_CA_CERTS={ca}",
]
# The gateway name resolves only through the bottle's /etc/hosts, which a
# nested container does not inherit, so hand it the address instead.
for var in ("HTTP_PROXY", "HTTPS_PROXY", "http_proxy", "https_proxy"):
value = os.environ.get(var)
if value:
entries.append(f"{var}={value.replace(name, ip)}")
# NO_PROXY keeps the name: it is matched against what a client asks for, and
# code inside a nested container still says "bot-bottle-gateway".
for var in ("NO_PROXY", "no_proxy"):
value = os.environ.get(var)
if value:
entries.append(f"{var}={value}")
path = Path(os.environ["CONTAINERS_CONF"])
path.write_text("\n".join([
"[containers]",
'cgroups="disabled"',
# podman copies the host's proxy vars into every container by default,
# and that copy *wins* over the env below — putting the unresolvable
# gateway name back. Turn it off so the address-bearing URLs stand.
"http_proxy=false",
'hosts_file="/etc/hosts"',
f'volumes=["{ca}:{ca}:ro"]',
"env=[",
*[f' "{entry}",' for entry in entries],
"]",
"[engine]",
'cgroup_manager="cgroupfs"',
'events_logger="file"',
"",
]), encoding="utf-8")
path.chmod(0o600)
PY
# Registry pulls egress through the bottle's proxy like everything else. The
# token-bearing proxy URL is already in the agent's environment; persisting it
# inside this disposable VM does not broaden its authority.
#
# This file is also what the Docker CLI copies into every container it starts,
# and being client-side it beats anything the podman service does — it is why
# containers.conf `env`, `http_proxy=false`, and the service's own environment
# all failed to change what a nested container saw. The address goes in here
# for the same reason it goes everywhere else: `bot-bottle-gateway` resolves
# in the bottle, never inside a nested container.
GATEWAY_NAME="$GATEWAY_NAME" GATEWAY_IP="$gateway_ip" python3 - <<'PY'
import json
import os
from pathlib import Path
name = os.environ["GATEWAY_NAME"]
ip = os.environ["GATEWAY_IP"]
proxy = os.environ.get("HTTPS_PROXY") or os.environ.get("https_proxy", "")
# NO_PROXY keeps the name: it is matched against what a client asks for, and
# code inside a nested container still says "bot-bottle-gateway".
no_proxy = os.environ.get("NO_PROXY") or os.environ.get("no_proxy", "")
config = {"proxies": {"default": {
"httpProxy": proxy.replace(name, ip),
"httpsProxy": proxy.replace(name, ip),
"noProxy": no_proxy,
}}}
path = Path.home() / ".docker" / "config.json"
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(json.dumps(config), encoding="utf-8")
path.chmod(0o600)
PY
if docker info >/dev/null 2>&1; then
exit 0
fi
# Belt to the ~/.docker/config.json braces above, which is what actually
# decides this for `docker run`. The service environment is what podman falls
# back to for anything the CLI does not stamp — its own registry pulls, and
# containers created through the API by something other than the Docker CLI.
# Cheap, and it keeps the address consistent across both paths.
#
# Assigned via parameter expansion, never echoed: these carry the bottle's
# identity token.
for var in HTTP_PROXY HTTPS_PROXY http_proxy https_proxy; do
eval "value=\${$var:-}"
[ -n "$value" ] || continue
eval "export $var=\"\${value%%$GATEWAY_NAME*}$gateway_ip\${value#*$GATEWAY_NAME}\""
done
log=/tmp/bot-bottle-nested-containers.log
nohup podman system service --time=0 \
"unix://$XDG_RUNTIME_DIR/podman.sock" \
>"$log" 2>&1 </dev/null &
@@ -0,0 +1,161 @@
"""Guest-local container engine for Apple-container bottles (issue #392).
The service and every nested container remain inside the existing per-bottle
VM. This module refuses to compensate for missing prerequisites with outer
capabilities, a privileged container, or a host Docker socket.
Podman is used rather than rootless Docker for one specific reason: Apple
Container's capability bounding set omits `CAP_SYS_ADMIN`, which the kernel
requires to write a multi-range `uid_map` via `newuidmap`. Rootless Docker
has no path that avoids that write. Podman does with no subordinate UID
range configured it falls back to a single-UID self-mapping, which an
unprivileged process may write itself. See
`docs/research/rootless-docker-in-apple-container-spike.md`.
That fallback is why `build_image` *removes* the agent user's `/etc/subuid`
and `/etc/subgid` entries instead of adding them: their presence is precisely
what would send podman down the `newuidmap` path that cannot work here.
The agent still talks to `docker` and `docker compose`; those speak to
podman's Docker-compatible API socket, so nothing in the agent's habits
changes.
"""
from __future__ import annotations
import shlex
import shutil
import tempfile
import time
from pathlib import Path
from typing import Callable
from ...log import die, info
_INIT = "/usr/local/libexec/bot-bottle/nested-containers-init"
# Deliberately cryptic and short. podman derives conmon's attach socket as
# `$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`, and a Unix socket
# path may not exceed 108 bytes (`sun_path`). The descriptive
# `/tmp/bot-bottle-podman-run` produced a 116-byte path — over the limit, so
# attach would have broken as soon as anything got far enough to attach. Do
# not lengthen this for readability; it buys 8 bytes of headroom.
_RUNTIME_DIR = "/tmp/bbp"
_SOCKET = f"{_RUNTIME_DIR}/podman.sock"
_LOG = "/tmp/bot-bottle-nested-containers.log"
IMAGE_SUFFIX = "-nested-containers"
READY_RETRIES = 30
# Apple Container creates both device nodes 0600 root:root, so the agent user
# cannot open them: /dev/fuse blocks the fuse-overlayfs storage driver and
# /dev/net/tun blocks slirp4netns, which rootless podman uses for the default
# bridge network that stock compose files expect. Relaxing the modes needs no
# capability the bottle does not already hold — unlike CAP_SYS_ADMIN, which is
# what killed the rootless-Docker approach.
_GUEST_DEVICES = ("/dev/fuse", "/dev/net/tun")
def build_image(
base_image: str,
build: Callable[..., None],
) -> str:
"""Layer the nested-container tooling onto an already-built agent image.
Podman and its networking stack live here rather than in the base agent
images so that bottles without the flag pay no image-size cost.
# TODO(#394): replace this hand-rolled Dockerfile with a docker-layer
# abstraction once that infrastructure exists.
"""
image = f"{base_image}{IMAGE_SUFFIX}"
init_script = Path(__file__).with_name("nested-containers-init.sh")
with tempfile.TemporaryDirectory(prefix="bot-bottle-nested-containers.") as tmp:
context = Path(tmp)
shutil.copy2(init_script, context / "nested-containers-init.sh")
(context / "Dockerfile").write_text(
"FROM docker:28-cli AS docker_cli\n"
f"FROM {base_image}\n"
"USER root\n"
"COPY --from=docker_cli /usr/local/bin/docker /usr/local/bin/docker\n"
"COPY --from=docker_cli /usr/local/libexec/docker/cli-plugins/"
"docker-compose /usr/local/libexec/docker/cli-plugins/docker-compose\n"
"RUN apt-get update \\\n"
# podman 5's networking stack, installed explicitly because
# --no-install-recommends omits it and each missing piece fails
# at a different, misleading layer:
# podman -> moved here from the base agent images so that
# bottles without nested_containers pay no cost
# passt -> `pasta`, the default rootless netns helper
# (podman 4 used slirp4netns); without it
# nothing starts: "could not find pasta"
# nftables -> `nft`, which netavark shells out to for the
# bridge network every compose file expects
# aardvark-dns -> name resolution *inside* nested containers;
# without it DNS fails while everything else
# looks healthy
# slirp4netns stays as the documented fallback for pasta.
" && apt-get install -y --no-install-recommends "
"aardvark-dns fuse-overlayfs netavark nftables passt podman "
"slirp4netns uidmap \\\n"
" && rm -rf /var/lib/apt/lists/* \\\n"
# Deliberate: an empty subordinate range keeps podman on the
# single-UID mapping that needs no CAP_SYS_ADMIN. Adding ranges
# here would reintroduce the newuidmap failure this design exists
# to route around.
" && sed -i '/^node:/d' /etc/subuid /etc/subgid\n"
"COPY nested-containers-init.sh "
"/usr/local/libexec/bot-bottle/nested-containers-init\n"
"RUN chmod 0755 /usr/local/libexec/bot-bottle/nested-containers-init\n"
"USER node\n",
encoding="utf-8",
)
build(image, str(context), dockerfile=str(context / "Dockerfile"))
return image
def guest_env(enabled: bool) -> dict[str, str]:
"""Environment consumed by the Docker CLI inside an enabled bottle."""
if not enabled:
return {}
return {
"DOCKER_HOST": f"unix://{_SOCKET}",
"XDG_RUNTIME_DIR": _RUNTIME_DIR,
}
def prepare_guest_devices(container_name: str, exec_as_root: Callable[..., None]) -> None:
"""Make /dev/fuse and /dev/net/tun openable by the agent user.
Runs as root inside the bottle because the agent must not be able to
re-mode device nodes itself. No outer capability is involved.
"""
exec_as_root(
container_name,
["sh", "-c", f"chmod 0666 {' '.join(_GUEST_DEVICES)}"],
)
def start(bottle: object) -> None:
"""Start and verify the unprivileged service through the bottle exec API."""
info("starting guest-local container engine")
result = bottle.exec(shlex.quote(_INIT)) # type: ignore[attr-defined]
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(f"nested-container bootstrap failed: {detail or '<no output>'}")
for _ in range(READY_RETRIES):
result = bottle.exec("docker info >/dev/null 2>&1") # type: ignore[attr-defined]
if result.returncode == 0:
info("guest-local container engine is ready")
return
time.sleep(0.2)
logs = bottle.exec( # type: ignore[attr-defined]
f"tail -n 80 {_LOG} 2>/dev/null || true"
)
die(
"guest-local container engine did not become ready without additional "
f"outer privileges:\n{(logs.stdout or logs.stderr or '<no log>').strip()}"
)
__all__ = ["build_image", "guest_env", "prepare_guest_devices", "start"]
@@ -0,0 +1,205 @@
"""The macOS orchestrator (control plane) as an Apple container (PRD 0070).
`MacosOrchestrator` is the Apple-Container implementation of the backend-neutral
`Orchestrator` service. The lean control-plane container joins the host-only
control network only (agents are never on it), mounts a container-only DB volume
(exactly one kernel writes `bot-bottle.db`), and holds the signing key (#469).
The host CLI and the gateway both reach it at its control-network address
Apple has no container DNS, so there's a single resolved URL, not docker's
loopback-vs-name split.
"""
from __future__ import annotations
import os
import time
import urllib.error
import urllib.request
from pathlib import Path
from ... import log
from ...paths import (
ORCHESTRATOR_TOKEN_ENV,
host_orchestrator_token,
)
from ...orchestrator.lifecycle import (
DEFAULT_HEALTH_TIMEOUT_SECONDS,
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
Orchestrator,
OrchestratorStartError,
source_hash,
)
from . import util as container_mod
from .gateway import CONTROL_NETWORK
ORCHESTRATOR_IMAGE = os.environ.get(
"BOT_BOTTLE_ORCHESTRATOR_IMAGE", "bot-bottle-orchestrator:latest"
)
ORCHESTRATOR_NAME = "bot-bottle-mac-orchestrator"
ORCHESTRATOR_LABEL = "bot-bottle-mac-orchestrator=1"
# Container-only volume holding bot-bottle.db, mounted ONLY into the
# orchestrator. One kernel writes it (never host-shared or cross-guest).
ORCHESTRATOR_DB_VOLUME = "bot-bottle-mac-db"
# BOT_BOTTLE_ROOT inside the orchestrator; host_db_path() resolves the DB to
# <root>/db/<filename>.
_DB_ROOT_IN_CONTAINER = "/var/lib/bot-bottle"
_SRC_IN_CONTAINER = "/bot-bottle-src"
_HEALTH_POLL_SECONDS = 0.25
_REPO_ROOT = Path(__file__).resolve().parents[3]
class MacosOrchestrator(Orchestrator):
"""The control plane as an Apple container on the host-only control network.
`ensure_built` builds `Dockerfile.orchestrator`; `ensure_running` starts it
and blocks until `/health` answers at its control-network address."""
def __init__(
self,
image_ref: str = ORCHESTRATOR_IMAGE,
*,
name: str = ORCHESTRATOR_NAME,
label: str = ORCHESTRATOR_LABEL,
port: int = DEFAULT_PORT,
control_network: str = CONTROL_NETWORK,
repo_root: Path = _REPO_ROOT,
db_volume: str = ORCHESTRATOR_DB_VOLUME,
) -> None:
self.image_ref = image_ref
self.name = name
self.label = label
self.port = port
self.control_network = control_network
self._repo_root = repo_root
self._db_volume = db_volume
def url(self) -> str:
"""The orchestrator's control-network address (host CLI + registration),
or "" while it has no address yet. Apple has no container DNS, so this is
also what the gateway resolves against (`gateway_url`)."""
ip = container_mod.try_container_ipv4_on_network(self.name, self.control_network)
return f"http://{ip}:{self.port}" if ip else ""
def gateway_url(self) -> str:
"""Same address the host uses — the gateway reaches the orchestrator by
control-network IP (Apple has no container DNS)."""
return self.url()
def is_healthy(self, *, timeout: float = DEFAULT_HEALTH_TIMEOUT_SECONDS) -> bool:
url = self.url()
if not url:
return False
try:
with urllib.request.urlopen(f"{url}/health", timeout=timeout) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
def is_running(self) -> bool:
return container_mod.container_is_running(self.name)
def _source_current(self, current_hash: str) -> bool:
"""True iff the running orchestrator was created from the current
bind-mounted control-plane source (it loads that code at startup and
won't reload it)."""
if not self.is_running():
return False
env = container_mod.container_env(self.name)
if not env:
return True # can't compare → don't churn a working container
return env.get("BOT_BOTTLE_SOURCE_HASH") == current_hash
def ensure_built(self) -> None:
"""Build the control-plane image. The source is bind-mounted so a code
change takes effect without a rebuild; the image still carries the
package for its entrypoint."""
container_mod.build_image(
self.image_ref, str(self._repo_root), dockerfile="Dockerfile.orchestrator")
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> None:
"""Ensure the control-plane container is up on current source; block
until healthy. Idempotent a healthy orchestrator on current source is
left untouched. Raises `OrchestratorStartError` on startup timeout.
The control network must already exist (the composer ensures it)."""
current_hash = source_hash(self._repo_root)
if self._source_current(current_hash) and self.is_healthy():
return
log.info("starting orchestrator container", context={"name": self.name})
self._run_container(current_hash)
self._wait_healthy(startup_timeout)
def _run_container(self, current_hash: str) -> None:
container_mod.force_remove_container(self.name)
_signing_key = host_orchestrator_token()
argv = [
"container", "run", "--detach",
"--name", self.name,
"--label", "bot-bottle.backend=macos-container",
"--label", self.label,
# Control network only — agents are never on it (L3-isolated).
"--network", self.control_network,
"--dns", container_mod.dns_server(),
# Container-only DB volume: exactly one kernel writes bot-bottle.db.
"--volume", f"{self._db_volume}:{_DB_ROOT_IN_CONTAINER}",
# Live control-plane source (a code change takes effect on relaunch).
"--mount",
container_mod.bind_mount_spec(
str(self._repo_root), _SRC_IN_CONTAINER, readonly=True),
"--env", f"PYTHONPATH={_SRC_IN_CONTAINER}",
"--env", f"BOT_BOTTLE_ROOT={_DB_ROOT_IN_CONTAINER}",
# Detect a real control-plane code change and recreate.
"--env", f"BOT_BOTTLE_SOURCE_HASH={current_hash}",
# The signing key — held ONLY by the orchestrator (issue #469). Bare
# `--env NAME` keeps the value off argv / `container inspect`.
"--env", ORCHESTRATOR_TOKEN_ENV,
self.image_ref,
# Dockerfile.orchestrator ENTRYPOINT is `-m bot_bottle.orchestrator`.
"--host", "0.0.0.0", "--port", str(self.port), "--broker", "stub",
]
result = container_mod.run_container_argv(
argv, env={**os.environ, ORCHESTRATOR_TOKEN_ENV: _signing_key})
if result.returncode != 0:
raise OrchestratorStartError(
f"orchestrator container failed to start: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _wait_healthy(self, startup_timeout: float) -> None:
deadline = time.monotonic() + startup_timeout
while True:
if self.is_healthy():
log.info("orchestrator healthy", context={"url": self.url()})
return
if time.monotonic() >= deadline:
raise OrchestratorStartError(
f"orchestrator did not become healthy within "
f"{startup_timeout:g}s"
)
time.sleep(_HEALTH_POLL_SECONDS)
def stop(self) -> None:
"""Remove the control-plane container (idempotent). The DB volume
persists."""
container_mod.force_remove_container(self.name)
def probe_orchestrator_url(port: int = DEFAULT_PORT) -> str:
"""The running orchestrator's control-plane URL, or "" if it isn't up. Used
by host-side control-plane discovery; safe on any host (returns "" when the
container or the `container` CLI isn't present)."""
ip = container_mod.try_container_ipv4_on_network(ORCHESTRATOR_NAME, CONTROL_NETWORK)
return f"http://{ip}:{port}" if ip else ""
__all__ = [
"MacosOrchestrator",
"ORCHESTRATOR_NAME",
"ORCHESTRATOR_LABEL",
"ORCHESTRATOR_IMAGE",
"ORCHESTRATOR_DB_VOLUME",
"probe_orchestrator_url",
]
@@ -8,7 +8,7 @@ from ...agent_provider import AgentProvisionPlan
from ...egress import EgressPlan
from ...env import ResolvedEnv
from ...git_gate import GitGatePlan
from ...supervise import SupervisePlan
from ...supervisor.plan import SupervisePlan
from ...manifest import Manifest
from .. import BottleSpec
from . import util as container_mod
@@ -44,4 +44,5 @@ def resolve_plan(
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_provision=agent_provision_plan,
nested_containers=manifest.bottle.nested_containers,
)
@@ -361,6 +361,12 @@ def exec_container(name: str, argv: list[str]) -> None:
)
def read_container_env(name: str, env_name: str) -> str:
"""Read one configured env value from a running container, or ``""``."""
result = _run_container_op([_CONTAINER, "exec", name, "printenv", env_name])
return result.stdout.strip() if result.returncode == 0 else ""
def exec_container_as_root(name: str, argv: list[str]) -> None:
"""`exec_container`, but as uid 0 inside the container.
+71
View File
@@ -0,0 +1,71 @@
"""Bottle-level provisioning for the consolidated launch sequence (PRD 0070).
Register a bottle with the orchestrator and provision its git-gate state into
the shared gateway (`provision_bottle`), and the inverse teardown
(`deprovision_bottle`). Backend-neutral each backend's consolidated_launch
drives these through its own `GatewayTransport` rather than re-implementing
them. The git-gate half of the work lives in `provision_gateway`.
"""
from __future__ import annotations
import dataclasses
from ..egress import EgressPlan
from ..git_gate import GitGatePlan
from ..orchestrator.client import OrchestratorClient, RegisteredBottle
from ..orchestrator.registration import registration_inputs
from ..orchestrator.store.secret_store import new_env_var_secret
from .provision_gateway import GatewayTransport, deprovision_git_gate, provision_git_gate
def provision_bottle(
client: OrchestratorClient,
source_ip: str,
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
transport: GatewayTransport,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
env_var_secret: str | None = None,
) -> RegisteredBottle:
"""Register the bottle and provision its git-gate state. Rolls back the
registration if provisioning fails so no orphan is left.
Generates a fresh ENV_VAR_SECRET, passes it to the orchestrator so it can
encrypt the token values at rest, and stamps the secret onto the returned
``RegisteredBottle`` so callers can inject it into the agent container's
environment."""
inputs = registration_inputs(egress_plan)
env_var_secret = env_var_secret or new_env_var_secret()
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens, env_var_secret=env_var_secret,
)
try:
provision_git_gate(transport, reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
return dataclasses.replace(reg, env_var_secret=env_var_secret)
def deprovision_bottle(
bottle_id: str,
transport: GatewayTransport,
*,
orchestrator_url: str,
timeout: float | None = None,
) -> None:
"""Deregister the bottle and remove its git-gate state. Both steps are
idempotent so this is safe from a cleanup trap."""
from ..orchestrator.store.config_store import DEFAULT_TEARDOWN_TIMEOUT_SECONDS
OrchestratorClient(
orchestrator_url,
timeout=timeout if timeout is not None else DEFAULT_TEARDOWN_TIMEOUT_SECONDS,
).teardown_bottle(bottle_id)
deprovision_git_gate(transport, bottle_id)
__all__ = ["provision_bottle", "deprovision_bottle"]
@@ -1,12 +1,14 @@
"""Provision one bottle's git-gate state into the running shared gateway
(PRD 0070, docker slice).
(PRD 0070). Backend-neutral: it drives any `GatewayTransport` (docker/apple
exec+cp, firecracker SSH), and the guest-side paths it writes are identical
inside every backend's gateway because they all run the same gateway image.
The consolidated gateway serves every bottle's repos under `/git/<bottle_id>/`
with per-repo credentials under `/git-gate/creds/<bottle_id>/`. When a bottle
is registered the launcher must place *its* deploy keys + known_hosts into
that per-bottle creds dir and init its bare repos there so this copies the
credential files into the live gateway container and runs the (namespaced,
init-only) provisioning script produced by `git_gate_render_provision`.
credential files into the live gateway and runs the (namespaced, init-only)
provisioning script produced by `git_gate_render_provision`.
Isolating each bottle's creds dir + repo root by id is what keeps one
bottle's push credentials out of another's repos on the shared gateway.
@@ -15,10 +17,9 @@ bottle's push credentials out of another's repos on the shared gateway.
from __future__ import annotations
import re
from typing import Protocol
from ...docker_cmd import run_docker
from ...git_gate import GitGatePlan, git_gate_render_provision
from ..git_gate import GitGatePlan, git_gate_render_provision
from ..gateway import GatewayProvisionError, GatewayTransport
# bottle ids index the gateway's per-bottle repo + creds dirs; they land in
# exec/cp path arguments, so validate before any path is built (a traversal
@@ -27,46 +28,6 @@ from ...git_gate import GitGatePlan, git_gate_render_provision
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
class GatewayProvisionError(RuntimeError):
"""A git-gate provisioning step against the running gateway failed."""
class GatewayTransport(Protocol):
"""How the launcher stages files + runs commands in the running gateway.
Backend-neutral so the same provisioning logic serves the docker gateway
(exec/cp over the docker socket) and the firecracker gateway VM (over
SSH)."""
def exec(self, argv: list[str]) -> None:
"""Run `argv` in the gateway, raising `GatewayProvisionError` on
failure."""
def cp_into(self, src: str, dest: str) -> None:
"""Copy host file `src` to `dest` in the gateway, raising on
failure."""
class DockerGatewayTransport:
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
def __init__(self, gateway: str) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
proc = run_docker(["docker", "exec", self.gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def cp_into(self, src: str, dest: str) -> None:
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
def _require_safe(bottle_id: str) -> None:
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise GatewayProvisionError(f"unsafe bottle id {bottle_id!r}")
@@ -128,5 +89,5 @@ def deprovision_git_gate(transport: GatewayTransport, bottle_id: str) -> None:
__all__ = [
"provision_git_gate", "deprovision_git_gate",
"GatewayProvisionError", "GatewayTransport", "DockerGatewayTransport",
"GatewayProvisionError", "GatewayTransport",
]
+21 -2
View File
@@ -26,8 +26,10 @@ from ..bottle_state import (
)
from ..egress import Egress, EgressPlan
from ..git_gate import GitGate, GitGatePlan
from ..log import die
from ..manifest import Manifest, ManifestBottle
from ..supervise import Supervise, SupervisePlan
from ..supervisor.plan import SupervisePlan
from ..orchestrator.supervisor import Supervisor
from . import BottleSpec
@@ -100,7 +102,7 @@ def prepare_supervise(bottle: ManifestBottle, slug: str) -> SupervisePlan | None
return None
supervise_dir = supervise_state_dir(slug)
supervise_dir.mkdir(parents=True, exist_ok=True)
return Supervise().prepare(slug, supervise_dir)
return Supervisor().prepare(slug, supervise_dir)
def merge_provision_env_vars(provision: AgentProvisionPlan) -> AgentProvisionPlan:
@@ -112,6 +114,22 @@ def merge_provision_env_vars(provision: AgentProvisionPlan) -> AgentProvisionPla
return replace(provision, guest_env=merged)
def reject_nested_containers(backend: str, manifest: Manifest) -> None:
"""Fail loudly when a backend cannot honor `nested_containers: true`.
Silently ignoring it would hand the agent a bottle where `docker` is not
there and the only sound alternatives on these backends (a host daemon
socket, a privileged container) are exactly what issue #392 rules out.
"""
if not manifest.bottle.nested_containers:
return
die(
f"nested_containers is not supported on the {backend} backend. "
"Only macos-container runs a guest-local container engine today; "
"mounting the host Docker socket is not an option bot-bottle offers."
)
def resolve_manifest_dockerfile(path_value: str, spec: BottleSpec) -> str:
"""Resolve a manifest-supplied dockerfile path relative to user_cwd."""
path = Path(os.path.expanduser(path_value))
@@ -122,6 +140,7 @@ def resolve_manifest_dockerfile(path_value: str, spec: BottleSpec) -> str:
__all__ = [
"merge_provision_env_vars",
"reject_nested_containers",
"mint_slug",
"prepare_agent_state_dir",
"prepare_egress",
+211
View File
@@ -0,0 +1,211 @@
"""Backend registry, selection, and active-agent enumeration.
Resolves which bottle backend to use (explicit name / `BOT_BOTTLE_BACKEND` /
auto-select), and enumerates running agents across every available backend. The
three concrete backends are imported lazily inside `_get_backends` so this
module and anything that only needs to *select* a backend stays cheap.
"""
from __future__ import annotations
import os
import sys
from typing import Any
from ..log import die, info, warn
from ..util import read_tty_line
from .base import ActiveAgent, BackendStatus, BottleBackend
# _backends is None until the first call to _get_backends(), at which
# point all three concrete backend classes are imported and instantiated.
# Keeping the imports out of module scope means that importing any
# backend sub-module (e.g. `backend.docker.util`) no longer drags the
# firecracker and macos-container implementations into memory.
#
# Tests may replace _backends with a {name: fake} dict via patch.object;
# _get_backends() returns the current module-level value as-is when it
# is not None, so test fakes take effect without triggering real imports.
_backends: dict[str, BottleBackend[Any, Any]] | None = None
def _get_backends() -> dict[str, BottleBackend[Any, Any]]:
"""Return the registry of all backend instances, loading lazily on first call."""
global _backends # pylint: disable=global-statement
if _backends is None:
from .docker import DockerBottleBackend
from .firecracker import FirecrackerBottleBackend
from .macos_container import MacosContainerBottleBackend
_backends = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
}
return _backends
def get_bottle_backend(
name: str | None = None,
*,
prompt: bool = True,
) -> BottleBackend[Any, Any]:
"""Resolve the bottle backend.
`name` precedence:
1. explicit arg (e.g. resume passes the recorded backend name)
2. BOT_BOTTLE_BACKEND env var
3. auto-selection: VM backend first, docker fallback with prompt
`prompt` controls whether auto-selection may block on an interactive
[i/d/q] prompt when falling back to docker. Pass `prompt=False` in
non-interactive contexts (headless launches, CI) so the call dies
with an actionable message instead of hanging.
Dies with a pointer at the known backends if the chosen name
isn't implemented."""
resolved = name or os.environ.get("BOT_BOTTLE_BACKEND")
if resolved is None:
resolved = _auto_select_backend(prompt=prompt)
backends = _get_backends()
if resolved not in backends:
known = ", ".join(sorted(backends))
die(f"unknown backend {resolved!r}; known backends: {known}")
return backends[resolved]
def _platform_vm_suggestion() -> str:
"""Platform-appropriate VM backend name for install suggestions."""
return "macos-container" if sys.platform == "darwin" else "firecracker"
def _print_vm_install_instructions() -> None:
"""Print platform-appropriate VM backend install instructions to stderr."""
vm = _platform_vm_suggestion()
if vm == "macos-container":
info("Install Apple Container: https://github.com/apple/container/releases")
info("Then start the service: container system start")
else:
info("Install Firecracker: https://github.com/firecracker-microvm/firecracker/releases")
info("Configure the host: ./cli.py backend setup")
def _auto_select_backend(prompt: bool = True) -> str:
"""Tier-1 / tier-2 backend auto-selection.
Tier 1: VM backend macos-container on macOS when Apple Container is
installed; firecracker on KVM-capable Linux even before the binary is
present (its preflight prints an install pointer).
Tier 2: docker, with a security warning and an interactive prompt.
When `prompt=False` (headless / CI), dies with an actionable message
instead of blocking on a TTY read. When docker is also absent, prints
VM install instructions and exits.
"""
# --- Tier 1: VM backend -----------------------------------------
if has_backend("macos-container"):
return "macos-container"
# A KVM-capable Linux host defaults to firecracker even when the
# `firecracker` binary isn't installed yet: selecting it here routes
# start through firecracker's preflight, which prints an install
# pointer, instead of silently falling back to docker.
from .firecracker import FirecrackerBottleBackend
if FirecrackerBottleBackend.is_host_capable():
return "firecracker"
# --- Tier 2: docker fallback ------------------------------------
if not has_backend("docker"):
info("No backend available on this host.")
_print_vm_install_instructions()
die("no backend available; install a VM backend and re-run")
vm = _platform_vm_suggestion()
warn(
"docker is less secure than VM backends — "
"containers share the host kernel."
)
if not prompt:
die(
f"no VM backend available; set BOT_BOTTLE_BACKEND=docker to proceed "
f"with docker, or install the {vm!r} backend."
)
sys.stderr.write(
f"bot-bottle: For better isolation, install the {vm!r} backend.\n"
f" [i] show {vm} install instructions and exit\n"
" [d] use docker anyway\n"
" [q] quit\n"
"bot-bottle: choice [i/d/q]: "
)
sys.stderr.flush()
reply = read_tty_line().strip().lower()
if reply == "d":
return "docker"
if reply == "i":
_print_vm_install_instructions()
die("not proceeding with docker; install a VM backend or set BOT_BOTTLE_BACKEND=docker")
def known_backend_names() -> tuple[str, ...]:
"""Sorted tuple of all backend keys in `_get_backends()`. Used by
argparse (`--backend` choices) and the dashboard's backend
picker."""
return tuple(sorted(_get_backends()))
def has_backend(name: str) -> bool:
"""Whether the named backend's runtime prerequisites are
available on the current host. Cross-backend callers (list,
cleanup) skip unavailable backends so a docker-only host
doesn't fail when the firecracker backend isn't usable,
and vice versa.
Returns False for unknown names so callers can pass
arbitrary input without separate validation."""
backends = _get_backends()
if name not in backends:
return False
return backends[name].is_available()
def is_backend_available(name: str) -> bool:
"""Cheap availability check: is the backend's binary on PATH?
Suitable for cleanup enumeration and auto-selection does NOT probe
the daemon or network pool. Use is_backend_ready() for a full
readiness check before launching tests."""
return has_backend(name)
def is_backend_ready(name: str, *, quiet: bool = False) -> bool:
"""Full readiness check: passes all of the backend's status() checks.
When quiet=False the backend prints diagnostic output explaining what
is missing intended for test-suite guards that run at discovery time
so the operator sees a concrete failure reason for each skip.
Returns False for unknown backend names."""
backends = _get_backends()
if name not in backends:
return False
return backends[name].status(quiet=quiet) == BackendStatus.READY
def enumerate_active_agents() -> list[ActiveAgent]:
"""All currently-running agents, across every available
backend. Used by CLI `active` and the dashboard's agents
pane so neither has to know which backends exist. Skips
backends whose `is_available()` reports False.
Sorted by `(started_at, slug)` so the list is stable across
dashboard refresh ticks agents don't shift position while
the operator navigates with arrow keys. ISO 8601 timestamps
sort lexicographically in chronological order; `slug` is the
deterministic tiebreaker. Agents with missing metadata
(`started_at == ""`) sort first."""
out: list[ActiveAgent] = []
backends = _get_backends()
for name in sorted(backends):
if not backends[name].is_available():
continue
out.extend(backends[name].enumerate_active())
out.sort(key=lambda a: (a.started_at, a.slug))
return out
+9 -110
View File
@@ -1,116 +1,15 @@
"""Main CLI dispatcher.
"""bot-bottle CLI package.
Commands: backend, cleanup, commit, edit, info, init, list, resume, start, supervise
The subcommand handlers live in `commands/` and are assembled into the
COMMANDS registry by `commands/__init__.py`; the dispatcher `main()` lives
in `__main__.py`. They are re-exported here so `bot_bottle.cli.main`,
`bot_bottle.cli.COMMANDS`, and `bot_bottle.cli.NO_MIGRATION_COMMANDS` stay
importable (the repo-root `cli.py` entry point and the tests use them).
"""
from __future__ import annotations
import sys
from .__main__ import main
from .commands import COMMANDS, NO_MIGRATION_COMMANDS
from ..errors import MissingEnvVarError
from ..log import Die, die, error
from ..manifest import ManifestError
from ..store_manager import StoreManager
from ._common import PROG
from . import list as _list_mod
from .backend import cmd_backend
from .cleanup import cmd_cleanup
from .commit import cmd_commit
from .edit import cmd_edit
from .info import cmd_info
from .init import cmd_init
from .login import cmd_login
from .resume import cmd_resume
from .start import cmd_start
from .supervise import cmd_supervise
cmd_list = _list_mod.cmd_list
COMMANDS = {
"backend": cmd_backend,
"cleanup": cmd_cleanup,
"commit": cmd_commit,
"edit": cmd_edit,
"info": cmd_info,
"init": cmd_init,
"list": cmd_list,
"login": cmd_login,
"resume": cmd_resume,
"start": cmd_start,
"supervise": cmd_supervise,
}
# Commands that manage host prerequisites (or are otherwise store-free) and
# must run before — or without — a migrated DB. `backend` provisions/probes
# the host (TAP pool, /dev/kvm, firecracker) and never opens the store, so
# gating it on the schema breaks preflight on a fresh CI runner where stdin
# isn't a TTY and the migration prompt can't be answered.
NO_MIGRATION_COMMANDS = frozenset({"backend", "login"})
def usage() -> None:
sys.stderr.write(f"usage: {PROG} <command> [args...]\n\n")
sys.stderr.write("Commands:\n")
sys.stderr.write(" backend set up / check / undo a backend's host prerequisites (setup|status|teardown)\n")
sys.stderr.write(" cleanup stop and remove all active bot-bottle containers\n")
sys.stderr.write(" commit snapshot a running bottle's container state to a Docker image\n")
sys.stderr.write(" edit open an agent in vim for editing\n")
sys.stderr.write(" info print env, skills, and prompt details for a named agent\n")
sys.stderr.write(" init interactively create a new agent and add it to bot-bottle.json\n")
sys.stderr.write(" list list available agents or active containers\n")
sys.stderr.write(" login register this host with a bot-bottle console\n")
sys.stderr.write(
" resume re-launch a bottle by its identity "
"(continues state from PRD 0016)\n"
)
sys.stderr.write(
" start boot a container for a named agent and "
"attach an interactive session\n"
)
sys.stderr.write(
" supervise view + approve/modify/reject pending supervise "
"proposals (PRD 0013)\n\n"
)
sys.stderr.write(f"Run '{PROG} <command> --help' for command-specific usage.\n")
def main(argv: list[str] | None = None) -> int:
if argv is None:
argv = sys.argv[1:]
if not argv:
usage()
return 2
command = argv[0]
rest = argv[1:]
if command in ("-h", "--help"):
usage()
return 0
handler = COMMANDS.get(command)
if handler is None:
usage()
die(f"unknown command: {command}")
mgr = StoreManager.instance()
if command not in NO_MIGRATION_COMMANDS and not mgr.is_migrated():
sys.stderr.write("bot-bottle: database schema is out of date\n")
sys.stderr.write("Migrate now? [y/N] ")
sys.stderr.flush()
try:
answer = sys.stdin.readline().strip().lower()
except EOFError:
answer = ""
if answer != "y":
error("migration required — re-run and confirm to migrate")
return 1
mgr.migrate()
try:
return handler(rest) or 0
except MissingEnvVarError as e:
error(str(e))
return 1
except ManifestError as e:
error(str(e))
return 1
except Die as e:
return e.code if isinstance(e.code, int) else 1
except KeyboardInterrupt:
return 130
__all__ = ["main", "COMMANDS", "NO_MIGRATION_COMMANDS"]
+55 -5
View File
@@ -1,15 +1,65 @@
"""Entry point for `python -m bot_bottle.cli`.
"""Entry point + dispatcher for `python -m bot_bottle.cli`.
`cli.py` at the repo root is the usual way in; this makes the package
runnable too, so the CLI works from an installed copy where there is no
`cli.py` on disk to point at.
Maps `bot-bottle <command>` to its handler in the COMMANDS registry
(`bot_bottle.cli.commands`), enforces the schema-migration gate, and
translates handler exceptions into process exit codes. The repo-root
`cli.py` is the usual way in; this makes the package runnable too, so the
CLI works from an installed copy where there is no `cli.py` on disk.
"""
from __future__ import annotations
import sys
from . import main
from ..errors import MissingEnvVarError
from ..log import Die, die, error
from ..manifest import ManifestError
from ..orchestrator.store.store_manager import StoreManager
from .commands import COMMANDS, NO_MIGRATION_COMMANDS
from .commands.help import cmd_help
def main(argv: list[str] | None = None) -> int:
if argv is None:
argv = sys.argv[1:]
if not argv:
cmd_help()
return 2
command = argv[0]
rest = argv[1:]
if command in ("-h", "--help"):
cmd_help()
return 0
handler = COMMANDS.get(command)
if handler is None:
cmd_help()
die(f"unknown command: {command}")
mgr = StoreManager.instance()
if command not in NO_MIGRATION_COMMANDS and not mgr.is_migrated():
sys.stderr.write("bot-bottle: database schema is out of date\n")
sys.stderr.write("Migrate now? [y/N] ")
sys.stderr.flush()
try:
answer = sys.stdin.readline().strip().lower()
except EOFError:
answer = ""
if answer != "y":
error("migration required — re-run and confirm to migrate")
return 1
mgr.migrate()
try:
return handler(rest) or 0
except MissingEnvVarError as e:
error(str(e))
return 1
except ManifestError as e:
error(str(e))
return 1
except Die as e:
return e.code if isinstance(e.code, int) else 1
except KeyboardInterrupt:
return 130
if __name__ == "__main__":
sys.exit(main())
-12
View File
@@ -1,12 +0,0 @@
"""Shared constants and tty helper for cli subcommands."""
from __future__ import annotations
import os
from pathlib import Path
from ..util import read_tty_line as read_tty_line
PROG = "cli.py"
USER_CWD = os.getcwd()
REPO_DIR = str(Path(__file__).resolve().parent.parent.parent)
+58
View File
@@ -0,0 +1,58 @@
"""CLI subcommand registry.
One module per `bot-bottle <command>`, each exposing a `cmd_<name>(argv)`
handler. This package `__init__` maps command names to their handlers
**lazily**: a short-lived CLI run dispatches exactly one command, so
importing all twelve handlers (and their transitive deps backend,
manifest, orchestrator, ) up front is wasted work. Each COMMANDS value is
a thin wrapper that imports its handler's module on first call. Shared CLI
helpers (`constants`, `tui`) stay one level up in the `cli` package.
"""
from __future__ import annotations
from importlib import import_module
from typing import Callable
# command name -> "<submodule>:<handler attr>". Kept as strings so building
# the registry imports nothing; the module loads only when dispatched.
_HANDLERS: dict[str, str] = {
"active": "active:cmd_active",
"backend": "backend:cmd_backend",
"cleanup": "cleanup:cmd_cleanup",
"commit": "commit:cmd_commit",
"edit": "edit:cmd_edit",
"help": "help:cmd_help",
"init": "init:cmd_init",
"list": "list:cmd_list",
"login": "login:cmd_login",
"resume": "resume:cmd_resume",
"start": "start:cmd_start",
"supervise": "supervise:cmd_supervise",
}
def _lazy(spec: str) -> Callable[[list[str]], "int | None"]:
"""Wrap a `<module>:<attr>` handler so its module is imported only when
the command is actually dispatched, not when the registry is built."""
module, attr = spec.split(":")
def run(argv: list[str]) -> "int | None":
handler = getattr(import_module(f".{module}", __name__), attr)
return handler(argv)
run.__name__ = attr
return run
COMMANDS = {name: _lazy(spec) for name, spec in _HANDLERS.items()}
# Commands that manage host prerequisites (or are otherwise store-free) and
# must run before — or without — a migrated DB. `backend` provisions/probes
# the host (TAP pool, /dev/kvm, firecracker) and never opens the store, so
# gating it on the schema breaks preflight on a fresh CI runner where stdin
# isn't a TTY and the migration prompt can't be answered. `help` and `login`
# likewise never touch the store.
NO_MIGRATION_COMMANDS = frozenset({"backend", "help", "login"})
__all__ = ["COMMANDS", "NO_MIGRATION_COMMANDS"]
@@ -1,14 +1,12 @@
"""list: list available agents or active bottles."""
"""active: list currently-running bot-bottle bottles."""
from __future__ import annotations
import argparse
import os
import sys
from ..backend import enumerate_active_agents
from ..manifest import ManifestIndex
from ._common import PROG, USER_CWD
from ...backend import enumerate_active_agents
from ..constants import PROG
_ANSI_COLOR_CODES: dict[str, str] = {
"red": "\033[91m",
@@ -34,20 +32,13 @@ def _ansi_label(text: str, color: str) -> str:
return f"{code}{text}{_ANSI_RESET}"
def cmd_list(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} list", add_help=True)
parser.add_argument("scope", choices=["available", "active"])
args = parser.parse_args(argv)
if args.scope == "available":
manifest = ManifestIndex.resolve(USER_CWD)
for name in manifest.all_agent_names:
print(name)
def cmd_active(argv: list[str]) -> int:
if argv and argv[0] in ("-h", "--help"):
sys.stderr.write(f"usage: {PROG} active\n")
sys.stderr.write("\nList all currently-running bot-bottle bottles.\n")
sys.stderr.write("Output: <backend>\\t<slug>\\t<label>\\t<services>\n")
return 0
# `active` enumerates every backend (docker, firecracker,
# macos-container) so non-docker bottles aren't hidden behind
# the env var.
active = enumerate_active_agents()
if not active:
print("no active bot-bottle bottles", file=sys.stderr)
@@ -15,8 +15,8 @@ from __future__ import annotations
import argparse
from ..backend import get_bottle_backend, known_backend_names
from ._common import PROG
from ...backend import get_bottle_backend, known_backend_names
from ..constants import PROG
def cmd_backend(args: list[str]) -> int:
@@ -21,9 +21,9 @@ from __future__ import annotations
import sys
from ..backend import get_bottle_backend, has_backend, known_backend_names
from ..log import info
from ._common import read_tty_line
from ...backend import get_bottle_backend, has_backend, known_backend_names
from ...log import info
from ...util import read_tty_line
def cmd_cleanup(_argv: list[str]) -> int:
@@ -12,12 +12,12 @@ from __future__ import annotations
import argparse
from ..backend import enumerate_active_agents
from ..backend.freeze import CommitCancelled, get_freezer
from ..bottle_state import read_metadata
from ..log import die
from ._common import PROG
from . import tui
from ...backend import enumerate_active_agents
from ...backend.freeze import CommitCancelled, get_freezer
from ...bottle_state import read_metadata
from ...log import die
from ..constants import PROG
from .. import tui
def cmd_commit(argv: list[str]) -> int:
@@ -27,7 +27,7 @@ def cmd_commit(argv: list[str]) -> int:
nargs="?",
default=None,
help=(
"bottle slug from `cli.py list active` "
"bottle slug from `cli.py active` "
"(omit to pick interactively)"
),
)
@@ -7,8 +7,8 @@ import json
import os
from pathlib import Path
from ..log import die
from ._common import PROG, USER_CWD
from ...log import die
from ..constants import PROG
def cmd_edit(argv: list[str]) -> int:
@@ -20,7 +20,7 @@ def cmd_edit(argv: list[str]) -> int:
if args.scope == "user":
target_file = Path(os.environ["HOME"]) / "bot-bottle.json"
else:
target_file = Path(USER_CWD) / "bot-bottle.json"
target_file = Path(os.getcwd()) / "bot-bottle.json"
if not target_file.is_file():
die(f"{target_file} does not exist")
+37
View File
@@ -0,0 +1,37 @@
"""help: print the top-level command list and usage.
Rendered by the dispatcher for the `help` command and for its
`-h`/`--help`, no-args, and unknown-command fallbacks. The per-command
summaries live here; keep them in sync with the COMMANDS table in
`bot_bottle.cli`.
"""
from __future__ import annotations
import sys
from ..constants import PROG
def cmd_help(argv: list[str] | None = None) -> int:
"""Write the top-level usage + command list to stderr. Returns 0;
the dispatcher chooses the process exit code per entry path (0 for an
explicit `help`/`-h`, 2 for the bare no-args usage error)."""
del argv # help takes no arguments
w = sys.stderr.write
w(f"usage: {PROG} <command> [args...]\n\n")
w("Commands:\n")
w(" active list currently-running bot-bottle bottles\n")
w(" backend set up / check / undo a backend's host prerequisites (setup|status|teardown)\n")
w(" cleanup stop and remove all active bot-bottle containers\n")
w(" commit snapshot a running bottle's container state to a Docker image\n")
w(" edit open an agent in vim for editing\n")
w(" help show this command list\n")
w(" init interactively create a new agent and add it to bot-bottle.json\n")
w(" list list available agents from bot-bottle.json\n")
w(" login register this host with a bot-bottle console\n")
w(" resume re-launch a bottle by its identity (continues state from PRD 0016)\n")
w(" start boot a container for a named agent and attach an interactive session\n")
w(" supervise view + approve/modify/reject pending supervise proposals (PRD 0013)\n\n")
w(f"Run '{PROG} <command> --help' for command-specific usage.\n")
return 0
@@ -10,8 +10,9 @@ import sys
from pathlib import Path
from typing import Any
from ..log import die, info, warn
from ._common import PROG, USER_CWD, read_tty_line
from ...log import die, info, warn
from ..constants import PROG
from ...util import read_tty_line
def cmd_init(argv: list[str]) -> int:
@@ -22,7 +23,7 @@ def cmd_init(argv: list[str]) -> int:
if args.scope == "user":
target_file = Path(os.environ["HOME"]) / "bot-bottle.json"
else:
target_file = Path(USER_CWD) / "bot-bottle.json"
target_file = Path(os.getcwd()) / "bot-bottle.json"
print(file=sys.stderr)
info(f"bot-bottle init — adding a new agent to {target_file}")
+21
View File
@@ -0,0 +1,21 @@
"""list: list available agents."""
from __future__ import annotations
import os
import sys
from ...manifest import ManifestIndex
from ..constants import PROG
def cmd_list(argv: list[str]) -> int:
if argv and argv[0] in ("-h", "--help"):
sys.stderr.write(f"usage: {PROG} list\n")
sys.stderr.write("\nList all available agents from bot-bottle.json.\n")
return 0
manifest = ManifestIndex.resolve(os.getcwd())
for name in manifest.all_agent_names:
print(name)
return 0
@@ -25,7 +25,7 @@ import urllib.request
from pathlib import Path
from typing import Any
from ..paths import bot_bottle_root
from ...paths import bot_bottle_root
_CONSOLE_URL_ENV = "BB_CONSOLE_URL"
_POLL_SLEEP = 2 # seconds between polls; matches console's poll_interval default
@@ -15,12 +15,13 @@ to bring up the replacement from the recorded state.
from __future__ import annotations
import argparse
import os
from ..backend import BottleSpec
from ..bottle_state import read_metadata
from ..log import die
from ..manifest import ManifestIndex
from ._common import PROG, USER_CWD
from ...backend import BottleSpec
from ...bottle_state import read_metadata
from ...log import die
from ...manifest import ManifestIndex
from ..constants import PROG
from .start import _launch_bottle
@@ -40,14 +41,14 @@ def cmd_resume(argv: list[str]) -> int:
f"check ~/.bot-bottle/state/ or run `cli.py start` to create a new bottle"
)
manifest = ManifestIndex.resolve(USER_CWD)
manifest = ManifestIndex.resolve(os.getcwd())
manifest.require_agent(metadata.agent_name)
spec = BottleSpec(
manifest=manifest,
agent_name=metadata.agent_name,
copy_cwd=metadata.copy_cwd,
user_cwd=metadata.cwd or USER_CWD,
user_cwd=metadata.cwd or os.getcwd(),
identity=metadata.identity,
bottle_names=tuple(metadata.bottle_names),
)
@@ -14,6 +14,7 @@ the private orchestrator `_launch_bottle`.
from __future__ import annotations
import argparse
import io
import os
import shutil
import sys
@@ -21,25 +22,26 @@ import tempfile
from pathlib import Path
from typing import Callable
from ..agent_provider import get_provider, runtime_for
from ..backend import (
from ...agent_provider import get_provider, runtime_for
from ...backend import (
Bottle,
BottleSpec,
enumerate_active_agents,
get_bottle_backend,
)
from ..backend.docker import util as docker_mod
from ..backend.docker.bottle_plan import DockerBottlePlan
from ..bottle_state import (
from ...backend.docker import util as docker_mod
from ...backend.docker.bottle_plan import DockerBottlePlan
from ...bottle_state import (
cleanup_state,
is_preserved,
mark_preserved,
)
from ..image_cache import StaleImageError
from ..log import info, die
from ..manifest import Manifest, ManifestIndex
from ._common import PROG, USER_CWD, read_tty_line
from . import tui
from ...image_cache import StaleImageError
from ...log import info, die
from ...manifest import Manifest, ManifestIndex
from ..constants import PROG
from ...util import read_tty_line
from .. import tui
def cmd_start(argv: list[str]) -> int:
@@ -115,7 +117,7 @@ def cmd_start(argv: list[str]) -> int:
# threading a no_cache field through every backend's plan dataclass.
os.environ["BOT_BOTTLE_NO_CACHE"] = "1"
manifest = ManifestIndex.resolve(USER_CWD)
manifest = ManifestIndex.resolve(os.getcwd())
if args.headless:
return _start_headless(
@@ -166,7 +168,7 @@ def cmd_start(argv: list[str]) -> int:
manifest=manifest,
agent_name=agent_name,
copy_cwd=args.cwd,
user_cwd=USER_CWD,
user_cwd=os.getcwd(),
label=label,
color=color,
bottle_names=bottle_names,
@@ -195,6 +197,16 @@ def _start_headless(
path, so the agent still execs on the inherited stdio/PTY an
orchestrator allocates that PTY and relays it to its
desktop/mobile clients."""
try:
stdin_fd = sys.stdin.fileno()
except io.UnsupportedOperation:
stdin_fd = -1
if not os.isatty(stdin_fd):
die(
"--headless requires a PTY on stdin; run via:\n"
" script -q /dev/null ./cli.py start ..."
)
agent_name = args.name
if not agent_name:
die("--headless requires an agent name: ./cli.py start <agent> --headless")
@@ -224,7 +236,7 @@ def _start_headless(
manifest=manifest,
agent_name=agent_name,
copy_cwd=args.cwd,
user_cwd=USER_CWD,
user_cwd=os.getcwd(),
label=label,
color=args.color or "",
bottle_names=bottle_names,
@@ -369,8 +381,8 @@ def _peek_agent_bottle(manifest: ManifestIndex, agent_name: str) -> str:
return manifest.agents[agent_name].bottle
return ""
from ..manifest_loader import scan_agent_names
from ..yaml_subset import YamlSubsetError, parse_frontmatter
from ...manifest.loader import scan_agent_names
from ...yaml_subset import YamlSubsetError, parse_frontmatter
home_agents = scan_agent_names(manifest.home_md / "agents")
cwd_agents: dict[str, Path] = {}
@@ -438,7 +450,7 @@ def _bottle_lineage(manifest: ManifestIndex) -> dict[str, str]:
if not bottles_dir.is_dir():
return {}
from ..yaml_subset import YamlSubsetError, parse_frontmatter
from ...yaml_subset import YamlSubsetError, parse_frontmatter
extends_of: dict[str, str] = {}
for path in bottles_dir.glob("*.md"):
@@ -524,6 +536,8 @@ def _manifest_to_yaml(manifest: Manifest) -> str:
lines.append(f" scheme: {r.AuthScheme}")
lines.append(f" supervise: {'true' if bottle.supervise else 'false'}")
if bottle.nested_containers:
lines.append(" nested_containers: true")
return "\n".join(lines)
@@ -19,22 +19,22 @@ from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from ..paths import bot_bottle_root
from ..log import Die, error, info
from ..orchestrator.client import (
from ...paths import bot_bottle_root
from ...log import Die, error, info
from ...orchestrator.client import (
OrchestratorClient,
OrchestratorClientError,
discover_orchestrator_url,
)
from ..supervise import (
from ...supervisor.types import (
Proposal,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
)
from ._common import PROG
from ..constants import PROG
_REFRESH_INTERVAL_MS = 1000
@@ -81,7 +81,7 @@ def _resolve_orchestrator_url() -> str:
try:
return discover_orchestrator_url()
except OrchestratorClientError:
from ..backend import get_bottle_backend
from ...backend import get_bottle_backend
backend = get_bottle_backend()
info(f"no orchestrator control plane running; starting one ({backend.name})…")
return backend.ensure_orchestrator()
+8
View File
@@ -0,0 +1,8 @@
"""Shared CLI constants.
Kept as a leaf module (imports nothing from the `cli` package) so both the
dispatcher (`cli/__init__.py`) and the command modules it imports can share
`PROG` without a circular import.
"""
PROG = "cli.py"
-49
View File
@@ -1,49 +0,0 @@
"""info: print env, skills, and prompt details for a named agent."""
from __future__ import annotations
import argparse
from ..log import info
from ..manifest import ManifestIndex
from ._common import PROG, USER_CWD
def cmd_info(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} info", add_help=True)
parser.add_argument("name", help="agent name defined in bot-bottle.json")
args = parser.parse_args(argv)
names = ManifestIndex.resolve(USER_CWD)
names.require_agent(args.name)
manifest = names.load_for_agent(args.name)
agent = manifest.agent
bottle = manifest.bottle
env_names = list(bottle.env.keys())
prompt_first_line = agent.prompt.splitlines()[0] if agent.prompt else ""
print()
info(f"agent : {args.name}")
info(f"env (names only): {', '.join(env_names) if env_names else '(none)'}")
info(f"skills : {' '.join(agent.skills) if agent.skills else '(none)'}")
info(
f"prompt : {len(agent.prompt)} chars; "
f"first line: {prompt_first_line or '(empty)'}"
)
info(f"bottle : {agent.bottle}")
identity = manifest.git_identity_summary()
if identity:
info(f" git identity : {identity}")
if bottle.git:
for e in bottle.git:
info(
f" git remote : {e.Name} -> {e.Upstream} "
f"(IdentityFile={e.IdentityFile})"
)
if e.KnownHostKey:
info(f" KnownHostKey: {e.KnownHostKey}")
else:
info(" git remotes : (none)")
print()
return 0
-1
View File
@@ -26,7 +26,6 @@ RUN apt-get update \
ca-certificates \
curl \
openssh-client \
podman \
ripgrep \
iproute2 \
dnsutils \
+16 -2
View File
@@ -35,6 +35,11 @@ if TYPE_CHECKING:
_SUPERVISE_MCP_NAME = "supervise"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
_IDENTITY_HEADER = "x-bot-bottle-identity"
# Placeholder stood in for the real identity token in the manual-recovery hint.
# The token is a per-bottle credential, so it must never be rendered into the
# host-side launch log (#476 review); the operator substitutes the value from
# inside the bottle (it rides in the agent's HTTPS_PROXY credentials).
_IDENTITY_TOKEN_PLACEHOLDER = "<bottle-identity-token>"
def _skills_dir(guest_home: str) -> str:
@@ -327,11 +332,20 @@ class ClaudeAgentProvider(AgentProvider):
user="node",
)
if r.returncode != 0:
# A placeholder — never the real token — keeps this per-bottle
# credential out of the host launch log (#476 review). The operator
# substitutes it from inside the bottle (it rides in the agent's
# HTTPS_PROXY credentials).
manual_header = (
f" --header {shlex.quote(f'{_IDENTITY_HEADER}: {_IDENTITY_TOKEN_PLACEHOLDER}')}"
if token else ""
)
warn(
f"`claude mcp add supervise` failed (exit {r.returncode}): "
f"{(r.stderr or r.stdout or '').strip()}. Inside the bottle, "
f"register manually with: "
f"claude mcp add --scope user --transport http supervise {supervise_url}"
f"register manually (substitute the bottle's identity token) with: "
f"claude mcp add --scope user --transport http "
f"supervise {supervise_url}{manual_header}"
)
def headless_prompt(self, prompt: str) -> list[str]:
-1
View File
@@ -11,7 +11,6 @@ RUN apt-get update \
ca-certificates \
curl \
openssh-client \
podman \
procps \
ripgrep \
&& rm -rf /var/lib/apt/lists/*
-1
View File
@@ -11,7 +11,6 @@ RUN apt-get update \
curl \
fd-find \
openssh-client \
podman \
ripgrep \
&& ln -s /usr/bin/fdfind /usr/local/bin/fd \
&& rm -rf /var/lib/apt/lists/*
+1 -1
View File
@@ -49,7 +49,7 @@ def get_provisioner(
GiteaDeployKeyProvisioner,
)
return GiteaDeployKeyProvisioner(token=token, api_url=api_url)
from .manifest_util import ManifestError
from .manifest.util import ManifestError
raise ManifestError(
f"unknown provisioned_key provider: {provider!r}; "
f"available: gitea"
-34
View File
@@ -1,34 +0,0 @@
"""Lean, framework-free `docker` subprocess primitive.
Deliberately a top-level module with a single stdlib import so it can be
reused from anywhere without cost. It is intentionally *not* placed in
`backend.docker.util`: importing that module runs `backend/__init__.py`,
which eagerly loads all three bottle backends (docker + firecracker +
macos) plus the manifest/egress/git-gate/supervise framework ~76 modules
which would drag the whole backend layer into the deliberately-lean
orchestrator. This primitive stays free of that so both the orchestrator's
docker components and (in time) `backend.docker.util` can share it."""
from __future__ import annotations
import subprocess
def run_docker(
argv: list[str], *, env: dict[str, str] | None = None,
) -> subprocess.CompletedProcess[str]:
"""Run a `docker` command, capturing stdout/stderr as text. Never raises
on a non-zero exit callers inspect `returncode` / `stderr` so they can
stay fail-closed or tolerate idempotent no-ops (e.g. removing an
already-absent container).
`env` sets the child process environment used to hand a secret to a bare
`--env NAME` flag (docker inherits its value from this process) so the
value never lands on argv or in `docker inspect`'s recorded command line."""
return subprocess.run(
argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
check=False, env=env,
)
__all__ = ["run_docker"]
+90
View File
@@ -0,0 +1,90 @@
"""Per-agent egress (PRD 0017).
The egress gateway is a TLS-terminating forward proxy that allow-lists a
bottle's outbound HTTP(S), scans payloads for secret exfil, and injects
per-route upstream credentials the agent never sees.
Layout:
* `service` the `Egress` host-side service (`prepare` the launch plan,
`resolve_token_values`, `agent_env_entries`) + the route-building /
rendering functions.
* `plan` `EgressPlan` / `EgressRoute`, the launch DTOs (in the backend
contract).
The runtime enforcement (the mitmproxy addon) lives in
`bot_bottle.gateway.egress.addon*`. Public names are re-exported lazily via
`__getattr__`, so `from bot_bottle.egress import ` keeps working and importing
`egress.plan` (the contract's dependency) stays light.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from .plan import EgressPlan, EgressRoute
from .service import (
CLAUDE_HOST_CREDENTIAL_TOKEN_REF,
CODEX_HOST_CREDENTIAL_TOKEN_REF,
EGRESS_HOSTNAME,
EGRESS_ROUTES_FILENAME,
EGRESS_ROUTES_IN_CONTAINER,
Egress,
egress_agent_env_entries,
egress_gateway_env_entries,
egress_manifest_routes,
egress_render_routes,
egress_resolve_token_values,
egress_routes_for_bottle,
egress_token_env_map,
)
_LAZY: dict[str, str] = {
"EgressPlan": ".plan",
"EgressRoute": ".plan",
"Egress": ".service",
"CLAUDE_HOST_CREDENTIAL_TOKEN_REF": ".service",
"CODEX_HOST_CREDENTIAL_TOKEN_REF": ".service",
"EGRESS_HOSTNAME": ".service",
"EGRESS_ROUTES_FILENAME": ".service",
"EGRESS_ROUTES_IN_CONTAINER": ".service",
"egress_agent_env_entries": ".service",
"egress_gateway_env_entries": ".service",
"egress_manifest_routes": ".service",
"egress_render_routes": ".service",
"egress_resolve_token_values": ".service",
"egress_routes_for_bottle": ".service",
"egress_token_env_map": ".service",
}
def __getattr__(name: str) -> Any:
src = _LAZY.get(name)
if src is None:
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
from importlib import import_module
value = getattr(import_module(src, __name__), name)
globals()[name] = value
return value
__all__ = [
"CLAUDE_HOST_CREDENTIAL_TOKEN_REF",
"CODEX_HOST_CREDENTIAL_TOKEN_REF",
"EGRESS_HOSTNAME",
"EGRESS_ROUTES_FILENAME",
"EGRESS_ROUTES_IN_CONTAINER",
"Egress",
"EgressPlan",
"EgressRoute",
"egress_manifest_routes",
"egress_render_routes",
"egress_resolve_token_values",
"egress_routes_for_bottle",
"egress_agent_env_entries",
"egress_gateway_env_entries",
"egress_token_env_map",
]
+48
View File
@@ -0,0 +1,48 @@
"""Egress launch DTOs (PRD 0017).
`EgressRoute` (the host-side extension of the addon's wire `Route`) and
`EgressPlan` (the launch plan the backend contract references). Pure value
types the route-building / rendering logic and the `Egress` service live in
`egress.service`.
"""
from __future__ import annotations
from dataclasses import dataclass
from pathlib import Path
from ..gateway.egress.addon_core import Route
@dataclass(frozen=True)
class EgressRoute(Route):
"""Host-side extension of the addon's `Route`.
Inherits `host`, `matches`, `auth_scheme`, and `token_env`
from `egress_addon_core.Route` those are the fields that cross the
YAML wire into the gateway. The fields below are host-only and
are never serialised to the addon.
`token_ref` is the host env var the CLI reads at launch and forwards
into the container's environ under `token_env`.
`roles` carries the manifest route's role tuple (reserved for
future use; always empty today)."""
token_ref: str = ""
roles: tuple[str, ...] = ()
@dataclass(frozen=True)
class EgressPlan:
slug: str
routes_path: Path
routes: tuple[EgressRoute, ...]
token_env_map: dict[str, str]
internal_network: str = ""
egress_network: str = ""
mitmproxy_ca_host_path: Path = Path()
mitmproxy_ca_cert_only_host_path: Path = Path()
log: int = 0
canary: str = ""
canary_env: str = ""
@@ -1,33 +1,33 @@
"""Per-bottle egress proxy (PRD 0017, PRD 0053).
"""The `Egress` host-side service (PRD 0017).
This module defines the abstract proxy (`Egress`), its plan
dataclass (`EgressPlan`), and the resolved per-route shape
(`EgressRoute`). The gateway's start/stop lifecycle is backend-
specific and lives on concrete subclasses (see
`bot_bottle/backend/docker/egress.py`).
`Egress` builds a bottle's egress plan at launch: resolve the manifest's egress
routes, render the gateway's `routes.yaml`, assign per-route token slots, and
plant the exfil canary. The service also resolves the launch-time token values
and the agent/gateway env entries the backend injects. The runtime enforcement
(the mitmproxy addon) lives in `bot_bottle.gateway.egress.addon*`.
"""
from __future__ import annotations
import dataclasses
import secrets
from abc import ABC
from dataclasses import dataclass
from pathlib import Path
from typing import TYPE_CHECKING
from .egress_addon_core import (
from ..gateway.egress.addon_core import (
ON_MATCH_REDACT,
HeaderMatch as CoreHeaderMatch,
MatchEntry as CoreMatchEntry,
PathMatch as CorePathMatch,
Route,
)
from .errors import MissingEnvVarError
from .log import die
from ..errors import MissingEnvVarError
from ..log import die
from .plan import EgressPlan, EgressRoute
if TYPE_CHECKING:
from .manifest import ManifestBottle
from ..manifest import ManifestBottle
CODEX_HOST_CREDENTIAL_TOKEN_REF = "BOT_BOTTLE_CODEX_HOST_ACCESS_TOKEN"
CLAUDE_HOST_CREDENTIAL_TOKEN_REF = "BOT_BOTTLE_CLAUDE_HOST_ACCESS_TOKEN"
@@ -82,39 +82,6 @@ def egress_agent_env_entries(plan: "EgressPlan") -> tuple[str, ...]:
return ()
@dataclass(frozen=True)
class EgressRoute(Route):
"""Host-side extension of the addon's `Route`.
Inherits `host`, `matches`, `auth_scheme`, and `token_env`
from `egress_addon_core.Route` those are the fields that cross the
YAML wire into the gateway. The fields below are host-only and
are never serialised to the addon.
`token_ref` is the host env var the CLI reads at launch and forwards
into the container's environ under `token_env`.
`roles` carries the manifest route's role tuple (reserved for
future use; always empty today)."""
token_ref: str = ""
roles: tuple[str, ...] = ()
@dataclass(frozen=True)
class EgressPlan:
slug: str
routes_path: Path
routes: tuple[EgressRoute, ...]
token_env_map: dict[str, str]
internal_network: str = ""
egress_network: str = ""
mitmproxy_ca_host_path: Path = Path()
mitmproxy_ca_cert_only_host_path: Path = Path()
log: int = 0
canary: str = ""
canary_env: str = ""
def egress_manifest_routes(
bottle: ManifestBottle,
@@ -147,6 +114,7 @@ def egress_manifest_routes(
inbound_detectors=r.InboundDetectors,
outbound_on_match=r.OutboundOnMatch,
preserve_auth=r.PreserveAuth,
inspect=r.Inspect,
))
return tuple(out)
@@ -226,9 +194,13 @@ def _yaml_str_escape(s: str) -> str:
def _route_to_yaml_fields(r: Route) -> dict[str, object]:
fields: dict[str, object] = {"host": r.host}
if not r.inspect:
fields["inspect"] = False
return fields
inspect: dict[str, object] = {}
if r.auth_scheme and r.token_env:
fields["auth_scheme"] = r.auth_scheme
fields["token_env"] = r.token_env
inspect["auth_scheme"] = r.auth_scheme
inspect["token_env"] = r.token_env
if r.matches:
matches_data: list[dict[str, object]] = []
for entry in r.matches:
@@ -252,28 +224,30 @@ def _route_to_yaml_fields(r: Route) -> dict[str, object]:
headers_data.append(hd)
entry_data["headers"] = headers_data
matches_data.append(entry_data)
fields["matches"] = matches_data
inspect["matches"] = matches_data
if r.git_fetch:
fields["git"] = {"fetch": True}
inspect["git"] = {"fetch": True}
if r.preserve_auth:
inspect["preserve_auth"] = True
if (
r.outbound_detectors is not None
or r.inbound_detectors is not None
or r.outbound_on_match
):
dlp: dict[str, object] = {}
if r.outbound_detectors is not None:
dlp["outbound_detectors"] = (
inspect["outbound_detectors"] = (
False if not r.outbound_detectors
else list(r.outbound_detectors)
)
if r.inbound_detectors is not None:
dlp["inbound_detectors"] = (
inspect["inbound_detectors"] = (
False if not r.inbound_detectors
else list(r.inbound_detectors)
)
if r.outbound_on_match:
dlp["outbound_on_match"] = r.outbound_on_match
fields["dlp"] = dlp
inspect["outbound_on_match"] = r.outbound_on_match
if inspect:
fields["inspect"] = inspect
return fields
@@ -281,30 +255,30 @@ def _render_match_entry(entry: dict[str, object]) -> list[str]:
lines: list[str] = []
first_key = True
if "paths" in entry:
lines.append(" - paths:")
lines.append(" - paths:")
first_key = False
for pd in entry["paths"]: # type: ignore[union-attr]
pd_dict: dict[str, str] = pd # type: ignore[assignment]
if "type" in pd_dict:
lines.append(f' - type: "{_yaml_str_escape(pd_dict["type"])}"')
lines.append(f' value: "{_yaml_str_escape(pd_dict["value"])}"')
lines.append(f' - type: "{_yaml_str_escape(pd_dict["type"])}"')
lines.append(f' value: "{_yaml_str_escape(pd_dict["value"])}"')
else:
lines.append(f' - value: "{_yaml_str_escape(pd_dict["value"])}"')
lines.append(f' - value: "{_yaml_str_escape(pd_dict["value"])}"')
if "methods" in entry:
methods_str = ", ".join(f'"{_yaml_str_escape(m)}"' for m in entry["methods"]) # type: ignore[union-attr]
prefix = " - " if first_key else " "
prefix = " - " if first_key else " "
lines.append(f'{prefix}methods: [{methods_str}]')
first_key = False
if "headers" in entry:
prefix = " - " if first_key else " "
prefix = " - " if first_key else " "
lines.append(f"{prefix}headers:")
first_key = False
for hd in entry["headers"]: # type: ignore[union-attr]
hd_dict: dict[str, str] = hd # type: ignore[assignment]
lines.append(f' - name: "{_yaml_str_escape(hd_dict["name"])}"')
lines.append(f' value: "{_yaml_str_escape(hd_dict["value"])}"')
lines.append(f' - name: "{_yaml_str_escape(hd_dict["name"])}"')
lines.append(f' value: "{_yaml_str_escape(hd_dict["value"])}"')
if first_key:
lines.append(" - {}")
lines.append(" - {}")
return lines
@@ -323,22 +297,30 @@ def egress_render_routes(
for r in routes:
f = _route_to_yaml_fields(r)
lines.append(f' - host: "{_yaml_str_escape(str(f["host"]))}"')
if "auth_scheme" in f:
lines.append(f' auth_scheme: "{_yaml_str_escape(str(f["auth_scheme"]))}"')
lines.append(f' token_env: "{_yaml_str_escape(str(f["token_env"]))}"')
if "matches" in f:
lines.append(" matches:")
for entry in f["matches"]: # type: ignore[union-attr]
if f.get("inspect") is False:
lines.append(" inspect: false")
continue
inspect: dict[str, object] = f.get("inspect", {}) # type: ignore[assignment]
if not inspect:
continue
lines.append(" inspect:")
if "auth_scheme" in inspect:
lines.append(f' auth_scheme: "{_yaml_str_escape(str(inspect["auth_scheme"]))}"')
lines.append(f' token_env: "{_yaml_str_escape(str(inspect["token_env"]))}"')
if "matches" in inspect:
lines.append(" matches:")
for entry in inspect["matches"]: # type: ignore[union-attr]
lines.extend(_render_match_entry(entry)) # type: ignore[arg-type]
if "git" in f:
git_dict: dict[str, object] = f["git"] # type: ignore
lines.append(" git:")
if "git" in inspect:
git_dict: dict[str, object] = inspect["git"] # type: ignore
lines.append(" git:")
if git_dict.get("fetch") is True:
lines.append(" fetch: true")
if "dlp" in f:
dlp_dict: dict[str, object] = f["dlp"] # type: ignore
lines.append(" dlp:")
for dk, dv in dlp_dict.items():
lines.append(" fetch: true")
if inspect.get("preserve_auth") is True:
lines.append(" preserve_auth: true")
for dk in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
if dk in inspect:
dv = inspect[dk]
if dv is False:
lines.append(f" {dk}: false")
elif isinstance(dv, list):
@@ -374,7 +356,11 @@ def egress_resolve_token_values(
return out
class Egress(ABC):
class Egress:
"""The host-side egress service. The backend drives `prepare` at launch,
then `resolve_token_values` to resolve the routes' upstream credentials and
`agent_env_entries` for the agent's egress env. Stateless."""
def prepare(
self,
bottle: ManifestBottle,
@@ -401,20 +387,13 @@ class Egress(ABC):
canary_env=_random_canary_env(),
)
__all__ = [
"CLAUDE_HOST_CREDENTIAL_TOKEN_REF",
"CODEX_HOST_CREDENTIAL_TOKEN_REF",
"EGRESS_HOSTNAME",
"EGRESS_ROUTES_FILENAME",
"EGRESS_ROUTES_IN_CONTAINER",
"Egress",
"EgressPlan",
"EgressRoute",
"egress_manifest_routes",
"egress_render_routes",
"egress_resolve_token_values",
"egress_routes_for_bottle",
"egress_agent_env_entries",
"egress_gateway_env_entries",
"egress_token_env_map",
]
def resolve_token_values(
self, token_env_map: dict[str, str], host_env: dict[str, str],
) -> dict[str, str]:
"""Resolve each route's upstream credential from the host env at launch.
Raises `MissingEnvVarError` for an unset/empty referenced host var."""
return egress_resolve_token_values(token_env_map, host_env)
def agent_env_entries(self, plan: EgressPlan) -> tuple[str, ...]:
"""The agent-visible egress env entries (the exfil canary)."""
return egress_agent_env_entries(plan)
+162
View File
@@ -0,0 +1,162 @@
"""The consolidated per-host gateway (PRD 0070).
The core consolidation win: **one** persistent gateway per host, shared by
every bottle, instead of a gateway per bottle. It's safe to share
because the attribution invariant (source IP + identity token, see
`registry`) lets the gateway attribute each request to the right bottle
so per-bottle policy lives in one long-lived process keyed on who's calling.
`Gateway` is the backend-neutral service contract: bind the single instance to
its orchestrator and bring it up (`connect_to_orchestrator`), report its
agent-facing address + CA, vend the provisioning transport, tear it down. The
docker implementation (`DockerGateway`) lives in `backend/docker/gateway.py`;
the macOS + firecracker gateways slot in beside it.
The defining behaviour is **idempotent singleton**: `connect_to_orchestrator`
starts the instance if absent and is a no-op if it's already up on the same
binding, so N bottle launches never spawn N gateways.
"""
from __future__ import annotations
import abc
import os
from pathlib import Path
from typing import Protocol
from ..paths import host_gateway_ca_dir
# The gateway's mitmproxy writes its CA a beat after the container starts, so
# reads poll for it rather than assuming it's there on a fresh launch.
CA_POLL_SECONDS = 0.5
DEFAULT_CA_TIMEOUT_SECONDS = 30.0
GATEWAY_NAME = "bot-bottle-orch-gateway"
GATEWAY_LABEL = "bot-bottle-orch-gateway=1"
# The single user-defined network the gateway and every agent bottle share.
# Agents attach here with a pinned IP and reach the gateway's egress /
# git-http / supervise ports by its address — no host port publishing, and
# the source IP the gateway attributes by is the address on this network.
GATEWAY_NETWORK = "bot-bottle-gateway"
# mitmproxy's CA dir in the bundle. The host's gateway-CA dir (see
# `host_gateway_ca_dir`) is bind-mounted here so the gateway's self-generated
# CA stays STABLE across container recreation — every agent installs this one
# CA to trust the shared gateway's TLS interception, so it must not rotate when
# the gateway restarts. A host bind-mount rather than a named volume: a named
# volume is silently wiped by `docker volume prune`, minting a fresh CA that
# breaks every running bottle (issue #450).
MITMPROXY_HOME = "/home/mitmproxy/.mitmproxy"
GATEWAY_CA_CERT = f"{MITMPROXY_HOME}/mitmproxy-ca-cert.pem"
# The CA material mitmproxy writes into its confdir. mitmproxy reuses these on
# startup when present and generates them only on first run, so persisting them
# is what makes the CA stable; deleting them (see `rotate_gateway_ca`) forces a
# fresh CA on the next start. `mitmproxy-ca.pem` (cert + private key) is the
# signing identity; the rest are derived encodings agents/clients consume.
GATEWAY_CA_GLOB = "mitmproxy-ca*"
# The gateway data-plane image + its Dockerfile. Kept as a local constant
# rather than imported from the backend layer, which would drag
# the whole backend layer into the lean orchestrator (see #359); unify when
# that lands. Env override matches the backend's BOT_BOTTLE_GATEWAY_IMAGE.
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
GATEWAY_DOCKERFILE = "Dockerfile.gateway"
REPO_ROOT = Path(__file__).resolve().parents[2]
def rotate_gateway_ca(ca_dir: Path | None = None) -> list[Path]:
"""Delete the persisted mitmproxy CA so the next gateway start mints a
fresh one the explicit, deliberate CA-rollover path (issue #450).
Persistence keeps the CA stable across restarts precisely because mitmproxy
reuses the on-disk CA; rotation is therefore just removing that material.
Returns the files removed (empty when there was no CA yet); idempotent.
This only clears the on-disk CA. It does NOT stop the running gateway (whose
mitmproxy still holds the old CA in memory) or re-provision agents the
caller recreates the gateway to mint the new CA and re-attaches bottles.
`rotate-ca` on the orchestrator CLI wires those steps together."""
ca_dir = ca_dir if ca_dir is not None else host_gateway_ca_dir()
removed: list[Path] = []
for path in sorted(ca_dir.glob(GATEWAY_CA_GLOB)):
path.unlink()
removed.append(path)
return removed
class GatewayError(Exception):
"""The shared gateway failed to build/start/stop (non-zero `docker` exit)."""
class GatewayProvisionError(RuntimeError):
"""A git-gate provisioning step against the running gateway failed."""
class GatewayTransport(Protocol):
"""How the launcher stages files + runs commands in the running gateway.
Backend-neutral so the same git-gate provisioning logic serves the docker
gateway container (exec/cp over the docker socket), the Apple gateway
container, and the firecracker gateway VM (over SSH)."""
def exec(self, argv: list[str]) -> None:
"""Run `argv` in the gateway, raising `GatewayProvisionError` on failure."""
def cp_into(self, src: str, dest: str) -> None:
"""Copy host file `src` to `dest` in the gateway, raising on failure."""
class Gateway(abc.ABC):
"""Provision + interact with the per-host gateway (data plane).
One concrete impl per backend (`backend/*/gateway.py`); the host composes it
with the Orchestrator service. The gateway **never holds the signing key**
it receives a pre-minted `gateway` token from the orchestrator and only
presents it. Backend-neutral."""
name: str
def ensure_built(self) -> None:
"""Ensure the gateway's image / rootfs exists, building it if needed.
Default: nothing to build (e.g. a stub or a pre-pulled image). Call
before `connect_to_orchestrator`."""
return
@abc.abstractmethod
def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
"""Bind the gateway to this orchestrator and bring it up: store the URL +
the pre-minted `gateway` token as instance state, then (re)start the
gateway unit carrying the mitmproxy CA + that token, resolving policy
against `orchestrator_url`. Idempotent a healthy, current gateway on
the same binding is left alone; a changed binding reconciles it."""
@abc.abstractmethod
def is_running(self) -> bool:
"""True iff the gateway instance is currently up."""
@abc.abstractmethod
def stop(self) -> None:
"""Remove the gateway. Idempotent — absent is success."""
@abc.abstractmethod
def address(self) -> str:
"""The agent-facing address agents dial for egress / git-http / supervise
(today's `gateway_ip`)."""
@abc.abstractmethod
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The mitmproxy CA (PEM) agents install to trust the gateway's TLS
interception. Polls mitmproxy writes it a beat after start."""
@abc.abstractmethod
def provisioning_transport(self) -> "GatewayTransport":
"""The cp/exec transport git-gate provisioning uses to place per-bottle
repos + deploy keys into the running gateway."""
__all__ = [
"Gateway", "GatewayError", "GatewayProvisionError", "GatewayTransport",
"rotate_gateway_ca",
"GATEWAY_NAME", "GATEWAY_LABEL", "GATEWAY_IMAGE", "GATEWAY_NETWORK",
"GATEWAY_CA_CERT", "GATEWAY_CA_GLOB",
]
@@ -5,18 +5,11 @@ the configured daemons (egress, git-gate, supervise),
forwards SIGTERM/SIGINT to each child, and propagates per-daemon
stdout+stderr to the container log with a `[name] ` prefix.
Failure policy (interim): when a child dies unexpectedly, the
supervisor logs the death and leaves the surviving children
running. The gateway stays up; whatever the dead daemon served
will start failing, surfacing in the agent's own error path.
The supervisor itself exits only when (a) the operator sends
SIGTERM/SIGINT, or (b) every child has died.
Failure policy (eventual): on unexpected death, the supervisor
restarts the daemon and emits a notification to the supervise
daemon so the operator sees the event. That lands in a later
PR; the interim policy is "don't take the gateway down for one
sick daemon."
Failure policy: when a child dies unexpectedly, the supervisor
restarts it automatically and logs the restart. The gateway stays
up; a temporary loss of one daemon (e.g. egress OOM-killed) is
recovered without manual container recreation. The supervisor
itself exits only when the operator sends SIGTERM/SIGINT.
Daemon subset is env-driven via `BOT_BOTTLE_GATEWAY_DAEMONS=egress`
for callers that don't use git-gate or supervise. Default: all
@@ -61,37 +54,40 @@ class _DaemonSpec:
_EGRESS_ONLY_ENV_PREFIXES: tuple[str, ...] = ("EGRESS_TOKEN_",)
_READY_GATED_DAEMONS: tuple[str, ...] = ("git-gate", "git-http")
# Daemons that must be requested explicitly via BOT_BOTTLE_GATEWAY_DAEMONS
# and are NOT started in the default (env-var-unset) case. The orchestrator
# only runs in the combined infra container, never in a standalone gateway.
_OPT_IN_DAEMONS: frozenset[str] = frozenset({"orchestrator"})
# The control-plane signing key is the orchestrator's alone (it verifies
# tokens), and the orchestrator runs in a separate container/VM — the gateway
# only ever holds the pre-minted `gateway` JWT its daemons present. Strip the
# key from every daemon's env as defense-in-depth, so a compromised data-plane
# daemon can't read it and mint a `cli` token even if it somehow leaked into the
# gateway container (issue #469 review). Value matches
# paths.ORCHESTRATOR_TOKEN_ENV; hardcoded here so this supervisor stays
# import-light.
_SIGNING_KEY_ENV = "BOT_BOTTLE_ORCHESTRATOR_TOKEN"
def _env_for_daemon(name: str, base_env: dict[str, str]) -> dict[str, str]:
"""Egress sees the full bundle env. Everyone else gets a copy
with `EGRESS_TOKEN_*` (and any other future egress-only
credential slots) stripped. Returns a fresh dict callers
can mutate without affecting `base_env`."""
if name == "egress":
return dict(base_env)
return {
k: v for k, v in base_env.items()
if not any(k.startswith(p) for p in _EGRESS_ONLY_ENV_PREFIXES)
}
"""Per-daemon env, scoped to what each process legitimately needs.
Returns a fresh dict callers can mutate without affecting `base_env`.
* `EGRESS_TOKEN_*` upstream-auth slots go to egress only.
* the control-plane signing key is stripped from every daemon the
gateway never holds it (it's the orchestrator's); the data-plane daemons
present the pre-minted `gateway` JWT instead."""
env = dict(base_env)
if name != "egress":
env = {
k: v for k, v in env.items()
if not any(k.startswith(p) for p in _EGRESS_ONLY_ENV_PREFIXES)
}
env.pop(_SIGNING_KEY_ENV, None)
return env
# The orchestrator is listed first so it starts before the gateway daemons,
# giving the control plane a head start to accept /resolve calls. The gateway
# daemons tolerate early /resolve failures and retry per-request.
_DAEMONS: tuple[_DaemonSpec, ...] = (
_DaemonSpec("orchestrator", (
"python3", "-m", "bot_bottle.orchestrator",
"--host", "0.0.0.0", "--port", "8099", "--broker", "stub",
)),
_DaemonSpec("egress", ("/bin/sh", "/app/egress-entrypoint.sh")),
_DaemonSpec("git-gate", ("/bin/sh", "/git-gate-entrypoint.sh")),
_DaemonSpec("git-http", ("python3", "-m", "bot_bottle.git_http_backend")),
_DaemonSpec("supervise", ("python3", "-m", "bot_bottle.supervise_server")),
_DaemonSpec("git-http", ("python3", "-m", "bot_bottle.gateway.git_gate.http_backend")),
_DaemonSpec("supervise", ("python3", "-m", "bot_bottle.gateway.supervisor.server")),
)
@@ -117,10 +113,8 @@ def _selected_daemons(
) -> tuple[_DaemonSpec, ...]:
"""Filter the daemon set by the BOT_BOTTLE_GATEWAY_DAEMONS env var.
When the var is unset/empty, return all non-opt-in daemons (the
standard gateway subset). Opt-in daemons (e.g. `orchestrator`) only
run when explicitly named they never start in a plain gateway
container that doesn't set the env var. Unknown names are ignored.
When the var is unset/empty, return all daemons (the standard gateway
subset). Unknown names are ignored.
`all_daemons` defaults to `_DAEMONS` resolved at call time (not at
definition time), so tests can pass a custom list."""
@@ -128,7 +122,7 @@ def _selected_daemons(
all_daemons = _DAEMONS
raw = env.get("BOT_BOTTLE_GATEWAY_DAEMONS", "").strip()
if not raw:
return tuple(d for d in all_daemons if d.name not in _OPT_IN_DAEMONS)
return tuple(all_daemons)
wanted = {n.strip() for n in raw.split(",") if n.strip()}
return tuple(d for d in all_daemons if d.name in wanted)
@@ -163,7 +157,7 @@ def _spawn(spec: _DaemonSpec) -> subprocess.Popen[bytes]:
return proc
class _Supervisor:
class _DaemonManager:
"""Holds the running children + shutdown state. Pulled out so
the test suite can drive it with fake commands."""
@@ -227,9 +221,10 @@ class _Supervisor:
"""One iteration of the watch loop. Returns True when every
child has exited and the supervisor can return.
A child dying unexpectedly is logged but does NOT initiate
shutdown see the module docstring's failure-policy
section. Shutdown is signal-driven only."""
A child dying unexpectedly is logged and restarted but does
NOT initiate shutdown see the module docstring's
failure-policy section. Shutdown is signal-driven only."""
restarted_children = bool(self._restart_requested)
self._drain_restart_requests()
for spec, p in self.procs:
@@ -238,14 +233,18 @@ class _Supervisor:
continue
self._logged_dead.add(spec.name)
if self.shutdown_at is None:
_log(
f"{spec.name} exited with code {rc}; leaving "
f"surviving daemons running (operator-visible "
f"via agent-side failure)"
)
_log(f"{spec.name} exited with code {rc}; scheduling restart")
self._restart_requested.add(spec.name)
else:
_log(f"{spec.name} exited with code {rc}")
# Restart deaths discovered above before checking whether all
# processes are done. Deferring this until the next tick would make a
# single-daemon supervisor return True and exit with the restart still
# queued.
restarted_children |= bool(self._restart_requested)
self._drain_restart_requests()
if self.shutdown_at is not None:
elapsed = time.monotonic() - self.shutdown_at
if elapsed > _GRACE_SECONDS:
@@ -259,7 +258,10 @@ class _Supervisor:
)
self._sigkill_all()
done = all(p.poll() is not None for _, p in self.procs)
done = (
not restarted_children
and all(p.poll() is not None for _, p in self.procs)
)
if done:
for _, p in self.procs:
if p.stdout is not None:
@@ -369,7 +371,7 @@ def main(argv: Sequence[str] | None = None) -> int:
_log("no daemons selected; nothing to do")
return 0
sup = _Supervisor(specs)
sup = _DaemonManager(specs)
sup.start_all()
signal.signal(signal.SIGTERM, lambda *_: sup.request_shutdown("SIGTERM")) # type: ignore
+9
View File
@@ -0,0 +1,9 @@
"""Gateway-side (data-plane) egress service: the mitmproxy addon and its
pure decision core, DLP detectors, and route/DLP config parsing.
These are the long-running data-plane pieces (loaded by mitmdump inside the
`bot-bottle-gateway` image), distinct from the host-side `bot_bottle.egress`
service that renders routes and prepares env. Import the concrete modules
directly (`from bot_bottle.gateway.egress.addon_core import ...`) this
package deliberately does no eager work so leaf imports stay cheap.
"""
@@ -16,8 +16,8 @@ import typing
from mitmproxy import http # type: ignore[import-not-found] # pylint: disable=import-error
from bot_bottle.constants import IDENTITY_HEADER
from bot_bottle.dlp_detectors import redact_tokens, strip_crlf
from bot_bottle.egress_addon_core import (
from bot_bottle.gateway.egress.dlp_detectors import redact_tokens, strip_crlf
from bot_bottle.gateway.egress.addon_core import (
LOG_BLOCKS,
LOG_FULL,
DEFAULT_OUTBOUND_ON_MATCH,
@@ -40,8 +40,13 @@ from bot_bottle.egress_addon_core import (
scan_inbound,
scan_outbound,
)
from bot_bottle import supervise as _sv
from bot_bottle.policy_resolver import PolicyResolver
from bot_bottle.gateway.policy_resolver import PolicyResolveError, PolicyResolver
from bot_bottle.supervisor.types import (
STATUS_APPROVED,
STATUS_MODIFIED,
STATUSES,
TOOL_EGRESS_TOKEN_ALLOW,
)
INTROSPECT_HOST = "_egress.local"
@@ -78,6 +83,15 @@ def _token_from_proxy_auth(header: str) -> str:
# Seconds the egress proxy holds a token-blocked request open waiting for the
# operator's supervisor decision (PRD 0062), overridable via env.
DEFAULT_TOKEN_ALLOW_TIMEOUT_SECONDS = 300.0
# Maximum bytes of a response body passed to the DLP inbound scan. mitmproxy
# buffers the full response before the hook fires; capping at scan time limits
# the additional memory amplification from decoded text and regex match strings.
# A cap is a security trade-off (content above the threshold is not scanned),
# but without it a single large download OOM-kills the shared egress process
# (issue #455). Override with EGRESS_INBOUND_SCAN_LIMIT_BYTES; set to 0 to
# disable the cap.
DEFAULT_INBOUND_SCAN_LIMIT_BYTES = 1 * 1024 * 1024 # 1 MiB
# Filesystem poll cadence while awaiting the operator's response.
TOKEN_ALLOW_POLL_INTERVAL_SECONDS = 0.5
@@ -97,10 +111,12 @@ class EgressAddon:
# comes from the orchestrator's /resolve (PRD 0070); there is no static
# per-bottle routes file, SIGHUP reload, or single-tenant fallback.
_resolver: "PolicyResolver"
# Class default so __new__-built addons have it (real runs get a fresh
# per-instance dict in __init__; only http_connect mutates it, which the
# request-flow tests don't exercise).
# Class defaults so __new__-built addons have them (real runs get fresh
# per-instance collections in __init__; only http_connect mutates them,
# which request-flow tests don't exercise unless they call http_connect).
_conn_tokens: "dict[str, str]" = {}
_passthrough_conns: "set[str]" = set()
_inbound_scan_limit: int = DEFAULT_INBOUND_SCAN_LIMIT_BYTES
def __init__(self) -> None:
# Resolver-only: the gateway is always multi-tenant, resolving each
@@ -125,7 +141,12 @@ class EgressAddon:
# `Proxy-Authorization` (HTTPS tunnels don't repeat it on the bumped
# inner requests). Keyed by client_conn.id; cleared on disconnect.
self._conn_tokens: dict[str, str] = {}
# Connections whose route carries `inspect: false` — mitmproxy tunnels
# these without TLS interception so the client sees the server's real
# cert. Keyed by client_conn.id; cleared on disconnect.
self._passthrough_conns: set[str] = set()
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
self._inbound_scan_limit = _inbound_scan_limit_from_env(os.environ)
@staticmethod
def _supervise_available(slug: str) -> bool:
@@ -298,32 +319,82 @@ class EgressAddon:
flow.request.headers.pop("Proxy-Authorization", None)
flow.request.headers.pop(IDENTITY_HEADER, None)
conn = flow.client_conn
if not token and conn is not None:
token = self._conn_tokens.get(getattr(conn, "id", ""), "")
conn_id = getattr(conn, "id", "") if conn is not None else ""
if not token and conn_id:
token = self._conn_tokens.get(conn_id, "")
# Remember the token per connection so a later token-block on this flow
# can attribute its supervise proposal by (source_ip, identity_token)
# over RPC — plain-HTTP requests carry it here, HTTPS tunnels captured
# it at CONNECT (http_connect); both land in `_conn_tokens`.
if token and conn_id:
self._conn_tokens[conn_id] = token
return token
def http_connect(self, flow: http.HTTPFlow) -> None:
"""Capture the identity token from an HTTPS tunnel's CONNECT (the inner
bumped requests won't carry `Proxy-Authorization`), keyed by client
connection, and strip it so it never reaches upstream."""
connection, and strip it so it never reaches upstream.
For `inspect: false` routes, also resolve the policy here to make the
allowlist decision before the TLS handshake: the tunnel is either
blocked immediately or marked for passthrough in `_passthrough_conns`
so `tls_clienthello` skips interception."""
token = _token_from_proxy_auth(
flow.request.headers.get("Proxy-Authorization", ""))
flow.request.headers.pop("Proxy-Authorization", None)
conn = flow.client_conn
if conn is not None and getattr(conn, "id", ""):
self._conn_tokens[conn.id] = token
conn_id = getattr(conn, "id", "") if conn is not None else ""
if conn_id:
self._conn_tokens[conn_id] = token
# Resolve the policy here for all HTTPS connections and stash it so
# request() reuses it without a second orchestrator round-trip. For
# passthrough hosts we also make the allowlist decision now because
# inner requests never reach request() after the TLS bypass.
client_ip = conn.peername[0] if conn is not None and conn.peername else ""
config, slug, env = resolve_client_context(self._resolver, client_ip, token)
self._stash_flow_ctx(flow, config, slug, env)
host = flow.request.pretty_host
route = match_route(config.routes, host)
if route is not None and not route.inspect:
decision = decide(config.routes, host, "/", env, deny_reason=config.deny_reason)
if decision.action == "block":
flow.response = http.Response.make(
403,
decision.reason.encode("utf-8"),
{"Content-Type": "text/plain; charset=utf-8"},
)
return
if conn_id:
self._passthrough_conns.add(conn_id)
def tls_clienthello(self, client_hello: typing.Any) -> None:
"""Skip TLS interception for `inspect: false` routes so the client sees
the server's real certificate rather than the MITM CA's leaf."""
conn_id = getattr(client_hello.context.client, "id", "")
if conn_id in self._passthrough_conns:
client_hello.ignore_connection = True
def client_disconnected(self, client: typing.Any) -> None:
"""Drop the per-connection token when the client goes away."""
self._conn_tokens.pop(getattr(client, "id", ""), None)
"""Drop the per-connection token and passthrough flag when the client
goes away."""
conn_id = getattr(client, "id", "")
self._conn_tokens.pop(conn_id, None)
self._passthrough_conns.discard(conn_id)
async def request(self, flow: http.HTTPFlow) -> None:
request_path, _, query = flow.request.path.partition("?")
config, slug, env = self._resolve_flow(flow)
# Stash for the response / websocket hooks so their DLP scans reuse this
# bottle's resolved policy (one /resolve per flow — see _flow_ctx).
self._stash_flow_ctx(flow, config, slug, env)
# Reuse the context stashed by http_connect for HTTPS flows (one
# orchestrator round-trip per connection). Plain-HTTP flows have no
# prior CONNECT stash, so resolve now and stash for response/websocket.
meta = getattr(flow, "metadata", None)
if isinstance(meta, dict) and _FLOW_CTX_KEY in meta:
config, slug, env = meta[_FLOW_CTX_KEY]
self._request_token(flow) # strip identity headers; token already resolved
else:
config, slug, env = self._resolve_flow(flow)
self._stash_flow_ctx(flow, config, slug, env)
# Introspection ("_egress.local/allowlist") reports the calling bottle's
# own resolved routes — served after resolution so it reflects this
@@ -335,8 +406,10 @@ class EgressAddon:
# DLP outbound scan BEFORE stripping auth — catches tokens the
# agent tried to smuggle in any header, path, query param, or body.
# Hostname is included to catch DNS-tunnelling exfiltration attempts.
# `inspect: false` routes skip scanning entirely (TLS is also not
# intercepted for HTTPS, so this branch only fires for plain HTTP).
route = match_route(config.routes, flow.request.pretty_host)
if route is not None:
if route is not None and route.inspect:
if not await self._handle_outbound_dlp(flow, route, slug, env):
return
# The redact policy may have rewritten the request line; recompute
@@ -531,42 +604,48 @@ class EgressAddon:
redact_tokens(request_path, env=env),
result,
)
proposal = _sv.Proposal.new(
bottle_slug=slug,
tool=_sv.TOOL_EGRESS_TOKEN_ALLOW,
proposed_file=payload,
justification=_TOKEN_ALLOW_JUSTIFICATION,
current_file_hash=_sv.sha256_hex(payload),
)
# Attribute the proposal by (source_ip, identity_token) over the control
# plane — the data plane no longer opens bot-bottle.db (PRD 0070 / #469).
# source_ip + token come from this bottle's connection; `slug` is kept
# only for its own DLP safelist.
conn = flow.client_conn
source_ip = conn.peername[0] if conn is not None and conn.peername else ""
token = self._conn_tokens.get(getattr(conn, "id", ""), "") if conn is not None else ""
try:
_sv.write_proposal(proposal)
except OSError as e:
proposal_id = self._resolver.propose_supervise(
source_ip, token,
tool=TOOL_EGRESS_TOKEN_ALLOW,
proposed_file=payload,
justification=_TOKEN_ALLOW_JUSTIFICATION,
)
except PolicyResolveError as e:
sys.stderr.write(
f"egress: could not queue token-allow proposal: {e}; "
"blocking request\n"
)
proposal_id = None
if proposal_id is None:
self._block(flow, f"egress DLP: {result.reason}", ctx=self._req_ctx(flow))
return False
sys.stderr.write(json.dumps({
"event": "egress_token_supervise",
"reason": f"egress DLP: {result.reason}",
"proposal": proposal.id,
"proposal": proposal_id,
**self._req_ctx(flow),
}) + "\n")
response = await self._await_token_response(proposal.id, slug)
_sv.archive_proposal(slug, proposal.id)
response = await self._await_token_response(proposal_id, source_ip, token)
if response is not None and response.status in (
_sv.STATUS_APPROVED, _sv.STATUS_MODIFIED,
if response is not None and response.get("status") in (
STATUS_APPROVED, STATUS_MODIFIED,
):
self._safe_tokens_for(slug).add(result.matched)
if self._flow_log(flow) >= LOG_BLOCKS:
sys.stderr.write(json.dumps({
"event": "egress_token_allowed",
"reason": f"egress DLP: {result.reason}",
"proposal": proposal.id,
"proposal": proposal_id,
**self._req_ctx(flow),
}) + "\n")
return True
@@ -584,19 +663,23 @@ class EgressAddon:
async def _await_token_response(
self,
proposal_id: str,
slug: str,
) -> "_sv.Response | None":
"""Poll the DB for the operator's response without blocking the
proxy event loop. Returns the Response, or None on timeout."""
source_ip: str,
token: str,
) -> "dict[str, object] | None":
"""Poll the control plane for the operator's decision without blocking
the proxy event loop. Returns the terminal `{status, ...}` payload once
decided, or None on timeout. A transient orchestrator error (or a
`pending`/`unknown` status) is retried until the deadline, then fails
closed the caller blocks the request."""
loop = asyncio.get_running_loop()
deadline = loop.time() + self._token_allow_timeout
while True:
try:
return _sv.read_response(slug, proposal_id)
except (OSError, ValueError, KeyError):
# Not written yet, or a partial/malformed write — retry until
# the deadline, then fail closed.
pass
result = self._resolver.poll_supervise(source_ip, token, proposal_id)
except PolicyResolveError:
result = None
if result is not None and result.get("status") in STATUSES:
return result
if loop.time() >= deadline:
return None
await asyncio.sleep(TOKEN_ALLOW_POLL_INTERVAL_SECONDS)
@@ -606,7 +689,7 @@ class EgressAddon:
bottle's resolved config (`request()` stashed it — see `_flow_ctx`)."""
config, _slug, env = self._flow_ctx(flow)
route = match_route(config.routes, flow.request.pretty_host)
if route is None:
if route is None or not route.inspect:
return
if flow.response is None:
return
@@ -614,6 +697,14 @@ class EgressAddon:
self._log_response(flow, env)
resp_headers = {k.lower(): v for k, v in flow.response.headers.items()}
body = flow.response.get_text(strict=False) or ""
if self._inbound_scan_limit and len(body) > self._inbound_scan_limit:
sys.stderr.write(json.dumps({
"event": "egress_scan_truncated",
"host": flow.request.pretty_host,
"body_bytes": len(body),
"scan_limit_bytes": self._inbound_scan_limit,
}) + "\n")
body = body[:self._inbound_scan_limit]
scan_text = build_inbound_scan_text(resp_headers, body)
if not scan_text:
return
@@ -652,7 +743,7 @@ class EgressAddon:
return
config, slug, env = self._flow_ctx(flow)
route = match_route(config.routes, flow.request.pretty_host)
if route is None:
if route is None or not route.inspect:
return
message = flow.websocket.messages[-1] # type: ignore[union-attr]
content = message.content.decode("utf-8", errors="replace")
@@ -676,6 +767,25 @@ class EgressAddon:
sys.stderr.write(f"egress DLP warn: {result.reason}\n")
def _inbound_scan_limit_from_env(env: "os._Environ[str]") -> int:
"""Read EGRESS_INBOUND_SCAN_LIMIT_BYTES; fall back to the default on an
unset or invalid value. Returns 0 to disable the cap."""
raw = env.get("EGRESS_INBOUND_SCAN_LIMIT_BYTES", "").strip()
if not raw:
return DEFAULT_INBOUND_SCAN_LIMIT_BYTES
try:
value = int(raw)
except ValueError:
value = -1
if value < 0:
sys.stderr.write(
"egress: invalid EGRESS_INBOUND_SCAN_LIMIT_BYTES="
f"{raw!r}; using default {DEFAULT_INBOUND_SCAN_LIMIT_BYTES}\n"
)
return DEFAULT_INBOUND_SCAN_LIMIT_BYTES
return value
def _token_allow_timeout_from_env(env: "os._Environ[str]") -> float:
"""Read EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS; fall back to the default on an
unset or invalid value (a bad value should not wedge egress at boot)."""
@@ -16,11 +16,11 @@ import re
import typing
from dataclasses import dataclass
from .yaml_subset import YamlSubsetError, parse_yaml_subset
from ...yaml_subset import YamlSubsetError, parse_yaml_subset
# DLP detector-config parsing lives in a sibling module. Re-exported below
# so existing `from egress_addon_core import ON_MATCH_*` callers keep working.
from .egress_dlp_config import (
from .dlp_config import (
DEFAULT_OUTBOUND_ON_MATCH,
INBOUND_DETECTOR_NAMES,
ON_MATCH_BLOCK,
@@ -28,7 +28,7 @@ from .egress_dlp_config import (
ON_MATCH_SUPERVISE,
OUTBOUND_DETECTOR_NAMES,
OUTBOUND_ON_MATCH_VALUES,
parse_dlp_block,
parse_inspect_block,
)
@@ -79,6 +79,8 @@ class Route:
# "" means unset → DEFAULT_OUTBOUND_ON_MATCH. See OUTBOUND_ON_MATCH_VALUES.
outbound_on_match: str = ""
preserve_auth: bool = False
# False tunnels HTTPS without TLS interception or HTTP-level controls.
inspect: bool = True
LOG_OFF = 0 # no logging
@@ -259,10 +261,31 @@ def _parse_one(idx: int, raw: object) -> Route:
host: object = raw_dict.get("host")
if not isinstance(host, str) or not host:
raise ValueError(f"{label}: 'host' must be a non-empty string")
legacy_flat = "inspect" not in raw_dict
inspect_raw = raw_dict.get("inspect", {})
if inspect_raw is False:
inspect = False
settings: dict[str, object] = {}
elif isinstance(inspect_raw, dict):
inspect = True
settings = (
{k: v for k, v in raw_dict.items() if k != "host"}
if legacy_flat
else typing.cast(dict[str, object], inspect_raw)
)
legacy_dlp = settings.pop("dlp", None)
if isinstance(legacy_dlp, dict):
settings.update(typing.cast(dict[str, object], legacy_dlp))
elif legacy_dlp is not None:
raise ValueError(
f"{label} ({host}): legacy 'dlp' must be an object"
)
else:
raise ValueError(f"{label} ({host}): 'inspect' must be false or an object")
# matches
matches: tuple[MatchEntry, ...] = ()
matches_raw = raw_dict.get("matches")
matches_raw = settings.get("matches")
if matches_raw is not None:
if not isinstance(matches_raw, list):
raise ValueError(f"{label} ({host}): 'matches' must be a list")
@@ -272,8 +295,8 @@ def _parse_one(idx: int, raw: object) -> Route:
)
# auth (unchanged wire format)
auth_scheme: object = raw_dict.get("auth_scheme", "")
token_env: object = raw_dict.get("token_env", "")
auth_scheme: object = settings.get("auth_scheme", "")
token_env: object = settings.get("token_env", "")
if not isinstance(auth_scheme, str):
raise ValueError(f"{label} ({host}): 'auth_scheme' must be a string")
if not isinstance(token_env, str):
@@ -287,7 +310,7 @@ def _parse_one(idx: int, raw: object) -> Route:
# git-over-HTTPS policy
git_fetch = False
git_raw = raw_dict.get("git")
git_raw = settings.get("git")
if git_raw is not None:
if not isinstance(git_raw, dict):
raise ValueError(f"{label} ({host}): 'git' must be an object")
@@ -305,22 +328,30 @@ def _parse_one(idx: int, raw: object) -> Route:
)
# dlp detectors
outbound_detectors, inbound_detectors, outbound_on_match = parse_dlp_block(
idx, host, raw_dict,
outbound_detectors, inbound_detectors, outbound_on_match = parse_inspect_block(
idx, host, settings,
)
preserve_auth_raw = raw_dict.get("preserve_auth", False)
preserve_auth_raw = settings.get("preserve_auth", False)
if preserve_auth_raw is not True and preserve_auth_raw is not False:
raise ValueError(
f"{label} ({host}): 'preserve_auth' must be a boolean"
)
preserve_auth: bool = preserve_auth_raw
for k in settings:
if k not in (
"matches", "auth_scheme", "token_env", "git", "preserve_auth",
"outbound_detectors", "inbound_detectors", "outbound_on_match",
):
raise ValueError(
f"{label} ({host}): inspect has unknown key {k!r}"
)
for k in raw_dict:
if k not in ("host", "matches", "auth_scheme", "token_env", "dlp", "git", "preserve_auth"):
if not legacy_flat and k not in ("host", "inspect"):
raise ValueError(
f"{label} ({host}): unknown key {k!r}; accepted keys "
f"are 'host', 'matches', 'auth_scheme', 'token_env', 'dlp', 'git', 'preserve_auth'"
f"are 'host' and 'inspect'"
)
return Route(
@@ -333,6 +364,7 @@ def _parse_one(idx: int, raw: object) -> Route:
inbound_detectors=inbound_detectors,
outbound_on_match=outbound_on_match,
preserve_auth=preserve_auth,
inspect=inspect,
)
@@ -369,24 +401,27 @@ def route_to_yaml_dict(r: Route) -> dict[str, object]:
proposal without translation. Fields that are empty/default are
omitted so the agent doesn't copy irrelevant keys."""
d: dict[str, object] = {"host": r.host}
if not r.inspect:
d["inspect"] = False
return d
inspected: dict[str, object] = {}
if r.auth_scheme:
d["auth_scheme"] = r.auth_scheme
d["token_env"] = r.token_env
inspected["auth_scheme"] = r.auth_scheme
inspected["token_env"] = r.token_env
if r.matches:
d["matches"] = [_match_entry_to_dict(m) for m in r.matches]
inspected["matches"] = [_match_entry_to_dict(m) for m in r.matches]
if r.git_fetch:
d["git"] = {"fetch": True}
dlp: dict[str, object] = {}
inspected["git"] = {"fetch": True}
if r.outbound_detectors is not None:
dlp["outbound_detectors"] = list(r.outbound_detectors)
inspected["outbound_detectors"] = list(r.outbound_detectors)
if r.inbound_detectors is not None:
dlp["inbound_detectors"] = list(r.inbound_detectors)
inspected["inbound_detectors"] = list(r.inbound_detectors)
if r.outbound_on_match:
dlp["outbound_on_match"] = r.outbound_on_match
if dlp:
d["dlp"] = dlp
inspected["outbound_on_match"] = r.outbound_on_match
if r.preserve_auth:
d["preserve_auth"] = True
inspected["preserve_auth"] = True
if inspected:
d["inspect"] = inspected
return d
@@ -758,6 +793,8 @@ def scan_outbound(
safe_tokens: typing.AbstractSet[str] | None = None,
crlf_text: str | None = None,
) -> ScanResult | None:
if not route.inspect:
return None
# Lazy import to avoid circular deps and keep dlp_detectors optional
# at import time (the gateway copies it flat alongside this file).
try:
@@ -855,6 +892,8 @@ def scan_inbound(
route: Route,
body: str | bytes,
) -> ScanResult | None:
if not route.inspect:
return None
try:
from dlp_detectors import scan_naive_injection # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
@@ -882,7 +921,7 @@ __all__ = [
"DEFAULT_OUTBOUND_ON_MATCH",
"OUTBOUND_DETECTOR_NAMES",
"INBOUND_DETECTOR_NAMES",
"parse_dlp_block",
"parse_inspect_block",
"Config",
"Decision",
"HeaderMatch",
@@ -1,6 +1,6 @@
"""DLP detector-config parsing for egress routes (PRD 0053, PRD 0062).
"""Inspection and DLP configuration parsing for egress routes.
A route's optional `dlp:` block names which outbound/inbound detectors run
A route's optional `inspect:` object names which outbound/inbound detectors run
and what the proxy does when an outbound detector matches a token
(`outbound_on_match`). This module owns parsing and validating that block,
kept apart from the request-time scan/decision flow in `egress_addon_core`
@@ -26,20 +26,14 @@ OUTBOUND_ON_MATCH_VALUES = (ON_MATCH_BLOCK, ON_MATCH_REDACT, ON_MATCH_SUPERVISE)
DEFAULT_OUTBOUND_ON_MATCH = ON_MATCH_SUPERVISE
def parse_dlp_block(
def parse_inspect_block(
idx: int,
host: str,
raw_dict: dict[str, object],
inspect: dict[str, object],
) -> tuple[tuple[str, ...] | None, tuple[str, ...] | None, str]:
"""Parse the optional `dlp` block on a route, returning
(outbound_detectors, inbound_detectors, outbound_on_match)."""
dlp_raw = raw_dict.get("dlp")
if dlp_raw is None:
return None, None, ""
"""Parse DLP settings from an inspected route."""
label = f"route[{idx}] ({host})"
if not isinstance(dlp_raw, dict):
raise ValueError(f"{label}: 'dlp' must be an object")
dlp = typing.cast(dict[str, object], dlp_raw)
dlp = inspect
def _parse_detector_field(
field: str,
@@ -52,18 +46,18 @@ def parse_dlp_block(
return ()
if not isinstance(val, list):
raise ValueError(
f"{label}: dlp.{field} must be false, a list, or omitted"
f"{label}: inspect.{field} must be false, a list, or omitted"
)
items = typing.cast(list[object], val)
names: list[str] = []
for j, item in enumerate(items):
if not isinstance(item, str):
raise ValueError(
f"{label}: dlp.{field}[{j}] must be a string"
f"{label}: inspect.{field}[{j}] must be a string"
)
if item not in valid_names:
raise ValueError(
f"{label}: dlp.{field}[{j}] {item!r} is not a valid "
f"{label}: inspect.{field}[{j}] {item!r} is not a valid "
f"detector name; valid names: {', '.join(sorted(valid_names))}"
)
names.append(item)
@@ -77,16 +71,9 @@ def parse_dlp_block(
if on_match_raw is not None:
if not isinstance(on_match_raw, str) or on_match_raw not in OUTBOUND_ON_MATCH_VALUES:
raise ValueError(
f"{label}: dlp.outbound_on_match must be one of "
f"{label}: inspect.outbound_on_match must be one of "
f"{', '.join(OUTBOUND_ON_MATCH_VALUES)} (got {on_match_raw!r})"
)
on_match = on_match_raw
for k in dlp:
if k not in ("outbound_detectors", "inbound_detectors", "outbound_on_match"):
raise ValueError(
f"{label}: dlp has unknown key {k!r}; accepted keys "
f"are 'outbound_detectors', 'inbound_detectors', "
f"'outbound_on_match'"
)
return outbound, inbound, on_match
@@ -19,7 +19,7 @@ from math import log2
from collections import Counter
from urllib.parse import quote as url_quote
from .egress_addon_core import ScanResult
from .addon_core import ScanResult
# ---------------------------------------------------------------------------

Some files were not shown because too many files have changed in this diff Show More