Compare commits

..

36 Commits

Author SHA1 Message Date
didericis-claude bbb8913382 refactor(git-gate): centralize hostname qualification in globalize_slug
test / unit (pull_request) Successful in 1m19s
test / integration (pull_request) Successful in 24s
test / coverage (pull_request) Successful in 1m24s
lint / lint (push) Successful in 2m37s
test / unit (push) Successful in 1m27s
test / integration (push) Successful in 35s
test / coverage (push) Successful in 1m36s
Update Quality Badges / update-badges (push) Successful in 1m29s
Adds globalize_slug(slug) to bottle_state alongside bottle_identity.
git_gate_provision now calls globalize_slug(slug) instead of inlining
socket.gethostname(), so the hostname-qualification logic has a single,
named home. Assumes slug is a mint_slug output.

Title format changes from bot-bottle:{host}:{slug}:{name}
to bot-bottle:{host}-{slug}:{name} to match the globalize_slug contract.
2026-07-16 23:07:12 -04:00
didericis-claude 59be808ab1 feat(git-gate): include hostname in deploy key title
Closes #388 (part 1 of 3). Deploy key titles now carry the machine
hostname so keys provisioned on different hosts don't collide with
each other on the forge when a prior bottle was never torn down.

Title format: bot-bottle:<hostname>:<slug>:<repo-name>
2026-07-16 23:07:12 -04:00
Quality Badge Bot eb63bd417d chore: update quality badges
- Coverage: 82%
- Core coverage: 95%

[skip ci]
2026-07-17 03:00:40 +00:00
didericis 943049733e test(supervise): stub the client in cmd_supervise crash-logging tests
test / unit (pull_request) Successful in 1m23s
test / integration (pull_request) Successful in 38s
test / coverage (pull_request) Successful in 1m42s
lint / lint (push) Successful in 2m15s
test / unit (push) Successful in 1m17s
test / integration (push) Successful in 31s
test / coverage (push) Successful in 1m31s
Update Quality Badges / update-badges (push) Successful in 1m35s
`cmd_supervise` establishes the orchestrator client up front (since the
HTTP-bridge move in 27fe03b), so these tests — which mock `curses.wrapper`
to exercise the KeyboardInterrupt / Die / crash-log paths — only reached
those paths when a live orchestrator happened to be reachable. On CI
(none reachable) the up-front connect errored and `cmd_supervise`
returned 1 before curses, failing all four. They passed locally only
because a dev orchestrator was up.

Stub `supervise_cli._client` in the class setUp so the tests isolate the
post-connect behavior they actually cover, independent of environment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:44:47 -04:00
didericis c15eed4f2e fix(supervise): list-egress-routes returns the bottle's real routes
In consolidated mode the gateway's static route table is empty (routes
are resolved per request by source IP), but `list-egress-routes` was
reading that static table via the `_egress.local/allowlist` introspection
endpoint — so it always returned an empty allowlist. Agents, told to call
it before composing an egress proposal, then sent a routes.yaml with only
the newly-needed host; approving it replaced the whole policy and silently
dropped base routes like api.anthropic.com, breaking the bottle's egress.

Answer `list-egress-routes` from the calling bottle's resolved policy
(same (source_ip, identity-token) attribution the proposal path uses,
same JSON shape the single-tenant introspection endpoint returns).
Fail-closed to an empty list on an unreachable orchestrator; falls back
to the introspection endpoint in single-tenant mode (no resolver).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:18:14 -04:00
didericis b1df380ae1 feat(supervise): show the bottle's human slug, not its opaque id
Consolidated proposals are keyed by the orchestrator-assigned bottle_id,
so the supervise TUI was rendering a hex id (e.g. 3601cbe883c2786d) as
the bottle name — and the resume hint printed `./cli.py resume <id>`,
which resume can't take (it wants the human identity/slug).

`supervise_pending` now tags each dict with `bottle_label` — the human
slug resolved from registry metadata, falling back to the id when the
bottle is gone. `QueuedProposal` carries the label; every display site
(list rows, detail view, status lines, resume hint) shows it, while
approve/reject still key on `proposal.bottle_slug` (the id).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:05:21 -04:00
didericis 5f59df9e10 fix(supervise): resolve approval target by bottle_id, not human slug
In consolidated mode the supervise server attributes each proposal to
the orchestrator-assigned bottle_id and stores that as the proposal's
`bottle_slug` (supervise_server `_attributed_config`). But
`_record_for_slug` only scanned registry metadata for a matching human
slug, so approving an egress proposal 409'd with "bottle <id> is no
longer registered; cannot apply the route change" — the id never
matched a human slug.

Resolve the record by bottle_id first (the consolidated reality),
keeping the metadata-slug scan as a fallback for legacy single-tenant
proposals. The prior tests keyed proposals by the human slug matching
the metadata, exercising only the fallback path — added a test that
mirrors production (proposal keyed by bottle_id, distinct human slug in
metadata).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:01:14 -04:00
didericis 2641ab70fd feat(supervise): start orchestrator on demand via backend-agnostic bring-up
Add BottleBackend.ensure_orchestrator() -> str: the backend-agnostic
entry point that brings up the per-host orchestrator + shared gateway
(idempotent) and returns its host-reachable control-plane URL. Docker
starts the orchestrator + gateway containers (OrchestratorService);
firecracker boots the infra VM; macos-container dies with a pointer
(no orchestrator). Previously bring-up was reachable only through each
backend's consolidated_launch, with no shared handle.

Wire it into `bot-bottle supervise`: supervise is often the first thing
an operator runs, before any bottle has booted the control plane, so
`_resolve_orchestrator_url` now starts the selected backend's
orchestrator on demand when discovery finds nothing, instead of failing
with "launch a bottle first".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 21:52:11 -04:00
didericis 265119d601 fix(supervise): docker gateway shares the one host DB (Step 2d)
lint / lint (push) Successful in 2m24s
test / unit (pull_request) Failing after 1m27s
test / integration (pull_request) Successful in 36s
test / coverage (pull_request) Failing after 1m16s
The docker gateway container ran the supervise daemon but bind-mounted no
DB and set no SUPERVISE_DB_PATH — so the daemon wrote proposals to a
container-local, ephemeral SQLite file, disconnected from the host DB the
orchestrator (and now the operator, over HTTP) uses. Same split-DB bug
firecracker had; Dockerfile.gateway even documents the mount
(`/run/supervise/bot-bottle.db bind-mounted at run time`) that
ensure_running never provided.

Bind-mount the host DB dir into the gateway at /run/supervise and set
SUPERVISE_DB_PATH, so the daemon queues into the same file the
orchestrator container opens (its BOT_BOTTLE_ROOT bind-mount) and the
operator reaches over the control plane. One DB per host, shared by
bind-mounts — docker now on the exact same supervise path as firecracker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 19:01:45 -04:00
didericis 27fe03b612 feat(supervise): host TUI drives approvals over HTTP, not the DB (Step 2b/2c)
The `bot-bottle supervise` operator TUI read and wrote the queue DB
directly and tried a backend-specific live "apply" (which was unwired —
it raised). It now talks only to the orchestrator control plane:

- OrchestratorClient gains supervise_pending() + supervise_respond().
- discover_orchestrator_url() finds the one running per-host control
  plane by health-probing the backends' well-known :8099 addresses
  (docker publishes on loopback; the firecracker infra VM serves it on
  the orchestrator TAP) — no backend branching in the TUI.
- discover_pending/approve/reject call the client; the server does the
  apply + response + audit atomically. The dead direct-DB apply/audit
  helpers and the docker/macos applicator imports are gone.
- A missing control plane is now a clean one-line error up front, not a
  mid-curses crash.

CLI tests move to mocking the client (the DB-write behaviour they used to
assert is now server-side, covered by test_orchestrator_service). Docker's
orchestrator-container DB wiring lands next so its /supervise endpoints hit
the same shared DB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:57:47 -04:00
didericis 9085d6f713 feat(supervise): orchestrator-side operator-approval API (Step 2a)
The orchestrator owns the single DB *and* the live policy, so operator
decisions belong there — applied server-side, reached over HTTP. Adds:

  GET  /supervise/proposals   -> pending proposals across bottles
  POST /supervise/respond     -> apply + record an operator decision

`supervise_respond` is one atomic server-side op on the one DB: approve/
modify on an egress tool rewrites the bottle's policy (so the gateway
serves the new routes on its next /resolve — the live "apply" that was a
documented TODO), then writes the queued Response (unblocking the agent's
MCP call) and an audit entry. reject records the response + audit only.
Fails closed (409) when the proposal is unknown or the bottle was torn
down before the operator acted (an egress apply would have no target).

This is the server half of unifying every backend onto one HTTP path for
supervise; the host TUI (direct-DB today) moves onto this client next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:51:55 -04:00
didericis 38bc555dbf fix(supervise): single migrated DB for firecracker — orchestrator owns supervise tables
lint / lint (push) Successful in 2m24s
test / unit (pull_request) Successful in 1m21s
test / integration (pull_request) Successful in 29s
test / coverage (pull_request) Successful in 1m29s
The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block:
it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray,
unmigrated SQLite file and `write_proposal` hit "no such table:
supervise_proposals". Meanwhile the in-VM control plane migrated only the
registry table (orchestrator_bottles) into its own DB, and the host
operator reads a third, disconnected DB — three files, none shared.

Consolidate to one DB per host, owned by the control plane on the
persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db):

- orchestrator startup now migrates the supervise queue + audit tables
  into the same file it migrates the registry into (StoreManager), so the
  control-plane DB carries every table.
- the in-VM supervise daemon is pointed at that same file via
  SUPERVISE_DB_PATH, so daemon and control plane share one queue.

`list-egress-routes` already worked (no DB); egress-allow now queues +
waits on the single persisted DB instead of erroring. Validated against a
live infra VM: migrate + write_proposal succeeds and the proposal is
queued. The host-operator HTTP bridge (so approvals complete from the
host, unifying docker onto the same path) is the follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:24:45 -04:00
didericis a208bcde08 feat(firecracker): stream buildah build output live during agent-image build
lint / lint (push) Successful in 2m20s
test / unit (pull_request) Successful in 1m19s
test / integration (pull_request) Successful in 25s
test / coverage (pull_request) Successful in 1m19s
The agent-image build ran over ssh with output fully captured, shown
only as a 20-line tail on failure — so a successful (or in-progress)
first build was a long silent wait through the base pull + apt/npm
installs. Stream the `buildah build` step's stdout/stderr straight to
our own (like the docker backend's `docker build`) via a non-capturing
_ssh_streamed helper, so the operator sees `STEP i/n` progress live.
Failure now points at the streamed output above instead of a captured
tail. The smoke test and rootfs export stay captured (short / piped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:01:59 -04:00
didericis c0066d2cd2 fix(firecracker): quote guest argv tokens so codex's multi-word prompt survives ssh
The interactive agent command is sent to the guest by spreading the
remote argv as separate ssh arguments; ssh space-joins everything after
the host into one line that the guest login shell re-parses. That only
works while every token is a "simple word" — which held for claude
(`--append-system-prompt-file <path>`) but not for codex's
`read_prompt_file` mode, whose positional is a whole sentence:
"Read and follow the instructions in <path>.". The guest shell re-split
it on spaces, so codex received `Read` as the prompt and `and`, `follow`,
… as extra args — failing with `unrecognized subcommand 'and'` the moment
an interactive codex session attached.

Pre-quote each remote token with shlex.quote before ssh joins them (the
same ssh→guest-shell discipline infra_vm/cp_in already use). Simple words
are unchanged, so existing behaviour and the parity/structure tests are
untouched; an arg with spaces now survives as a single argument.

Regression test round-trips the joined remote command back through
shlex.split and asserts codex's prompt comes out as exactly one arg.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:01:59 -04:00
didericis 7118480d0a fix(codex): register supervise MCP via config.toml http_headers, not mcp add --header
lint / lint (push) Successful in 2m39s
test / unit (pull_request) Successful in 1m37s
test / integration (pull_request) Successful in 29s
test / coverage (pull_request) Successful in 1m44s
The Codex CLI has no `codex mcp add --header` flag (verified against
0.144.5 and the codex-rs `AddMcpStreamableHttpArgs` surface: only
`--url`, `--bearer-token-env-var`, `--oauth-*`, `--env`). The old call
therefore exited nonzero on every codex bottle; provisioning only
warned and continued, so supervise was silently unregistered — and
under mandatory (source_ip, token) attribution the suggested manual
recovery (`codex mcp add supervise --url ...`, no token) could not
restore access either.

Write the `[mcp_servers.supervise]` streamable-HTTP entry directly into
`~/.codex/config.toml` instead, delivering the identity token via the
Codex-supported `http_headers` key (the only way to attach a static
request header to an HTTP MCP server). Registration failure is now
FATAL when supervise is enabled, rather than a warning.

Test validates the generated entry against the real Codex config
surface: it must parse as TOML into a streamable-HTTP server carrying
the token as `http_headers["x-bot-bottle-identity"]`, and must use only
keys accepted by `RawMcpServerConfig` (config.toml is
`deny_unknown_fields`). Also covers custom `CODEX_HOME`, the no-token
case, and the now-fatal failure path.

Refs: PR #354 review (codex P1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 17:42:36 -04:00
didericis d4b27ebf1f feat(gateway): mandatory identity-token attribution on every data plane (PR #354 review)
lint / lint (push) Successful in 2m12s
test / unit (pull_request) Successful in 1m12s
test / integration (pull_request) Successful in 27s
test / coverage (pull_request) Successful in 1m21s
Codex review: the /31 TAP doesn't make source IP unspoofable, and the
app-layer token was returned by launch but never delivered or enforced, so
a spoofed source could select a victim bottle's policy/tokens. Make the
token mandatory and deliver it on each attributed plane (anti-spoof landed
separately as the network boundary).

Enforcement (control plane):
- `Orchestrator.resolve` now requires a matching (source_ip, identity_token)
  pair (constant-time) — no source-IP-only fallback. `/resolve` fail-closes
  (403) on a missing/empty/mismatched token.

Delivery, per plane (the token is `token_urlsafe`, safe in a URL):
- egress: proxy credentials (`HTTPS_PROXY=http://bottle:<token>@gw`). The
  addon reads `Proxy-Authorization` — from the request (HTTP) or captured at
  the CONNECT for HTTPS tunnels (keyed by client conn, cleared on disconnect)
  — validates, and strips it (+ the legacy header) before upstream.
- git-http: a URL-scoped `http.<gate>/.extraHeader: x-bot-bottle-identity`
  in the agent's git config (only over the http transport).
- supervise: `mcp add --header x-bot-bottle-identity: <token>` (claude +
  codex); the server reads the header and passes it to resolve.

Wiring: thread `ctx.identity_token` onto the firecracker + docker plans and
into the agent env/config at launch.

Verified on a KVM host: egress with the correct proxy-cred token returns
200 (HTTP and HTTPS/CONNECT), and no-token / wrong-token return 403; a real
`cli.py start --backend=firecracker` launch provisions git config + the
supervise MCP header and reaches the agent session, all under mandatory
enforcement. Fixed a `claude mcp add` arg-order bug (--header must follow
the positional name/url) found by that launch.

Transparent proxy for tools that ignore proxy env is deferred to a
follow-up (see thread); anti-spoof + host firewall remain the fail-closed
boundary.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 17:02:41 -04:00
didericis 914f01fa8f fix(firecracker): serialize infra-VM create + agent builds (PR #354 review)
lint / lint (push) Successful in 2m8s
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 27s
test / coverage (pull_request) Successful in 1m22s
Codex flagged that parallel `start`s race on shared state: two cold
launches could both stop/build/boot the singleton on the same rootfs/PID,
concurrent builds share the infra VM's buildah store, the cleanup did
`buildah rm -a` (nuking a peer build's container), and the rootfs cache
was populated non-atomically.

- `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the
  cold stop/build/boot path, with a double-checked health re-test under
  the lock so a second launcher adopts rather than re-boots. The healthy
  fast-path stays lock-free.
- `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`)
  around cache-lookup + build + publish; build into `.building-<digest>`
  and publish by atomic `os.rename`, so a partial build never appears as
  `agent-<digest>`.
- scoped cleanup: per-build named working containers
  (`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`;
  also cleared before the build to recover from a crashed prior run.

Verified: agent image builds via the locked/atomic path, infra VM stays
healthy, agent boots, `claude --version` = 2.1.172. Full unit suite +
pyright green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 16:15:58 -04:00
didericis 43c3d4408e fix(firecracker-netpool): per-TAP anti-spoof so source-IP attribution is sound (PR #354 review)
The /31 point-to-point TAP does NOT make a guest source address
unspoofable: root in an agent VM can source another bottle's guest IP on
its own bbfc TAP. The isolation table only matched iifname class + port
(DNAT) and never bound iifname to its assigned ip saddr — so a spoofed
source was DNAT'd to the gateway and attributed to the *victim* bottle,
getting the victim's policy/tokens. Source-IP attribution was therefore
not actually sound.

Add one anti-spoof rule per slot in the isolation forward chain, before
the established/DNAT accepts: `iifname bbfcN ip saddr != <guestN> drop`.
Generated in the existing setup loop — no new dependency, ~pool_size
lines. Legit traffic (correct saddr) is unchanged; a spoofed saddr on any
bbfc TAP is dropped before it can be attributed.

Apply with a nixos-rebuild (the systemd unit re-runs this script). Codex
review blocker; the app-layer identity token (defense-in-depth) is wired
separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 16:09:45 -04:00
didericis 0a26b8795a refactor(gateway): install gitleaks by pinned+verified download, drop third-party base image (PR #354 review)
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 24s
test / coverage (pull_request) Successful in 1m27s
The gateway used `FROM zricethezav/gitleaks AS gitleaks-src` purely to
COPY the binary out — a supply-chain surface (a whole third-party image as
a build input, tying us to its cadence). Install gitleaks from its official
release instead, pinned by version + SHA256 and verified. python (already
in the image) does the download, so no curl/wget is added. trixie apt also
ships gitleaks but an older 8.16; the pinned download keeps the verified
8.30.1 (byte-identical to what the image provided).

Verified: gateway + infra images build, gitleaks 8.30.1 runs in both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 15:53:40 -04:00
didericis c60e6b7e9f refactor(firecracker): single infra VM builds too — buildah in one image (PR #354 review)
lint / lint (push) Successful in 2m18s
test / unit (pull_request) Successful in 1m13s
test / integration (pull_request) Successful in 26s
test / coverage (pull_request) Successful in 1m22s
Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:

- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
  trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
  (Debian bookworm). trixie ships buildah 1.39, which can build agent
  Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
  instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
  orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
  definition of the control-plane payload; the infra image `COPY --from`s
  it (same trixie base → clean copy, and future deps like iroh are added
  once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
  over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
  `ensure_built` builds orchestrator + gateway before infra (FROM gateway,
  COPY --from orchestrator).

Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 15:23:36 -04:00
didericis 2d37965249 fix(firecracker-netpool): make nft table installs delete-first idempotent
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 28s
test / coverage (pull_request) Successful in 1m22s
Each nft table (isolation, orchestrator-egress nat, agent->gateway route)
was re-applied as a plain `table {...}` block, which on a bare `up`
re-apply (not the systemd down->up path) would append duplicate rules or
error on the existing base chains. Use the standard delete-first pattern
(create empty, delete, recreate) so `up` lands identical state regardless
of history — the setup reproduces cleanly on a fresh install and on
re-apply, not just via a full down->up cycle.

No functional change to the resulting ruleset; only its idempotency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 4873030550 feat(firecracker): git-gate over SSH into the gateway VM (Stage B, 7/n)
git-gate now works end-to-end for git-upstream bottles on the infra VM.
Three fixes surfaced by driving a real clone through git-http:

- `SshGatewayTransport.cp_into` preserves the source file mode (docker cp
  does). The access-hook is staged 0700 and git-http execs it directly; a
  plain `cat >` landed it 0644 -> EACCES. Keys stay 0600.
- the infra init runs `BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise`:
  the VM backend reaches git over git-http (9420), so the git:// daemon
  (git-gate, whose /git-gate-entrypoint.sh the consolidated model doesn't
  stage) is left out instead of crash-looping.
- `build_infra_rootfs_dir` folds the init's content-hash into the rootfs
  cache key, so an init change actually rebuilds the rootfs (the base image
  digest alone wouldn't catch it).

Verified on a KVM host: launch_consolidated provisions a git-upstream
bottle's repo + creds into the gateway VM over SSH; an agent VM clones via
git-http (source-IP attributed) and the access-hook resolves the
provisioned key + known_hosts and attempts the upstream SSH fetch —
failing closed only because the test used a dummy key + fake upstream
("refusing to serve stale data"). With real creds the fetch serves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis b93b14f5c2 feat(firecracker): persistent registry volume for the infra VM (Stage B, 6/n)
The infra VM's rootfs is ephemeral (rebuilt each boot), so the bottle
registry DB needs durable storage across restarts. Give the infra VM a
firecracker analogue of a docker volume: a host-side ext4 file attached as
a second virtio-block device (guest /dev/vdb), mounted at the control
plane's DB dir (/var/lib/bot-bottle, where host_db_path lives at
db/bot-bottle.db).

- firecracker_vm.boot/_config take an optional `data_drive` (a non-root,
  RW second drive).
- infra_vm creates the volume on first use (`mke2fs` an empty ext4 at
  <fc-cache>/infra/registry.ext4) and mounts /dev/vdb in the PID-1 init
  before the control plane starts. It's a plain ext4 file, so
  `sudo mount -o loop <path>` (VM stopped) inspects bot-bottle.db directly.

Verified on a KVM host: register a bottle, restart the infra VM (fresh
rootfs, same volume) — /dev/vdb re-mounts and the registry `db/` dir + a
marker file survive the restart.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 957eb19368 feat(firecracker): launch through the infra VM, not Docker (Stage B, 5/n)
consolidated_launch now drives the persistent infra VM instead of the
`_FirecrackerOrchestratorService` Docker containers — the point Docker
leaves the Firecracker launch path.

- launch_consolidated: `infra_vm.ensure_running()` (singleton) for the
  control plane + gateway; register the bottle over HTTP at the infra VM's
  guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the
  gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`.
- teardown_consolidated deregisters + deprovisions but does NOT stop the
  infra VM (persistent per-host singleton shared by every bottle).
- new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the
  stable key + orchestrator link IP, so teardown needs no live handle).
- drop the DockerGateway/OrchestratorService machinery from the firecracker
  consolidated path (still used by the docker backend).

launch.py already calls launch_consolidated, so the whole firecracker
launch path now uses the VM. pyright + unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 5dfb9b0d75 refactor(gateway): parameterize git-gate provisioning transport (Stage B prep)
git-gate provisioning into the running gateway was hard-wired to docker
exec/cp. Extract a backend-neutral `GatewayTransport` (exec + cp_into) so
the same provisioning logic serves both the docker gateway container and
the firecracker gateway VM (over SSH, added with the launch swap).

- `provision_git_gate` / `deprovision_git_gate` now take a transport
  instead of a gateway name; `DockerGatewayTransport` wraps the existing
  docker exec/cp behavior. deprovision is best-effort (catches the
  transport error) — matching the prior idempotent teardown.
- both consolidated_launch callers pass `DockerGatewayTransport(name)` —
  no behavior change; the firecracker swap flips only its own to SSH.

Pure refactor: docker path unchanged, gateway_provision tests updated to
construct the transport, all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis bb434b14d7 feat(firecracker): persistent infra-VM singleton lifecycle (Stage B, 4/n)
The infra VM must outlive the short-lived `start` launcher and be reused
across launches. Add an idempotent singleton:

- `ensure_running()` adopts the infra VM when its control plane is already
  healthy (a prior launcher booted it), else clears any stale VM and boots
  a fresh one. Returns a handle usable for CA fetch / git-gate provisioning
  whether we booted it or adopted it.
- boot is `detached` (firecracker in its own session via start_new_session)
  so it survives the launcher exiting; its PID is recorded so a later
  process can `stop()` it. `_kill_pidfile` SIGTERM/SIGKILLs but only if the
  PID is still a firecracker process (guards a recycled PID).
- a STABLE SSH key (generated once under the infra cache dir, re-injected
  each boot via the cmdline) so any launcher can SSH in to fetch the CA /
  provision, not just the one that booted the VM. `InfraVm.vm` is None in
  the adopted case; teardown then goes through the PID file.

Verified on a KVM host: first ensure_running boots; a second adopts it
(same PID, no reboot) and can still reach /health and fetch the gateway CA
over SSH; stop() tears it down (control plane then unreachable).

Next: git-gate provisioning into the VM over SSH (today docker exec/cp),
then swap consolidated_launch.py onto the infra VM.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis d79d5b295a feat(firecracker): route agent VMs to the gateway VM (Stage B, 3/n)
Agent VMs must reach the shared gateway that now runs in the infra VM
(egress:9099 / supervise:9100 / git-http:9420 at the orchestrator link's
guest IP). Add a PREROUTING DNAT: agents keep addressing their own
host-side TAP IP on the gateway ports, and the rule redirects that to the
infra VM. The isolation table's existing `ct status dnat accept` forward
rule lets the DNAT'd traffic through; every other agent egress stays
dropped, so a bottle still reaches only the gateway and nothing else.

Source IP is deliberately NOT masqueraded: the gateway attributes each
request to the originating bottle by its guest IP, which the /31 TAP + the
bot_bottle_fc nft table make unspoofable. Keeping the agent addressed at
its own host TAP IP means no per-bottle config change vs the docker-DNAT
path it replaces.

- scripts/firecracker-netpool.sh: `_install_gateway_route` adds
  `table ip <table>_gw` (prerouting dstnat -> orch_guest on the gateway
  ports); wired into up/down/status. The nix module needs no change — it
  runs this script, and the ports are baked in.

Verified on a KVM host: an agent VM's `curl -x http://<its-host-tap>:9099`
reaches mitmproxy in the infra VM and gets a 403 (correct policy denial
for an unregistered bottle) — i.e. the route lands end-to-end. Persist
with a nixos-rebuild; the imperative rule holds until then.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis e1610121c0 feat(firecracker): run the gateway data plane in the infra VM too (Stage B, 2/n)
The single infra VM now runs BOTH the orchestrator control plane and the
gateway data plane (egress / supervise / git-http), multi-tenant against
the local control plane — the single-VM shape from the Stage B design.

- Dockerfile.infra: the firecracker infra image = the gateway image +
  the baked control-plane source (FROM bot-bottle-gateway, COPY
  bot_bottle). Reuses the gateway payload rather than copying mitmproxy/
  gitleaks into a third image; the docker backend keeps its two separate
  images.
- infra_vm: build from source (gateway then infra image), and the PID-1
  init now also launches `gateway_init` with
  BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:8099. Adds `gateway_ca_pem`
  (fetch the mitmproxy CA over SSH) and the agent-facing port constants.
- init exports PATH — a bare-init shell resolves its own execs via a
  built-in default path, but that isn't in the environment, so
  gateway_init's `python3 ...` daemons would otherwise fail to spawn.

Verified on a KVM host: infra VM boots, control plane /health -> 200, and
egress:9099 / supervise:9100 / git-http:9420 all listen and are reachable
from the host over the TAP link; the gateway CA is retrievable. (git-gate
stays down until a bottle provisions its per-bottle entrypoint/creds, same
as a fresh docker gateway — non-fatal, the supervisor keeps the rest up.)

Next: agent->gateway VM-to-VM routing so bbfc* VMs reach these ports at
the infra VM and nowhere else.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 1614172423 feat(firecracker): run the orchestrator control plane as a VM (Stage B, 1/n)
First slice of Stage B: the orchestrator control plane runs as a
persistent Firecracker infra VM instead of a Docker container. The host
CLI reaches it over HTTP at the orchestrator link's guest IP; agent VMs
will reach its gateway ports (added next) over VM-to-VM routing.

- Dockerfile.orchestrator bakes the stdlib-only control-plane source
  (COPY bot_bottle) so the image is self-contained and runs from a built
  image with no runtime bind-mount — a guest VM can't bind-mount host
  source. (Build-from-source stays the default; a pull-from-registry mode
  lands later. The docker backend's dev bind-mount still overlays this.)
- util.build_base_rootfs_dir / inject_guest_boot take a `variant` +
  `init_script`, so the same orchestrator image is prepared two ways
  without a cache collision: the builder VM keeps the SSH-only agent init;
  the infra VM gets a control-plane PID-1 init.
- new firecracker/infra_vm.py: boot the infra VM on the orchestrator link,
  run `python -m bot_bottle.orchestrator` as PID 1, and poll /health.

Verified on a KVM host: infra VM boots, control plane answers
`GET /health -> 200 {"status":"ok"}` from the host over the TAP link.
Next: fold gateway_init (egress/git-gate/supervise) into the same VM,
then agent->gateway routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 4edd7803e8 feat(firecracker): build agent images in a builder VM, not host docker (Stage 3)
Replace the host `docker build` + `docker export` behind the Firecracker
agent rootfs with an in-VM buildah build. `image_builder.build_agent_rootfs_dir`
boots a throwaway builder VM (the orchestrator image, which carries
buildah) on the NAT'd orchestrator link, sends the Dockerfile over SSH,
`buildah build`s it, smoke-tests the result with `buildah run` (the
image's own PATH, so it catches an npm silent-failure stub), and streams
the rootfs tar back into the content-addressed cache dir — the same base
dir `util.build_rootfs_ext4` already turns into a bootable ext4 with
`mke2fs -d`. No host Docker daemon, no root-equivalent `docker` group;
an untrusted Dockerfile runs in a confined microVM, not on the host.

- new firecracker/image_builder.py (boot → build → smoke → stream).
- launch.py: `_build_agent_image` → `_build_agent_base`, returning the
  base dir from the builder VM. The committed-snapshot (freeze/migrate)
  path still exports via host docker until it too is ported.
- util: `_inject_guest_boot` → public `inject_guest_boot` (shared with
  the builder). unit tests for the cache decision + smoke-test paths.

Verified on a KVM host end-to-end: builds the real claude Dockerfile
(node:22-slim + npm claude-code) in-VM in ~60s, the produced agent VM
boots and `claude --version` returns 2.1.172; the content cache skips
the rebuild on a repeat launch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 81a2f15046 feat(firecracker): NAT'd egress link for the orchestrator/builder VM
The orchestrator/gateway VM is trusted infra, not an isolated agent: it
builds agent images in-VM (buildah must FROM-pull + apt/npm) and, in the
Stage B cutover, forwards agent egress upstream. Give it a dedicated TAP
(`bborch0`) on a /31 at the top of the IP_BASE /16 (clear of the bbfc*
agent pool at the bottom), NAT'd out the host uplink — while agent VMs
keep their fail-closed, gateway-only isolation table.

- netpool.defaults.env / netpool.py: new BOT_BOTTLE_FC_ORCH_IFACE +
  `orch_slot()` (index -1 sentinel; host x.y.255.0 / guest x.y.255.1).
- scripts/firecracker-netpool.sh: create + address the orchestrator TAP;
  `bot_bottle_fc_nat` table masquerades its /31 out the uplink and
  accepts its forward path. Because bootstrap still runs Docker (whose
  FORWARD policy is DROP), a best-effort, guarded, idempotent DOCKER-USER
  ACCEPT is added too (skipped once Docker is gone). down/status updated.
- nix/firecracker-netpool.nix: mirror the option, pass it via the unit
  Environment= (the store-copied script can't read the defaults file),
  and add iptables to the unit path for the DOCKER-USER step.

Agent isolation is unchanged: the new rules only ever accept/masquerade
the orchestrator link and never drop, so they can't weaken the bbfc*
drops. Applied by re-running `sudo ./scripts/firecracker-netpool.sh up`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 75b122398d feat(orchestrator): add buildah for in-VM agent-image builds (PRD 0069 Stage 3)
The Firecracker backend's remaining host-Docker dependency is building
users' agent Dockerfiles. Move that build *into the orchestrator VM*:
give the orchestrator image buildah (rootless, daemonless) so it builds
agent images itself, and the host needs no Docker daemon and no
root-equivalent `docker` group. An untrusted Dockerfile then builds
inside the confined orchestrator VM rather than on the host — strictly
more isolated than host `docker build`.

- buildah + crun (OCI runtime) + netavark/aardvark-dns (network backend
  for FROM pulls and RUN egress), installed explicitly since
  --no-install-recommends strips buildah's helper deps.
- vfs storage + chroot isolation (STORAGE_DRIVER/BUILDAH_ISOLATION) so
  buildah needs neither fuse-overlayfs / an overlay kernel module nor
  configured subuid maps — it works unconditionally as root in a minimal
  microVM rootfs. (Slower than overlay; a build-cache pass is deferred.)

Bootstrap: the orchestrator rootfs is still produced from this image via
host docker export; a later step pulls a pre-built image instead.

Verified: the orchestrator image boots as a Firecracker VM and buildah
runs inside it as real VM root (reaches the base-image pull; build-time
egress wiring is the next step).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis-claude 2e738c3338 fix(docker): use run_docker in docker_exec, docker_cp, verify_agent_image
The rebase onto lazy-backend-imports converts existing helpers (image_exists,
container_exists, etc.) to run_docker; the three new functions added in this
branch still called subprocess.run directly. Switch them over for consistency.
2026-07-16 14:36:54 -04:00
didericis-claude 9369fb7de5 feat(firecracker): implement consolidated orchestrator launch (PRD 0070)
Replace the per-bottle Docker sidecar bundle with the shared per-host
orchestrator + gateway, mirroring what the Docker backend already has.

- Add `bot_bottle/backend/firecracker/consolidated_launch.py`:
  `_FirecrackerOrchestratorService` (subclasses `OrchestratorService`,
  overrides `_gateway()` to return a `DockerGateway` with host port
  bindings so Firecracker VMs can reach it via their TAP link);
  `launch_consolidated()` registers the bottle by guest IP (attribution
  key), provisions git-gate into the shared gateway, and returns the
  shared CA + orchestrator URL for teardown; `teardown_consolidated()`
  deregisters and cleans up.

- Rewrite `bot_bottle/backend/firecracker/launch.py`: removes the
  per-bottle sidecar bundle (`_start_sidecar_bundle`, `_stage_git_gate`,
  etc.) and `_mint_certs`; wires `launch_consolidated()` instead. The VM
  still sends to `host_tap_ip:PORT` — Docker's PREROUTING DNAT + the nft
  `ct status dnat accept` rule in the forward chain route the traffic to
  the shared gateway container.

- Extend `DockerGateway` with `host_port_bindings` so the Firecracker
  gateway publishes its ports on the host (`0.0.0.0:PORT`).

- Parameterise `OrchestratorService` with `orchestrator_name` /
  `orchestrator_label` so Docker and Firecracker orchestrators can
  coexist on the same host (`bot-bottle-orchestrator` vs
  `bot-bottle-fc-orchestrator`).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-16 14:36:54 -04:00
didericis 9afdeff619 refactor(firecracker): use docker_mod instead of hand-rolled docker helpers
firecracker/launch.py reimplemented docker build/image-exists/rm/exec/cp
as private functions instead of the shared docker_mod used by the
docker and macos-container backends. Switching to docker_mod dedupes
the logic and gets --no-cache support for free (docker_mod.build_image
already reads BOT_BOTTLE_NO_CACHE); docker_mod gains docker_exec/
docker_cp general-purpose helpers to cover what the private versions did.
2026-07-16 14:36:54 -04:00
didericis e45df03bd9 fix: smoke-test agent images after build, add start --no-cache
npm treats optionalDependencies failures as non-fatal, so a transient
network blip fetching claude-code's platform-native binary during
`npm install -g` left a stub CLI in an image that still "built"
successfully — then got baked into the Docker/Container layer cache
until forced to rebuild. Post-build smoke test (provider-declared
argv, run in a throwaway container of the freshly built image) fails
the launch loudly instead of shipping a broken image; --no-cache
gives an escape hatch to force a from-scratch rebuild.

Closes #353.
2026-07-16 14:36:54 -04:00
56 changed files with 3014 additions and 453 deletions
+28 -18
View File
@@ -34,35 +34,45 @@
# 9420 git-gate smart HTTP (VM-backend agent-facing transport)
# 9100 supervise (MCP HTTP)
# Stage 1: gitleaks binary. The upstream gitleaks image is alpine
# with the binary at /usr/bin/gitleaks. Pinned by digest in lockstep
# with Dockerfile.git-gate's prior base (now deleted at chunk 3).
FROM zricethezav/gitleaks@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f AS gitleaks-src
# Stage 2: assembly. mitmproxy/mitmproxy is debian-slim-based with
# Python + mitmdump pre-installed — heavier than the others, so
# this stage starts there and pulls the standalone binaries in.
FROM mitmproxy/mitmproxy:11.1.3
# Run as root inside the bundle. The bundle is the isolation
# boundary; per-daemon user separation inside it is not load-bearing
# and complicates the supervisor's spawn path.
USER root
# Based on `python:3.12-slim` (Debian trixie) rather than the
# `mitmproxy/mitmproxy` image (Debian bookworm) so the whole stack —
# gateway here, and the firecracker infra image that builds FROM this —
# lands on trixie, whose buildah (1.39) can build agent Dockerfiles that
# use heredocs. mitmproxy is pip-installed to the same effect as the
# upstream image. (bookworm's buildah is 1.28, which can't parse
# `RUN ... <<EOF`; see the infra image + PR discussion.)
FROM python:3.12-slim
# Runtime system deps:
# git supplies the `git daemon` subcommand (no separate package)
# plus the core `git` binary the pre-receive hook invokes.
# openssh-client supplies the upstream SSH transport the
# pre-receive hook uses to forward accepted refs.
# ca-certificates is needed for mitmdump upstream TLS (the
# base image already has it; listed for explicitness).
# ca-certificates is needed for mitmdump upstream TLS.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
git openssh-client ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Pull the standalone binaries into the final image.
COPY --from=gitleaks-src /usr/bin/gitleaks /usr/bin/gitleaks
# mitmdump (the egress data plane). The upstream mitmproxy image baked
# this in; on the plain python base we pip-install the same pinned
# version. Its CA dir is set explicitly via `--set confdir=` in
# egress-entrypoint.sh, so it doesn't depend on a `mitmproxy` home user.
RUN pip install --no-cache-dir mitmproxy==11.1.3
# gitleaks (the pre-receive hook's secret scanner). Installed from its
# official release, pinned by version + SHA256 and verified — rather than
# using a third-party image as a build stage (supply-chain surface, and it
# would pin us to that image's cadence). python (already present) does the
# download so we add no curl/wget. trixie apt also ships gitleaks, but an
# older 8.16; the pinned download keeps the verified 8.30.1.
ARG GITLEAKS_VERSION=8.30.1
ARG GITLEAKS_SHA256=551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb
RUN url="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
&& python3 -c "import sys,urllib.request; urllib.request.urlretrieve(sys.argv[1], '/tmp/gitleaks.tar.gz')" "$url" \
&& echo "${GITLEAKS_SHA256} /tmp/gitleaks.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/gitleaks.tar.gz -C /usr/bin gitleaks \
&& rm /tmp/gitleaks.tar.gz
# Project Python: addon + server modules + the init supervisor.
# Kept flat under /app/ so mitmdump's loader resolves them as
+45
View File
@@ -0,0 +1,45 @@
# Firecracker single infra-VM image (PRD 0070 Stage B).
#
# The per-host infra VM runs the orchestrator control plane, the gateway
# data plane, AND builds agent images (buildah) — all in one microVM (see
# backend/firecracker/infra_vm.py). It composes:
# * FROM the gateway image (mitmproxy / git / gitleaks / supervise + the
# flat daemon modules) — now trixie-based, so buildah 1.39 is available;
# * `COPY --from` the orchestrator image's content (the single definition
# of the control-plane payload — see Dockerfile.orchestrator), so this
# VM and the docker backend share one orchestrator definition; and
# * buildah, installed HERE only (the docker orchestrator/gateway images
# never carry it).
#
# multi-`FROM` can't union two bases (that's multi-stage, not multiple
# inheritance), so the orchestrator content is pulled in via `COPY --from`
# rather than a second base. Both images share the trixie `python:3.12-slim`
# base, so the copy is clean (same python; future installed deps copy too).
#
# The docker backend keeps orchestrator + gateway as separate images; this
# combined image exists only for the Firecracker single-VM cut. Splitting a
# service back into its own VM later is a routing change, not a repackaging
# (PRD 0070's "secret concentration"; a disposable builder can boot from
# this same image on its own TAP).
FROM bot-bottle-gateway:latest
# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
# The Firecracker backend builds users' agent Dockerfiles *inside this VM*
# with buildah (rootless, daemonless) instead of on the host — no host
# Docker daemon, no root-equivalent `docker` group. `crun` is the OCI
# runtime; `netavark` + `aardvark-dns` are the network backend for `FROM`
# pulls + `RUN` egress. Requires the trixie base (buildah 1.39: bookworm's
# 1.28 can't parse Dockerfile heredocs that agent images use).
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
# vfs + chroot: buildah works as root in the bare microVM (no
# fuse-overlayfs / overlay module / subuid maps). Matches image_builder.
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
# The orchestrator content, pulled from its single definition. The gateway
# image already has the flat daemon modules under /app; this adds the full
# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves.
COPY --from=bot-bottle-orchestrator:latest /app/bot_bottle /app/bot_bottle
+25 -16
View File
@@ -1,27 +1,36 @@
# Orchestrator control-plane image (PRD 0070, #384).
#
# The per-host orchestrator runs `python3 -m bot_bottle.orchestrator`.
# The `bot_bottle` package is **stdlib-only** by design, so the control
# plane needs nothing but a Python runtime — none of the gateway's
# mitmproxy / git / gitleaks payload (that is the separate
# `bot-bottle-gateway` image, Dockerfile.gateway). Splitting them keeps
# the secret-dense control plane (it concentrates every bottle's egress
# tokens — see PRD 0070's "secret concentration") on a minimal
# dependency surface.
# This is the **single definition of the orchestrator's content** — the
# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
# * the docker backend, which runs this image directly as the lean
# control-plane container; and
# * the firecracker infra image (Dockerfile.infra), which `COPY --from`s
# this image's `/app/bot_bottle` so the single infra VM runs the same
# control plane. Keeping it in one place means future orchestrator deps
# (e.g. iroh) are added here once, not duplicated per backend.
#
# The repo is bind-mounted read-only into the container at run time (see
# `orchestrator/lifecycle.py`), so the source is NOT copied in here: the
# image is just the runtime. `ensure_running` recreates the container
# only when the bind-mounted source hash changes (#381), which is why
# the code stays a mount rather than a baked layer.
# It stays deliberately lean: the control plane is **stdlib-only** today, so
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
# builder, and lives only in Dockerfile.infra). Keeping the secret-dense
# control plane on a minimal dependency surface is the point (PRD 0070's
# "secret concentration").
#
# Shares the trixie `python:3.12-slim` base with the gateway image, so when
# the orchestrator grows real deps they can be `COPY --from`'d into the
# infra image cleanly (same base/python — installed packages copy safely).
FROM python:3.12-slim
# No third-party deps to install — stdlib only. Kept as an explicit,
# self-documenting stage so a future confinement step (baking the
# package, dropping the bind mount) has an obvious home.
WORKDIR /app
# The orchestrator content. Baked so the image is self-contained (runs from
# a built image, no runtime bind-mount); the docker backend may still
# bind-mount /app for dev live-reload, which simply overlays this copy.
# `.dockerignore` keeps .git/docs/*.md out of the context. (Future deps like
# iroh go here too — a shared requirements installed on this same base.)
COPY bot_bottle /app/bot_bottle
# Documentation only; lifecycle.py overrides the entrypoint to
# `python3 -m bot_bottle.orchestrator` with the runtime flags.
ENTRYPOINT ["python3", "-m", "bot_bottle.orchestrator"]
+1 -1
View File
@@ -5,7 +5,7 @@
# bot-bottle
[![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
[![coverage](https://img.shields.io/badge/coverage-84%25-brightgreen)](https://coverage.readthedocs.io/)
[![coverage](https://img.shields.io/badge/coverage-82%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-95%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
+8
View File
@@ -61,6 +61,13 @@ class AgentProviderRuntime:
prompt_mode: PromptMode
bypass_args: tuple[str, ...]
resume_args: tuple[str, ...]
# argv run inside a throwaway container of a freshly built agent
# image, right after `build_image()`, to catch a build that
# exited 0 but produced a broken CLI (e.g. an npm
# optionalDependencies fetch for a platform-native binary that
# silently no-ops on a transient failure). Empty tuple skips the
# check — not every provider has opted in yet.
smoke_test: tuple[str, ...] = ()
@dataclass(frozen=True)
@@ -259,6 +266,7 @@ class AgentProvider(ABC):
gate_scheme = getattr(plan, "git_gate_insteadof_scheme", "git")
content = git_gate_render_gitconfig(
manifest_bottle.git, gate_host, scheme=gate_scheme,
identity_token=getattr(plan, "identity_token", ""),
)
guest_gitconfig = f"{plan.guest_home}/.gitconfig"
with tempfile.NamedTemporaryFile(
+12
View File
@@ -511,6 +511,18 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
del plan
return ""
def ensure_orchestrator(self) -> str:
"""Bring up this backend's per-host orchestrator + shared gateway
(idempotent) and return the host-reachable control-plane URL.
This is the backend-agnostic bring-up entry point: `launch` calls
it as part of starting a bottle, and operator tools (`supervise`)
call it to start the control plane on demand when none is running
yet. Docker starts the orchestrator + gateway containers;
firecracker boots the infra VM. Backends with no orchestrator
(macos-container) die with a pointer — the default here."""
die(f"backend {self.name!r} has no orchestrator control plane")
@abstractmethod
def prepare_cleanup(self) -> CleanupT:
"""Enumerate orphaned resources from previous bottles. No side
+4
View File
@@ -105,6 +105,10 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def ensure_orchestrator(self) -> str:
from ...orchestrator.lifecycle import OrchestratorService
return OrchestratorService().ensure_running()
def supervise_mcp_url(self, plan: DockerBottlePlan) -> str:
"""Docker bottles reach the supervise daemon via the
compose-network alias `supervise:9100`. No per-bottle URL
+4
View File
@@ -35,6 +35,10 @@ class DockerBottlePlan(BottlePlan):
# Likewise the supervise MCP endpoint at the gateway (`http://<gw>:9100/`);
# empty → the single-tenant `supervise` alias.
agent_supervise_url: str = ""
# Per-bottle identity token the agent presents on every attributed request
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
@property
def container_name(self) -> str:
@@ -31,7 +31,13 @@ def consolidated_agent_compose(
) -> dict[str, Any]:
"""A compose spec with only the agent service, on the external gateway
network at `source_ip`, proxying egress through `gateway_ip`."""
proxy_url = f"http://{gateway_ip}:{EGRESS_PORT}"
# Deliver the identity token as egress proxy credentials — the gateway
# reads Proxy-Authorization, validates the (source_ip, token) pair, and
# strips it before upstream. git-http/supervise get it via their own
# headers (git config extraHeader / MCP header).
token = getattr(plan, "identity_token", "")
cred = f"bottle:{token}@" if token else ""
proxy_url = f"http://{cred}{gateway_ip}:{EGRESS_PORT}"
# git-http + supervise live on the gateway too and must NOT go through the
# egress proxy — the agent reaches them directly by the gateway address.
no_proxy = f"localhost,127.0.0.1,{gateway_ip}"
@@ -29,7 +29,11 @@ from ...orchestrator.gateway import GATEWAY_NAME, GATEWAY_NETWORK
from ...orchestrator.lifecycle import OrchestratorService
from ...orchestrator.registration import registration_inputs
from .gateway_net import next_free_ip
from .gateway_provision import deprovision_git_gate, provision_git_gate
from .gateway_provision import (
DockerGatewayTransport,
deprovision_git_gate,
provision_git_gate,
)
class ConsolidatedLaunchError(RuntimeError):
@@ -121,7 +125,8 @@ def launch_consolidated(
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(gateway_name, reg.bottle_id, git_gate_plan)
provision_git_gate(
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
@@ -142,7 +147,7 @@ def teardown_consolidated(
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(gateway_name, bottle_id)
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id)
__all__ = [
+63 -37
View File
@@ -15,14 +15,15 @@ bottle's push credentials out of another's repos on the shared gateway.
from __future__ import annotations
import re
from typing import Protocol
from ...docker_cmd import run_docker
from ...git_gate import GitGatePlan, git_gate_render_provision
# bottle ids index the gateway's per-bottle repo + creds dirs; they land in
# `docker cp`/`rm` path arguments, so validate before any path is built (a
# traversal id like "../etc" must never reach the container). Registry ids are
# token_hex — this is defense in depth at the docker boundary.
# exec/cp path arguments, so validate before any path is built (a traversal
# id like "../etc" must never reach the gateway). Registry ids are token_hex —
# this is defense in depth at the transport boundary.
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
@@ -30,6 +31,42 @@ class GatewayProvisionError(RuntimeError):
"""A git-gate provisioning step against the running gateway failed."""
class GatewayTransport(Protocol):
"""How the launcher stages files + runs commands in the running gateway.
Backend-neutral so the same provisioning logic serves the docker gateway
(exec/cp over the docker socket) and the firecracker gateway VM (over
SSH)."""
def exec(self, argv: list[str]) -> None:
"""Run `argv` in the gateway, raising `GatewayProvisionError` on
failure."""
def cp_into(self, src: str, dest: str) -> None:
"""Copy host file `src` to `dest` in the gateway, raising on
failure."""
class DockerGatewayTransport:
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
def __init__(self, gateway: str) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
proc = run_docker(["docker", "exec", self.gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def cp_into(self, src: str, dest: str) -> None:
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
def _require_safe(bottle_id: str) -> None:
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise GatewayProvisionError(f"unsafe bottle id {bottle_id!r}")
@@ -39,27 +76,11 @@ def _creds_dir(bottle_id: str) -> str:
return f"/git-gate/creds/{bottle_id}"
def _exec(gateway: str, argv: list[str]) -> None:
"""`docker exec` a command in the gateway, raising on non-zero exit."""
proc = run_docker(["docker", "exec", gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def _cp_into(gateway: str, src: str, dest: str) -> None:
"""`docker cp` a host file into the gateway, raising on non-zero exit."""
proc = run_docker(["docker", "cp", src, f"{gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
def provision_git_gate(gateway: str, bottle_id: str, plan: GitGatePlan) -> None:
"""Place `bottle_id`'s git-gate credentials into the running `gateway`
container and init its bare repos under `/git/<bottle_id>/`.
def provision_git_gate(
transport: GatewayTransport, bottle_id: str, plan: GitGatePlan,
) -> None:
"""Place `bottle_id`'s git-gate credentials into the running gateway and
init its bare repos under `/git/<bottle_id>/`.
Copies each upstream's identity key (and known_hosts, when present) into
`/git-gate/creds/<bottle_id>/`, then runs the namespaced provisioning
@@ -70,31 +91,36 @@ def provision_git_gate(gateway: str, bottle_id: str, plan: GitGatePlan) -> None:
# The pre-receive + access hooks are bottle-agnostic and shared by every
# bottle's repos; install them into the gateway (idempotent — same content
# each time). The per-bottle model cp'd these into each bundle at start.
_exec(gateway, ["mkdir", "-p", "/etc/git-gate"])
_cp_into(gateway, str(plan.hook_script), "/etc/git-gate/pre-receive")
_cp_into(gateway, str(plan.access_hook_script), "/etc/git-gate/access-hook")
transport.exec(["mkdir", "-p", "/etc/git-gate"])
transport.cp_into(str(plan.hook_script), "/etc/git-gate/pre-receive")
transport.cp_into(str(plan.access_hook_script), "/etc/git-gate/access-hook")
creds = _creds_dir(bottle_id)
_exec(gateway, ["mkdir", "-p", creds])
transport.exec(["mkdir", "-p", creds])
for u in plan.upstreams:
if u.identity_file:
_cp_into(gateway, u.identity_file, f"{creds}/{u.name}-key")
transport.cp_into(u.identity_file, f"{creds}/{u.name}-key")
known_hosts = str(u.known_hosts_file)
if known_hosts and known_hosts != ".":
_cp_into(gateway, known_hosts, f"{creds}/{u.name}-known_hosts")
transport.cp_into(known_hosts, f"{creds}/{u.name}-known_hosts")
# Init the bare repos + per-repo credential config for this namespace.
script = git_gate_render_provision(bottle_id, plan.upstreams)
_exec(gateway, ["sh", "-c", script])
transport.exec(["sh", "-c", script])
def deprovision_git_gate(gateway: str, bottle_id: str) -> None:
def deprovision_git_gate(transport: GatewayTransport, bottle_id: str) -> None:
"""Remove a bottle's repos + creds from the gateway on teardown. Idempotent
— an already-absent namespace is a clean no-op (best effort; a stray dir
can't leak, since attribution is by source IP and the bottle is gone)."""
_require_safe(bottle_id)
run_docker([
"docker", "exec", gateway, "rm", "-rf",
f"/git/{bottle_id}", _creds_dir(bottle_id),
])
try:
transport.exec([
"rm", "-rf", f"/git/{bottle_id}", _creds_dir(bottle_id),
])
except GatewayProvisionError:
pass # best-effort teardown; absent namespace is success
__all__ = ["provision_git_gate", "deprovision_git_gate", "GatewayProvisionError"]
__all__ = [
"provision_git_gate", "deprovision_git_gate",
"GatewayProvisionError", "GatewayTransport", "DockerGatewayTransport",
]
+5
View File
@@ -36,6 +36,7 @@ from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...egress import egress_resolve_token_values
from ...git_gate import (
provision_git_gate_dynamic_keys,
@@ -110,6 +111,9 @@ def launch(
plan.image, _REPO_DIR,
dockerfile=plan.dockerfile_path,
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any, before
# provisioning the bottle's repos into the shared gateway.
@@ -163,6 +167,7 @@ def launch(
egress_plan=egress_plan,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
identity_token=ctx.identity_token,
)
# Step 5: render + up the agent-only compose, pinned on the shared
+51 -1
View File
@@ -4,11 +4,13 @@ existence, and building images."""
from __future__ import annotations
import os
import re
import shutil
import subprocess
from typing import Iterable, Iterator
from ...docker_cmd import run_docker
from ...log import die, info
# from ...workspace import WorkspacePlan
@@ -88,6 +90,29 @@ def docker_exec_root(container: str, argv: list[str]) -> None:
)
def docker_exec(container: str, argv: list[str], *, user: str = "") -> None:
"""Run `docker exec` in the named container, dying with the
command's own stderr on failure. Pass `user=\"0\"` to run as root."""
cmd = ["docker", "exec"]
if user:
cmd += ["-u", user]
cmd += [container, *argv]
result = run_docker(cmd)
if result.returncode != 0:
die(
f"docker exec in {container} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def docker_cp(src: str, dest: str) -> None:
"""Run `docker cp`, dying with the command's own stderr on failure."""
result = run_docker(["docker", "cp", src, dest])
if result.returncode != 0:
die(f"docker cp {src} -> {dest} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
_SLUG_RE = re.compile(r"[^a-z0-9]+")
@@ -108,15 +133,40 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
`dockerfile` is an optional path (relative to `context`, or
absolute) for callers that need to build from a non-default
Dockerfile in the same context — e.g. `Dockerfile.git-gate`."""
Dockerfile in the same context — e.g. `Dockerfile.git-gate`.
Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
`--no-cache`. The npm/curl installers some provider Dockerfiles
shell out to can silently no-op on a transient network failure —
e.g. an `optionalDependencies` fetch for a platform-native binary —
and Docker will then cache that broken layer indefinitely."""
info(f"building image {ref} from {context} (layer cache keeps repeat builds fast)")
args = ["docker", "build", "-t", ref]
if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
args.append("--no-cache")
if dockerfile:
args.extend(["-f", dockerfile])
args.append(context)
subprocess.run(args, check=True)
def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
"""Run `argv` inside a throwaway container of a freshly built agent
image and die loudly if it fails, instead of shipping an image
whose CLI only breaks at first real use. No-op when the provider
hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
if not argv:
return
result = run_docker(["docker", "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]])
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(
f"agent image {image!r} failed its post-build smoke test "
f"({' '.join(argv)}): {detail}\n"
f"Try rebuilding from scratch: bot-bottle start --no-cache"
)
# def build_image_with_cwd(
# derived: str,
# base: str,
@@ -110,3 +110,7 @@ class FirecrackerBottleBackend(
def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
return plan.agent_supervise_url
def ensure_orchestrator(self) -> str:
from . import infra_vm
return infra_vm.ensure_running().control_plane_url
+12 -5
View File
@@ -105,10 +105,9 @@ class FirecrackerBottle(Bottle):
# root-owned and unreadable by node, which breaks Node's
# process.cwd(), the shell-snapshot machinery, and `/doctor`.
# Use `env --chdir` rather than a `sh -c 'cd … && exec "$@"'`
# wrapper: ssh space-joins everything after the host into one
# string for the guest shell, so a quoted script + $@ would be
# re-split and mangled (exec'ing the $0 placeholder). All-simple
# words survive that join.
# wrapper: it keeps the guest command a flat argv that `agent_argv`
# can quote token-by-token for the ssh→guest-shell round trip,
# avoiding a fragile nested-quoting `"$@"` script.
workdir = self.agent_workdir or _HOME_FOR["node"]
remote = ["runuser", "-u", "node", "--",
"env", f"--chdir={workdir}",
@@ -117,7 +116,15 @@ class FirecrackerBottle(Bottle):
return remote
def agent_argv(self, argv: list[str], *, tty: bool = True) -> list[str]:
return [*self._ssh(tty=tty), "--", *self._agent_remote_argv(argv)]
# ssh space-joins everything after the host into one line the guest
# shell re-parses, so pre-quote each remote token for that shell.
# Simple words are unchanged (existing behaviour); an arg containing
# spaces — e.g. codex's `read_prompt_file` positional "Read and follow
# the instructions in <path>." — is quoted so it survives as ONE
# argument instead of being re-split (which made codex parse "and" as
# a subcommand).
remote = self._agent_remote_argv(argv)
return [*self._ssh(tty=tty), "--", *(shlex.quote(t) for t in remote)]
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
agent_argv = self.agent_argv(argv, tty=tty)
@@ -18,6 +18,10 @@ class FirecrackerBottlePlan(BottlePlan):
agent_proxy_url: str = ""
agent_git_gate_url: str = ""
agent_supervise_url: str = ""
# Per-bottle identity token the agent presents on every attributed request
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
@property
def container_name(self) -> str:
@@ -0,0 +1,109 @@
"""Consolidated bottle launch sequence for the Firecracker backend
(PRD 0070, Stage B).
The shared gateway + orchestrator control plane run in a single persistent
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
the gateway's egress / supervise / git-http ports at the infra VM via a
PREROUTING DNAT on their own host-side TAP IP (see
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
over HTTP at the infra VM's guest IP.
Attribution is by the agent VM's guest IP, unspoofable by construction: the
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
expected VM can source-IP that address.
Sequence:
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
prior launcher may already have booted it);
2. register the bottle by its guest IP (attribution key) → bottle id +
identity token;
3. provision its git-gate repos/creds into the gateway VM (over SSH);
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
The TAP slot allocation, rootfs build, and VM boot are the caller's job.
"""
from __future__ import annotations
from dataclasses import dataclass
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.lifecycle import (
OrchestratorStartError, # re-exported so callers can catch it
)
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from . import infra_vm
class ConsolidatedLaunchError(RuntimeError):
"""The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
class LaunchContext:
"""What the Firecracker launch needs from the consolidated sequence."""
bottle_id: str
identity_token: str
source_ip: str # the VM's guest IP — the attribution key
gateway_ca_pem: str # the shared gateway CA the provisioner installs
orchestrator_url: str
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
guest_ip: str,
image_ref: str = "",
tokens: dict[str, str] | None = None,
) -> LaunchContext:
"""Ensure the infra VM is up, register the bottle by its guest IP, and
provision its git-gate state into the gateway VM. Returns the context the
agent-VM launch needs. Raises on failure — the caller tears down."""
infra = infra_vm.ensure_running()
url = infra.control_plane_url
client = OrchestratorClient(url)
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
guest_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(
infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
# The shared gateway CA every agent on this host trusts for TLS
# interception — fetched from the infra VM over SSH.
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=guest_ip,
gateway_ca_pem=infra.gateway_ca_pem(),
orchestrator_url=url,
)
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
NOT stop the infra VM — it's a persistent per-host singleton shared by
every bottle."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
__all__ = [
"LaunchContext",
"launch_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
]
@@ -78,20 +78,32 @@ def _config(
vcpus: int,
mem_mib: int,
guest_mac: str,
data_drive: Path | None = None,
) -> dict[str, object]:
drives: list[dict[str, object]] = [
{
"drive_id": "rootfs",
"path_on_host": str(rootfs),
"is_root_device": True,
"is_read_only": False,
}
]
# A second virtio-block device (guest /dev/vdb) — the infra VM's
# persistent registry "volume", a host-side ext4 file that outlives the
# ephemeral rootfs across VM restarts.
if data_drive is not None:
drives.append({
"drive_id": "data",
"path_on_host": str(data_drive),
"is_root_device": False,
"is_read_only": False,
})
return {
"boot-source": {
"kernel_image_path": str(util.kernel_path()),
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
},
"drives": [
{
"drive_id": "rootfs",
"path_on_host": str(rootfs),
"is_root_device": True,
"is_read_only": False,
}
],
"drives": drives,
"network-interfaces": [
{
"iface_id": "eth0",
@@ -118,9 +130,16 @@ def boot(
vcpus: int = 2,
mem_mib: int = 2048,
guest_mac: str = "06:00:AC:10:00:02",
detached: bool = False,
data_drive: Path | None = None,
) -> VmHandle:
"""Write the config and launch the VMM. Returns once the process is
spawned; callers wait for SSH readiness separately."""
spawned; callers wait for SSH readiness separately.
`detached` starts the VMM in its own session (`start_new_session`) so it
survives the launcher exiting — used for the persistent per-host infra
VM, which must outlive the short-lived `start` process (agent VMs stay
attached and are torn down with the launcher)."""
run_dir.mkdir(parents=True, exist_ok=True)
config_path = run_dir / "config.json"
console_log = run_dir / "console.log"
@@ -128,6 +147,7 @@ def boot(
_config(
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
data_drive=data_drive,
),
indent=2,
))
@@ -137,6 +157,7 @@ def boot(
process = subprocess.Popen(
["firecracker", "--no-api", "--config-file", str(config_path)],
stdout=log_fh, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
start_new_session=detached,
)
return VmHandle(process=process, guest_ip=guest_ip, console_log=console_log)
@@ -0,0 +1,224 @@
"""Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
Agent Dockerfiles build **inside the persistent per-host infra VM**
(`infra_vm.py`), which carries buildah (rootless, daemonless): no host Docker
daemon, no root-equivalent `docker` group. The build runs over SSH against the
infra VM and its rootfs streams back to the host, where the existing
`mke2fs -d` path (`util.build_rootfs_ext4`) turns it into a bootable ext4.
Building in the infra VM — rather than a throwaway builder VM — means there is
one buildah image (`bot-bottle-infra`) and no contention for the orchestrator
TAP. Tradeoff: an untrusted Dockerfile's `RUN` steps share the VM with the
control plane + gateway (buildah `--isolation chroot` isn't a hard boundary) —
the accepted single-VM blast-radius tradeoff, re-splittable into a disposable
builder (booted from this same image on its own TAP) later.
"""
from __future__ import annotations
import fcntl
import hashlib
import os
import shutil
import subprocess
from contextlib import contextmanager
from pathlib import Path
from typing import Generator
from ...log import die, info
from . import infra_vm, util
# vfs + chroot: buildah works as root in the microVM (no fuse-overlayfs /
# overlay module / subuid maps). `--isolation` is a build/run-only flag;
# `from`/`mount` take just the store.
_BUILD_FLAGS = "--isolation chroot --storage-driver vfs"
_STORE_FLAG = "--storage-driver vfs"
_BUILD_TIMEOUT_SECONDS = 900.0
def _dockerfile_hash(dockerfile: Path) -> str:
"""Cache key: the Dockerfile's content. The shipped agent Dockerfiles
COPY nothing from the build context (see .dockerignore), so their content
fully determines the image; a Dockerfile that adds COPY will want the
context folded in here too."""
return hashlib.sha256(dockerfile.read_bytes()).hexdigest()[:16]
def build_agent_rootfs_dir(
dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
) -> Path:
"""Build `dockerfile` in the infra VM (buildah, no host docker), export its
rootfs, inject the guest boot bits, and return the cached base dir — the
same shape `util.build_rootfs_ext4` consumes. Cached by Dockerfile content,
so a repeat launch skips the rebuild.
`smoke_test` (the provider's declared argv, e.g. `("claude","--version")`)
is run in the freshly built image before export, catching an npm
silent-failure image at build time rather than at first agent use."""
digest = _dockerfile_hash(dockerfile)
base = util.cache_dir() / "rootfs" / f"agent-{digest}"
if (base / ".bb-ready").is_file():
info(f"using cached agent rootfs {base.name}")
return base
# Serialize builds: the infra VM's buildah store + this cache dir are
# shared, so concurrent `start`s must not build into them at once. The
# lock covers the cache lookup + build + atomic publish; the ready
# fast-path above takes no lock.
with _build_lock():
if (base / ".bb-ready").is_file(): # another build finished while we waited
info(f"using cached agent rootfs {base.name}")
return base
# Build into a temp dir and publish by atomic rename, so a partial
# build is never visible as `agent-<digest>`.
staging = util.cache_dir() / "rootfs" / f".building-{digest}"
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True)
info(f"building agent image {image_tag!r} in the infra VM")
_build_in_infra(dockerfile, staging, smoke_test, digest)
util.inject_guest_boot(staging)
(staging / ".bb-ready").write_text("ok\n")
shutil.rmtree(base, ignore_errors=True)
os.rename(staging, base)
return base
@contextmanager
def _build_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing agent-image builds (shared infra
buildah store + cache dir). flock auto-releases on a crash."""
lock_path = util.cache_dir() / "rootfs" / ".build.lock"
lock_path.parent.mkdir(parents=True, exist_ok=True)
handle = open(lock_path, "w", encoding="utf-8")
try:
fcntl.flock(handle, fcntl.LOCK_EX)
yield
finally:
handle.close()
def _build_in_infra(
dockerfile: Path, base: Path, smoke_test: tuple[str, ...], digest: str,
) -> None:
"""Ensure the infra VM is up, `buildah build` the Dockerfile in it, smoke
test the image, and stream its rootfs into `base`. The infra VM persists;
only the per-build container/image/context are cleaned up."""
infra = infra_vm.ensure_running()
key, ip = infra.private_key, infra.guest_ip
tag = f"bot-bottle-agent-build-{digest}"
ctx = f"/tmp/agent-build-{digest}"
smoke_ctr, export_ctr = f"{tag}-smoke", f"{tag}-export"
def _cleanup() -> None:
# Remove only THIS build's working containers/image/context — never
# `buildah rm -a`, which would nuke a concurrent build's container.
_ssh(key, ip,
f"buildah rm {smoke_ctr} {export_ctr} >/dev/null 2>&1; "
f"buildah rmi {_STORE_FLAG} {tag} >/dev/null 2>&1; rm -rf {ctx}",
timeout=60)
_cleanup() # clear leftovers from a crashed prior build of this digest
try:
prep = _ssh(key, ip, f"mkdir -p {ctx}/ctx")
if prep.returncode != 0:
die(f"preparing build dir in the infra VM failed: {prep.stderr.strip()}")
_send_dockerfile(key, ip, dockerfile, ctx)
_buildah_build(key, ip, ctx, tag)
_smoke_test(key, ip, tag, smoke_ctr, smoke_test)
_stream_rootfs(key, ip, tag, export_ctr, base)
finally:
_cleanup()
def _ssh(private_key: Path, guest_ip: str, script: str,
*, timeout: float = 60.0) -> subprocess.CompletedProcess[str]:
return subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [script],
capture_output=True, text=True, timeout=timeout, check=False,
)
def _ssh_streamed(private_key: Path, guest_ip: str, script: str,
*, timeout: float) -> int:
"""Run an SSH command letting the remote's stdout/stderr flow straight to
ours (no capture), for long chatty steps where live progress beats a
silent wait. Returns the exit code."""
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [script],
timeout=timeout, check=False,
)
return proc.returncode
def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path, ctx: str) -> None:
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [f"cat > {ctx}/Dockerfile"],
input=dockerfile.read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
die(f"sending Dockerfile to the infra VM failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def _buildah_build(private_key: Path, guest_ip: str, ctx: str, tag: str) -> None:
# Stream buildah's step-by-step output straight to our stderr (like the
# docker backend's `docker build`), so a long first build (base pull +
# apt/npm installs) shows live progress instead of a silent wait. The
# remote stderr is where buildah writes its `STEP i/n` lines.
info(f"buildah build {tag} in the infra VM (streaming output)")
rc = _ssh_streamed(
private_key, guest_ip,
f"buildah build {_BUILD_FLAGS} -t {tag} -f {ctx}/Dockerfile {ctx}/ctx",
timeout=_BUILD_TIMEOUT_SECONDS,
)
if rc != 0:
die(f"buildah build in the infra VM failed (exit {rc}); "
"see the build output above.")
def _smoke_test(private_key: Path, guest_ip: str, tag: str, ctr: str,
argv: tuple[str, ...]) -> None:
"""Run the provider's smoke argv inside the freshly built image
(`buildah run`, which uses the image's own PATH), failing the build
loudly if the CLI is a broken stub. No-op without a declared test. Uses a
named working container (`ctr`) so cleanup is scoped to this build."""
if not argv:
return
cmd = (
f"set -e; buildah from {_STORE_FLAG} --name {ctr} {tag} >/dev/null; "
f"buildah run {_BUILD_FLAGS} {ctr} -- {' '.join(argv)}; rc=$?; "
f"buildah rm {ctr} >/dev/null 2>&1 || true; exit $rc"
)
result = _ssh(private_key, guest_ip, cmd, timeout=120)
if result.returncode != 0:
detail = (result.stdout + result.stderr).strip().splitlines()[-10:]
die(f"agent image failed its post-build smoke test "
f"({' '.join(argv)}):\n" + "\n".join(detail))
def _stream_rootfs(private_key: Path, guest_ip: str, tag: str, ctr: str, base: Path) -> None:
"""`buildah mount` the built image in the infra VM and pipe its rootfs tar
straight into `base` on the host (extracted as the non-root host user, so
uid 0 isn't preserved — the guest init restores /root ownership). Uses a
named working container so cleanup is scoped to this build."""
export = (
f"set -e; buildah from {_STORE_FLAG} --name {ctr} {tag} >/dev/null; "
f"mnt=$(buildah mount {_STORE_FLAG} {ctr}); "
f"tar -C \"$mnt\" -cf - ."
)
ssh_proc = subprocess.Popen(
util.ssh_base_argv(private_key, guest_ip) + [export],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
)
assert ssh_proc.stdout is not None
untar = subprocess.run(
["tar", "-x", "-C", str(base)], stdin=ssh_proc.stdout, check=False,
)
ssh_proc.stdout.close()
ssh_err = (ssh_proc.stderr.read().decode(errors="replace")
if ssh_proc.stderr else "")
rc = ssh_proc.wait()
if rc != 0 or untar.returncode != 0:
die(f"exporting the built rootfs from the infra VM failed: "
f"{ssh_err.strip() or '<no stderr>'}")
+419
View File
@@ -0,0 +1,419 @@
"""The per-host infra VM for the Firecracker backend (PRD 0070 Stage B).
A single persistent microVM that runs the orchestrator **control plane** (and,
in a following step, the gateway **data plane**) the trusted per-host service
the docker backend runs as containers. It boots on the NAT'd orchestrator link
(`netpool.orch_slot()`): the host CLI reaches its control plane over HTTP at the
guest IP, and agent VMs reach its gateway ports over VM-to-VM routing.
Build-from-source (the default while the design churns): the rootfs is exported
from the locally built orchestrator image, which bakes the stdlib-only
control-plane source. A pull-from-registry mode (Gitea's OCI registry) becomes
the default later.
SSH is left enabled for debugging; the control plane is the load-bearing
surface.
"""
from __future__ import annotations
import fcntl
import hashlib
import os
import shlex
import signal
import stat
import subprocess
import time
import urllib.error
import urllib.request
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Generator
from ...log import die, info
from ..docker import util as docker_mod
from ..docker.gateway_provision import GatewayProvisionError
from . import firecracker_vm, netpool, util
# The single infra-VM image: gateway data plane + baked control-plane source
# (Dockerfile.infra FROM the gateway image). Built from source by default;
# a pull-from-registry mode lands later.
_INFRA_IMAGE = "bot-bottle-infra:latest"
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
_REPO_ROOT = Path(__file__).resolve().parents[3]
CONTROL_PLANE_PORT = 8099
# Gateway data-plane ports (agent-facing): egress proxy, supervise MCP,
# git-http. Reached by agent VMs over VM-to-VM routing (added next).
EGRESS_PORT = 9099
SUPERVISE_PORT = 9100
GIT_HTTP_PORT = 9420
# mitmproxy writes its CA here a beat after start; agents install it to trust
# the gateway's TLS interception.
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
# The infra VM makes direct upstream connections (gateway egress, and buildah
# during builds), and the kernel `ip=` cmdline sets no resolver. Public for
# now; routing DNS through a filtered path is a later refinement.
_INFRA_RESOLVER = "1.1.1.1"
_HEALTH_TIMEOUT_SECONDS = 45.0
_HEALTH_POLL_SECONDS = 0.5
_CA_TIMEOUT_SECONDS = 30.0
@dataclass
class InfraVm:
"""A handle to the per-host infra VM: its guest IP and the stable SSH key
used to fetch the gateway CA / provision git-gate. `vm` is the live VMM
handle when this process booted it, and None when adopting a singleton a
prior launcher started (teardown then goes through the PID file)."""
guest_ip: str
private_key: Path
vm: firecracker_vm.VmHandle | None = None
@property
def control_plane_url(self) -> str:
return f"http://{self.guest_ip}:{CONTROL_PLANE_PORT}"
def terminate(self) -> None:
"""Stop the infra VM — via the live handle if we booted it, else the
PID file (adopting-process case)."""
if self.vm is not None:
self.vm.terminate()
else:
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def gateway_ca_pem(self, *, timeout: float = _CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) that agents install to trust its
TLS interception. Generated a moment after boot, so this polls over
SSH until it appears (mirrors DockerGateway.ca_cert_pem)."""
deadline = time.monotonic() + timeout
while True:
proc = subprocess.run(
util.ssh_base_argv(self.private_key, self.guest_ip)
+ [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True, timeout=15, check=False,
)
if proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout:
return proc.stdout
if time.monotonic() >= deadline:
die(f"gateway CA not available after {timeout:g}s: "
f"{proc.stderr.strip() or 'empty'}")
time.sleep(_HEALTH_POLL_SECONDS)
def ensure_built() -> None:
"""Build the infra image from source (bootstrap via host docker). The
infra image `COPY --from`s the orchestrator image and is `FROM` the
gateway image, so both must exist first. A pull-from-registry mode
replaces this later."""
docker_mod.build_image(
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
docker_mod.build_image(
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra")
def build_infra_rootfs_dir() -> Path:
"""The infra VM's base rootfs: the infra image prepared with the
control-plane + gateway init as PID 1. The init's content is folded into
the cache key so an init change rebuilds the rootfs (the base image digest
alone wouldn't catch it)."""
init = _infra_init()
tag = hashlib.sha256(init.encode()).hexdigest()[:8]
return util.build_base_rootfs_dir(
_INFRA_IMAGE, variant=f"-infra-{tag}", init_script=init,
)
def ensure_running() -> InfraVm:
"""Idempotent per-host singleton. Adopt the infra VM if its control plane
is already healthy (a prior launcher booted it it outlives short-lived
`start` processes); otherwise clear any stale VM and boot a fresh one.
Returns a handle usable for CA fetch / git-gate provisioning.
Concurrency-safe: the cold stop/build/boot path is serialized by a host
flock, so two simultaneous first launches don't both boot on the same
rootfs/PID. The healthy fast-path takes no lock."""
slot = netpool.orch_slot()
url = f"http://{slot.guest_ip}:{CONTROL_PLANE_PORT}"
key = _infra_dir() / "id_ed25519"
if key.exists() and _health_ok(url):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
with _singleton_lock():
# Re-check under the lock: another launcher may have booted it while
# we waited for the lock (double-checked, so we adopt not re-boot).
if key.exists() and _health_ok(url):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
stop() # clear a stale/hung VM holding the link before booting fresh
ensure_built()
infra = boot()
wait_for_health(infra)
return infra
@contextmanager
def _singleton_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing the infra VM's cold create path
(`stop`/`ensure_built`/`boot`). flock auto-releases if the launcher
crashes, so the lock is never leaked."""
lock_path = _infra_dir() / "singleton.lock"
handle = open(lock_path, "w", encoding="utf-8")
try:
fcntl.flock(handle, fcntl.LOCK_EX)
yield
finally:
handle.close()
def stop() -> None:
"""Stop the infra VM singleton (idempotent — absent is success)."""
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def boot() -> InfraVm:
"""Boot the infra VM (detached, so it outlives the launcher) on the
orchestrator link, recording its PID. Prefer `ensure_running`."""
slot = netpool.orch_slot()
if not netpool.tap_present(slot.iface):
die(f"orchestrator link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
base = build_infra_rootfs_dir()
run_dir = _infra_dir()
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, rootfs, slack_mib=8192)
private_key, pubkey = _stable_keypair()
info(f"booting infra VM on {slot.iface} (guest {slot.guest_ip})")
vm = firecracker_vm.boot(
name="bot-bottle-infra", rootfs=rootfs, tap=slot.iface,
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
run_dir=run_dir, mem_mib=4096, detached=True,
data_drive=_ensure_registry_volume(),
)
_pid_file().write_text(str(vm.process.pid))
return InfraVm(guest_ip=slot.guest_ip, private_key=private_key, vm=vm)
def _infra_dir() -> Path:
d = util.cache_dir() / "infra"
d.mkdir(parents=True, exist_ok=True)
return d
def _pid_file() -> Path:
return _infra_dir() / "vm.pid"
# The registry "volume": a host-side ext4 file attached to the infra VM as a
# second virtio-block device (guest /dev/vdb), mounted at the control plane's
# DB dir. It outlives the ephemeral rootfs, so the bottle registry survives an
# infra-VM restart — the firecracker analogue of a docker volume. It is a
# plain ext4 file: `sudo mount -o loop <path>` on the host (with the VM
# stopped) to inspect bot-bottle.db directly.
_REGISTRY_SIZE = "512M"
def registry_volume_path() -> Path:
return _infra_dir() / "registry.ext4"
def _ensure_registry_volume() -> Path:
"""Create the empty ext4 registry volume on first use; reuse it after."""
vol = registry_volume_path()
if vol.exists():
return vol
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
proc = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
vol.unlink(missing_ok=True)
die(f"creating registry volume failed: {proc.stderr.strip()}")
return vol
def _stable_keypair() -> tuple[Path, str]:
"""The infra VM's SSH keypair — generated once and reused, so any later
launcher can SSH in (fetch CA / provision) even though a different process
booted the VM. The pubkey is re-injected on every boot via the cmdline."""
d = _infra_dir()
key, pub = d / "id_ed25519", d / "id_ed25519.pub"
if key.exists() and pub.exists():
return key, pub.read_text().strip()
key.unlink(missing_ok=True)
pub.unlink(missing_ok=True)
subprocess.run(
["ssh-keygen", "-t", "ed25519", "-N", "", "-q", "-f", str(key),
"-C", "bot-bottle-infra"],
check=True,
)
return key, pub.read_text().strip()
def _kill_pidfile() -> None:
"""SIGTERM (then SIGKILL) the recorded infra VMM, if it's still ours.
Guards against a recycled PID by checking the process is firecracker."""
try:
pid = int(_pid_file().read_text().strip())
except (OSError, ValueError):
return
try:
comm = Path(f"/proc/{pid}/comm").read_text().strip()
except OSError:
return # already gone
if comm != "firecracker":
return # PID recycled by an unrelated process
try:
os.kill(pid, signal.SIGTERM)
for _ in range(50):
if not Path(f"/proc/{pid}").exists():
return
time.sleep(0.1)
os.kill(pid, signal.SIGKILL)
except OSError:
pass
def _health_ok(url: str) -> bool:
try:
with urllib.request.urlopen(f"{url}/health", timeout=1.0) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
class SshGatewayTransport:
"""`GatewayTransport` for the gateway running in the infra VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
# Preserve the source mode (docker cp does): the access-hook is staged
# 0700 and git-http execs it directly — a plain `cat >` would land it
# 0644 and the exec fails with EACCES; keys stay 0600.
mode = stat.S_IMODE(os.stat(src).st_mode)
q = shlex.quote(dest)
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip)
+ [f"cat > {q} && chmod {mode:o} {q}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def gateway_transport() -> SshGatewayTransport:
"""git-gate provisioning transport for the gateway in the infra VM, built
from the stable key + the orchestrator link's guest IP. Needs no live VM
handle, so teardown can use it too."""
return SshGatewayTransport(
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
def wait_for_health(
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
) -> None:
"""Poll the control plane's /health until it answers 200 or the deadline
passes. Dies (with the console tail) if the VMM exits early."""
url = f"{infra.control_plane_url}/health"
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if infra.vm is not None and not infra.vm.is_alive():
die(f"infra VM exited during boot (rc={infra.vm.process.returncode}).\n"
f"{firecracker_vm._console_tail(infra.vm.console_log)}")
try:
with urllib.request.urlopen(url, timeout=1.0) as resp:
if resp.status == 200:
info(f"infra control plane healthy at {infra.control_plane_url}")
return
except (urllib.error.URLError, TimeoutError, OSError):
pass
time.sleep(_HEALTH_POLL_SECONDS)
tail = (firecracker_vm._console_tail(infra.vm.console_log)
if infra.vm is not None else "")
die(f"infra control plane at {url} did not become healthy within "
f"{timeout:.0f}s.\n{tail}")
def _infra_init() -> str:
"""PID-1 init for the infra VM: mount the pseudo-filesystems, wire a
resolver, start dropbear (debug SSH), then launch the control plane and
the gateway data plane (multi-tenant against the local control plane)."""
return f"""#!/bin/sh
# bot-bottle Firecracker infra VM init (PID 1).
mount -t proc proc /proc 2>/dev/null
mount -t sysfs sys /sys 2>/dev/null
mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Export a real PATH: a bare-init shell resolves its own execs via a
# built-in default path, but that isn't in the *environment*, so
# gateway_init's subprocess daemons (spawned as `python3 ...`) would
# inherit no PATH and fail to find python3. Export it for all children.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Direct upstream resolver (control-plane / gateway egress + buildah).
printf 'nameserver {_INFRA_RESOLVER}\\n' > /etc/resolv.conf 2>/dev/null
# Debug SSH: install the per-boot pubkey from the kernel cmdline.
KEY=$(sed -n 's/.*bb_pubkey=\\([^ ]*\\).*/\\1/p' /proc/cmdline | base64 -d 2>/dev/null)
if [ -n "$KEY" ]; then
mkdir -p /root/.ssh
printf '%s\\n' "$KEY" > /root/.ssh/authorized_keys
chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys
fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run /var/lib/bot-bottle
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted
# at the control plane's DB dir, so bot-bottle.db survives infra-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
/bb-dropbear -R -E -p 22 &
# Control plane. Source is baked at /app; the package is stdlib-only.
cd /app
BOT_BOTTLE_ROOT=/var/lib/bot-bottle python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {CONTROL_PLANE_PORT} --broker stub &
# Gateway data plane, multi-tenant: each request resolves source-IP ->
# policy against the local control plane. The VM backend reaches git over
# git-http (9420), so the git:// daemon (git-gate, needs a per-bottle
# entrypoint the consolidated model doesn't use) is left out.
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{CONTROL_PLANE_PORT} \\
SUPERVISE_DB_PATH=/var/lib/bot-bottle/db/bot-bottle.db \\
python3 /app/gateway_init.py &
# Reap as PID 1; children are backgrounded, so `wait` blocks.
while : ; do wait ; done
"""
+243 -20
View File
@@ -1,25 +1,63 @@
"""Launch flow for the Firecracker backend — temporarily disabled (#385).
"""Launch flow for the Firecracker backend (PRD 0070, consolidated).
The firecracker backend launched a per-bottle companion container (the
egress / git-gate / supervise data plane) alongside each microVM. That
per-bottle-companion architecture was removed in the companion-container removal;
firecracker's replacement — the consolidated per-host gateway — lands in
its own cutover (#354).
Per bottle:
1. build the agent image (docker), export it to a cached ext4 rootfs;
2. ensure the per-host orchestrator + shared gateway are up;
3. claim a free TAP pool slot (rootless flock);
4. register the bottle on the orchestrator by the VM's guest IP (the
attribution key) and provision its git-gate state into the gateway;
5. boot the microVM on that TAP; wait for SSH;
6. provision (shared gateway CA, prompt, skills, workspace, git, supervise)
over SSH.
Until that lands, launching a firecracker bottle fails closed rather than
silently running the removed path. `prepare` / `status` / cleanup still
work, so `backend status --backend=firecracker` and orphan cleanup are
unaffected.
The per-bottle Docker sidecar bundle is gone. The shared gateway handles
egress / git-gate / supervise for every VM; Docker's PREROUTING DNAT routes
the VMs' traffic to it, and the nft table's `ct status dnat accept` rule
in the forward chain lets it pass. The VM still sends to `host_tap_ip:PORT`
the address its world is, by nft design, limited to.
Isolation is enforced by the operator-provisioned nft table (checked
fail-closed in preflight): a VM reaches only the sidecar (DNAT'd from
the host TAP IP) and nothing else.
"""
from __future__ import annotations
from contextlib import contextmanager
import dataclasses
import os
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...log import die
from ...agent_provider import runtime_for
from ...bottle_state import (
egress_state_dir,
git_gate_state_dir,
read_committed_image,
)
from ...egress import (
egress_agent_env_entries,
egress_resolve_token_values,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from ...supervise import SUPERVISE_PORT
from ..docker import util as docker_mod
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
from .consolidated_launch import (
launch_consolidated,
teardown_consolidated,
)
_GIT_HTTP_PORT = 9420
@contextmanager
@@ -28,12 +66,197 @@ def launch(
*,
provision: Callable[[FirecrackerBottlePlan, "FirecrackerBottle"], str | None],
) -> Generator[FirecrackerBottle, None, None]:
"""Fail closed: the firecracker backend is disabled while its
consolidated (gateway-backed) launch is built in #354."""
del plan, provision
die(
"the firecracker backend is temporarily disabled during the "
"companion-container removal (#385); its consolidated relaunch "
"lands in #354. Use --backend=docker for now."
"""Build, launch, and provision a Firecracker bottle via the consolidated
orchestrator. Teardown on exit."""
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
def teardown() -> None:
teardown_exc: BaseException | None = None
try:
stack.close()
except BaseException as exc: # noqa: W0718 - teardown must continue
teardown_exc = exc
warn(f"firecracker teardown failed: {exc!r}")
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
if teardown_exc is not None:
raise teardown_exc
try:
# Step 1: agent rootfs. Built from the Dockerfile inside a Firecracker
# builder VM (buildah, no host docker); a committed snapshot is reused
# when present. Returns the base dir the per-bottle ext4 is made from.
plan, agent_base = _build_agent_base(plan)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
# Step 3: claim a TAP slot; the flock is held until teardown.
slot, lock = netpool.allocate(plan.slug)
stack.callback(lock.close)
info(f"firecracker slot {slot.iface}: host={slot.host_ip} "
f"guest={slot.guest_ip}")
# Step 4: register on the orchestrator + provision this bottle's
# git-gate state into the shared gateway. The per-bottle egress tokens
# are resolved from the host env now and handed to the orchestrator
# (in memory) for the gateway to inject — the agent never sees them.
# Attribution is by the VM's guest IP (unspoofable via /31 TAP + nft).
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan,
guest_ip=slot.guest_ip,
image_ref=plan.image,
tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
)
# Step 5: install the SHARED gateway CA (replaces the per-bottle CA).
# Write it to a stable host path so the provisioner can copy it over SSH.
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(ctx.gateway_ca_pem)
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=ca_file,
mitmproxy_ca_cert_only_host_path=ca_file,
)
# Point the agent's git-gate insteadOf rewrites and supervise MCP URL
# at the shared gateway (reached at the slot's host TAP IP — the VM
# sends there and Docker DNAT routes to the gateway container).
git_gate_url = (
f"http://{slot.host_ip}:{_GIT_HTTP_PORT}" if git_gate_plan.upstreams else ""
)
supervise_url = (
f"http://{slot.host_ip}:{SUPERVISE_PORT}/"
if plan.supervise_plan is not None else ""
)
plan = dataclasses.replace(
plan,
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
identity_token=ctx.identity_token,
# Deliver the identity token as egress proxy credentials — clients
# honor `HTTPS_PROXY=http://id:token@gw` without app changes; the
# gateway reads Proxy-Authorization, validates the (source_ip,
# token) pair, and strips it before upstream.
agent_proxy_url=(
f"http://bottle:{ctx.identity_token}"
f"@{slot.host_ip}:{EGRESS_PORT}"
),
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
# Step 6: build the per-bottle rootfs + SSH key, then boot.
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(agent_base, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
vm = firecracker_vm.boot(
name=plan.container_name,
rootfs=rootfs,
tap=slot.iface,
guest_ip=slot.guest_ip,
host_ip=slot.host_ip,
pubkey=pubkey,
run_dir=run_dir,
)
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
# Authoritative fail-closed egress-boundary check, before the agent
# runs: prove the VM cannot reach the host directly.
isolation_probe.verify_isolation(private_key, slot.guest_ip)
bottle = FirecrackerBottle(
plan.container_name,
private_key=private_key,
guest_ip=slot.guest_ip,
guest_env=_agent_guest_env(plan, slot.host_ip),
agent_command=plan.agent_command,
agent_prompt_mode=plan.agent_prompt_mode,
agent_provider_template=plan.agent_provider_template,
terminal_title=(
f"{plan.spec.label} ({plan.spec.agent_name})"
if plan.spec.label else plan.spec.agent_name
),
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
yield bottle
finally:
teardown()
def _build_agent_base(
plan: FirecrackerBottlePlan,
) -> tuple[FirecrackerBottlePlan, Path]:
"""Produce the agent's base rootfs dir. Primary path: build the Dockerfile
inside a Firecracker builder VM (buildah, no host docker), smoke-testing
the image before export. A committed snapshot (freeze/migrate) is still
exported via the host docker path until that is ported too."""
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
plan = dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
)
return plan, util.build_base_rootfs_dir(committed)
base = image_builder.build_agent_rootfs_dir(
Path(plan.dockerfile_path),
image_tag=plan.image,
smoke_test=runtime_for(plan.agent_provider_template).smoke_test,
)
yield # unreachable — `die` raises; keeps this a generator/contextmanager
return plan, base
# --- agent guest env -------------------------------------------------
def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str]:
"""Env injected into every agent/exec call over SSH. The VM has no
baked process env (it just runs init), so the proxy/CA/git/supervise
wiring is applied per-invocation."""
# Carries the identity token as proxy credentials (set in `launch`).
proxy_url = plan.agent_proxy_url or f"http://{host_ip}:{EGRESS_PORT}"
no_proxy = f"localhost,127.0.0.1,{host_ip}"
env: dict[str, str] = {
"HTTPS_PROXY": proxy_url, "HTTP_PROXY": proxy_url,
"https_proxy": proxy_url, "http_proxy": proxy_url,
"NO_PROXY": no_proxy, "no_proxy": no_proxy,
"NODE_EXTRA_CA_CERTS": AGENT_CA_PATH,
"SSL_CERT_FILE": AGENT_CA_BUNDLE,
"REQUESTS_CA_BUNDLE": AGENT_CA_BUNDLE,
}
if plan.agent_git_gate_url:
env["GIT_GATE_URL"] = plan.agent_git_gate_url
if plan.agent_supervise_url:
env["MCP_SUPERVISE_URL"] = plan.agent_supervise_url
for entry in egress_agent_env_entries(plan.egress_plan):
key, _, value = entry.partition("=")
env[key] = value
env.update(plan.agent_provision.guest_env)
# Forwarded (bare-name) env: resolve host values now, since the VM
# can't inherit them from a `docker run --env NAME`.
for name in plan.forwarded_env:
value = os.environ.get(name)
if value is not None:
env[name] = value
return env
@@ -15,3 +15,11 @@ BOT_BOTTLE_FC_POOL_SIZE=8
BOT_BOTTLE_FC_IP_BASE=10.243.0.0
BOT_BOTTLE_FC_IFACE_PREFIX=bbfc
BOT_BOTTLE_FC_NFT_TABLE=bot_bottle_fc
# The orchestrator/gateway VM's own TAP — a dedicated link OUTSIDE the
# bbfc* agent pool. Unlike agent VMs (which reach only their gateway),
# the orchestrator is trusted infra that needs real NAT'd internet
# egress: to FROM-pull + apt/npm during in-VM agent-image builds
# (buildah) and to forward agent egress upstream (Stage B gateway). Its
# /31 is the top of the IP_BASE /16 (host x.y.255.0, guest x.y.255.1),
# clear of the pool near the bottom of the block.
BOT_BOTTLE_FC_ORCH_IFACE=bborch0
+25
View File
@@ -79,6 +79,12 @@ def _cfg(key: str) -> str:
IFACE_PREFIX = _cfg("BOT_BOTTLE_FC_IFACE_PREFIX")
NFT_TABLE = _cfg("BOT_BOTTLE_FC_NFT_TABLE")
# The orchestrator/gateway VM's dedicated TAP — outside the bbfc* agent
# pool and, unlike it, NAT'd to the internet (see `orch_slot`). The
# orchestrator is trusted infra: it builds agent images in-VM (buildah
# needs to FROM-pull + apt/npm) and forwards agent egress upstream.
ORCH_IFACE = _cfg("BOT_BOTTLE_FC_ORCH_IFACE")
def pool_size() -> int:
return int(_cfg("BOT_BOTTLE_FC_POOL_SIZE"))
@@ -123,6 +129,25 @@ def all_slots() -> list[Slot]:
return [slot(i) for i in range(pool_size())]
def orch_slot() -> Slot:
"""The orchestrator/gateway VM's dedicated link — its own TAP
(`ORCH_IFACE`) on a /31 at the TOP of the IP_BASE /16 (host
x.y.255.0, guest x.y.255.1), well clear of the agent pool near the
bottom of the block. Unlike a pool `Slot`, this link is NAT'd out to
the internet by the setup (the orchestrator is trusted infra), so it
is deliberately *not* one of the isolated `bbfc*` slots.
`index` is -1 (sentinel: not a pool index)."""
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
host = base16 + 0xFF00
return Slot(
index=-1,
iface=ORCH_IFACE,
host_ip=str(ipaddress.IPv4Address(host)),
guest_ip=str(ipaddress.IPv4Address(host + 1)),
)
# --- fail-closed verification ---------------------------------------
def _run_ok(argv: list[str]) -> bool:
+17 -8
View File
@@ -159,15 +159,22 @@ def docker_image_id(ref: str) -> str:
return result.stdout.strip().replace("sha256:", "")[:16]
def build_base_rootfs_dir(image_ref: str) -> Path:
"""Export the agent image's filesystem and inject the guest init +
static dropbear. Cached by image digest the per-bottle bits
def build_base_rootfs_dir(
image_ref: str, *, variant: str = "", init_script: str | None = None,
) -> Path:
"""Export the image's filesystem and inject the guest init + static
dropbear. Cached by image digest the per-bottle bits
(authorized_keys, IP) are passed at boot via the kernel cmdline, so
this tree carries nothing bottle-specific and is safely shared.
`variant` suffixes the cache key so the same image can be prepared
with a different `init_script` (e.g. the infra VM boots the same
orchestrator image as the builder but runs the control plane as
PID 1, not the SSH-only agent init) without a cache collision.
Returns the prepared directory (read as the `mke2fs -d` source)."""
digest = docker_image_id(image_ref)
base = cache_dir() / "rootfs" / digest
base = cache_dir() / "rootfs" / f"{digest}{variant}"
ready = base / ".bb-ready"
if ready.is_file():
return base
@@ -200,17 +207,19 @@ def build_base_rootfs_dir(image_ref: str) -> Path:
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
_inject_guest_boot(base)
inject_guest_boot(base, init_script=init_script)
ready.write_text("ok\n")
return base
def _inject_guest_boot(rootfs: Path) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs."""
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs.
`init_script` defaults to the SSH-only agent init; the infra VM
passes its own (control plane + gateway) init."""
shutil.copy2(dropbear_path(), rootfs / "bb-dropbear")
os.chmod(rootfs / "bb-dropbear", 0o755)
init = rootfs / "bb-init"
init.write_text(_GUEST_INIT)
init.write_text(init_script or _GUEST_INIT)
os.chmod(init, 0o755)
+31 -1
View File
@@ -60,13 +60,21 @@ def dns_server() -> str:
def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
"""Build an OCI image with Apple's BuildKit-backed `container build`."""
"""Build an OCI image with Apple's BuildKit-backed `container build`.
Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
`--no-cache`. The npm/curl installers some provider Dockerfiles
shell out to can silently no-op on a transient network failure
e.g. an `optionalDependencies` fetch for a platform-native binary
and the builder will then cache that broken layer indefinitely."""
info(
f"building image {ref} from {context} with Apple Container "
"(layer cache keeps repeat builds fast)"
)
_ensure_builder_dns()
args = [_CONTAINER, "build", "-t", ref, "--dns", dns_server()]
if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
args.append("--no-cache")
if dockerfile:
# `container build` resolves -f relative to the current working
# directory, not the build context. Anchor a relative Dockerfile to
@@ -78,6 +86,28 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
subprocess.run(args, check=True)
def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
"""Run `argv` inside a throwaway container of a freshly built agent
image and die loudly if it fails, instead of shipping an image
whose CLI only breaks at first real use. No-op when the provider
hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
if not argv:
return
result = subprocess.run(
[_CONTAINER, "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(
f"agent image {image!r} failed its post-build smoke test "
f"({' '.join(argv)}): {detail}\n"
f"Try rebuilding from scratch: bot-bottle start --no-cache"
)
def commit_container(container_name: str, image_tag: str) -> None:
"""Snapshot a running Apple Container as a local image.
+16
View File
@@ -46,6 +46,17 @@ def cmd_start(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} start", add_help=True)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--cwd", action="store_true", help="copy host cwd into the running bottle")
parser.add_argument(
"--no-cache",
action="store_true",
help=(
"rebuild agent/sidecar images from scratch, bypassing the "
"build layer cache. Use when an image looks broken after a "
"dependency bump — e.g. an installer's optionalDependencies "
"fetch silently no-op'd on a transient failure and got baked "
"into a cached layer."
),
)
parser.add_argument(
"--backend",
choices=known_backend_names(),
@@ -97,6 +108,11 @@ def cmd_start(argv: list[str]) -> int:
args = parser.parse_args(argv)
dry_run = args.dry_run or os.environ.get("BOT_BOTTLE_DRY_RUN") == "1"
if args.no_cache or os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
# Read by build_image() in each backend's util module — set here
# so both the interactive and --headless paths pick it up without
# threading a no_cache field through every backend's plan dataclass.
os.environ["BOT_BOTTLE_NO_CACHE"] = "1"
manifest = ManifestIndex.resolve(USER_CWD)
backend_name: str | None = args.backend
+84 -87
View File
@@ -20,32 +20,19 @@ from datetime import datetime, timezone
from pathlib import Path
from ..paths import bot_bottle_root
from ..bottle_state import read_metadata
from ..backend.docker.egress_apply import (
EgressApplyError,
applicator as _docker_applicator,
)
from ..backend.macos_container.egress_apply import (
applicator as _macos_applicator,
)
from ..log import Die, error, info
from ..orchestrator.client import (
OrchestratorClient,
OrchestratorClientError,
discover_orchestrator_url,
)
from ..supervise import (
COMPONENT_FOR_TOOL,
AuditEntry,
Proposal,
Response,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
list_all_pending_proposals,
render_diff,
write_audit_entry,
write_response,
)
from ._common import PROG
@@ -60,30 +47,61 @@ _REPORT_ONLY_TOOLS: tuple[str, ...] = (TOOL_GITLEAKS_ALLOW, TOOL_EGRESS_TOKEN_AL
@dataclass(frozen=True)
class QueuedProposal:
"""A pending proposal from the supervise queue."""
"""A pending proposal from the supervise queue.
`label` is the operator-facing bottle name (the human slug the
orchestrator resolved from the registry); `proposal.bottle_slug` is the
opaque bottle_id every operator action is keyed by. Display uses `label`;
respond calls use `proposal.bottle_slug`."""
proposal: Proposal
label: str = ""
# Errors any remediation engine may raise. Caught by the TUI key
# handlers and surfaced in the status line so a failed apply keeps
# the proposal pending rather than crashing curses.
ApplyError = (EgressApplyError,)
# A failed operator action (orchestrator unreachable, bottle torn down,
# 409) is caught by the TUI key handlers and surfaced in the status line so
# the proposal stays pending rather than crashing curses.
ApplyError = (OrchestratorClientError,)
def apply_routes_change(slug: str, content: str) -> tuple[str, str]:
meta = read_metadata(slug)
backend = meta.backend if meta is not None else ""
if backend == "macos-container":
return _macos_applicator.apply_routes_change(slug, content)
return _docker_applicator.apply_routes_change(slug, content)
# The one per-host orchestrator, discovered lazily on first use. Every
# operator action — list, approve, reject — goes through its HTTP control
# plane (the orchestrator owns the single DB + live policy); there is no
# direct-DB path and no backend branching here.
_client_instance: OrchestratorClient | None = None
def _resolve_orchestrator_url() -> str:
"""URL of the running orchestrator control plane, starting one on demand.
Supervise is often the first thing an operator runs before any bottle
has booted the control plane. So when discovery finds nothing, bring up
the selected backend's orchestrator + gateway (idempotent) rather than
failing with "launch a bottle first"."""
try:
return discover_orchestrator_url()
except OrchestratorClientError:
from ..backend import get_bottle_backend
backend = get_bottle_backend()
info(f"no orchestrator control plane running; starting one ({backend.name})…")
return backend.ensure_orchestrator()
def _client() -> OrchestratorClient:
global _client_instance # noqa: PLW0603 — CLI-session singleton
if _client_instance is None:
_client_instance = OrchestratorClient(_resolve_orchestrator_url())
return _client_instance
def discover_pending() -> list[QueuedProposal]:
"""Collect pending proposals across bottles."""
"""Collect pending proposals across bottles from the orchestrator."""
out = [
QueuedProposal(proposal=proposal)
for proposal in list_all_pending_proposals()
QueuedProposal(
proposal=Proposal.from_dict(d),
label=str(d.get("bottle_label") or d.get("bottle_slug") or ""),
)
for d in _client().supervise_pending()
]
out.sort(key=lambda q: q.proposal.arrival_timestamp)
return out
@@ -91,8 +109,8 @@ def discover_pending() -> list[QueuedProposal]:
def _approval_status(qp: QueuedProposal, verb: str) -> str:
"""Status-line text after a successful approval."""
base = f"{verb} {qp.proposal.tool} for [{qp.proposal.bottle_slug}]"
return f"{base}; resume: ./cli.py resume {qp.proposal.bottle_slug}"
base = f"{verb} {qp.proposal.tool} for [{qp.label}]"
return f"{base}; resume: ./cli.py resume {qp.label}"
def _detail_lines(
@@ -103,7 +121,7 @@ def _detail_lines(
"""Return the detail-view body as (text, curses-attr) tuples."""
p = qp.proposal
out: list[tuple[str, int]] = [
(f"bottle: {p.bottle_slug}", 0),
(f"bottle: {qp.label}", 0),
(f"tool: {p.tool}", 0),
(f"id: {p.id}", 0),
(f"arrived: {p.arrival_timestamp}", 0),
@@ -136,39 +154,27 @@ def approve(
notes: str = "",
final_file: str | None = None,
) -> None:
"""Apply the proposal, write the waiting response, and audit it."""
status = STATUS_MODIFIED if final_file is not None else STATUS_APPROVED
file_to_apply = final_file if final_file is not None else qp.proposal.proposed_file
diff_before, diff_after = "", ""
if qp.proposal.tool in (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK):
diff_before, diff_after = apply_routes_change(
qp.proposal.bottle_slug,
file_to_apply,
)
response = Response(
proposal_id=qp.proposal.id,
status=status,
"""Approve (or, with `final_file`, modify-then-approve) via the
orchestrator: it applies the route change to the bottle's live policy,
writes the response that unblocks the agent, and audits it one atomic
server-side op. Raises `OrchestratorClientError` on failure."""
_client().supervise_respond(
qp.proposal.id,
bottle_slug=qp.proposal.bottle_slug,
decision="modify" if final_file is not None else "approve",
notes=notes,
final_file=final_file,
)
write_response(qp.proposal.bottle_slug, response)
_write_audit(
qp, action=status, notes=notes,
diff_before=diff_before, diff_after=diff_after,
)
def reject(qp: QueuedProposal, *, reason: str) -> None:
"""Write a rejection response and an audit entry."""
response = Response(
proposal_id=qp.proposal.id,
status=STATUS_REJECTED,
"""Reject via the orchestrator (writes the response + audit)."""
_client().supervise_respond(
qp.proposal.id,
bottle_slug=qp.proposal.bottle_slug,
decision="reject",
notes=reason,
final_file=None,
)
write_response(qp.proposal.bottle_slug, response)
_write_audit(qp, action=STATUS_REJECTED, notes=reason, diff_before="", diff_after="")
def _approve_from_tui(
@@ -188,29 +194,6 @@ def _approve_from_tui(
return _approval_status(qp, verb)
def _write_audit(
qp: QueuedProposal,
*,
action: str,
notes: str,
diff_before: str,
diff_after: str,
) -> None:
"""Audit log for egress tool."""
component = COMPONENT_FOR_TOOL.get(qp.proposal.tool)
if component is None:
return
write_audit_entry(AuditEntry(
timestamp=datetime.now(timezone.utc).isoformat(),
bottle_slug=qp.proposal.bottle_slug,
component=component,
operator_action=action,
operator_notes=notes,
justification=qp.proposal.justification,
diff=render_diff(diff_before, diff_after, label=component),
))
# --- $EDITOR integration --------------------------------------------------
@@ -245,6 +228,20 @@ def cmd_supervise(argv: list[str]) -> int:
)
args = parser.parse_args(argv)
# Establish the orchestrator connection up front so a missing control
# plane is a clean one-line error, not a curses crash mid-loop. This also
# starts the orchestrator on demand when none is running (see `_client`).
try:
_client()
except OrchestratorClientError as e:
error(str(e))
return 1
except Die as e:
# Backend has no orchestrator to start (e.g. macos-container).
if e.message:
error(e.message)
return e.code if isinstance(e.code, int) else 1
if args.once:
return _list_once()
try:
@@ -299,7 +296,7 @@ def _list_once() -> int:
for qp in pending:
sys.stdout.write(
f"{qp.proposal.arrival_timestamp} "
f"[{qp.proposal.bottle_slug}] "
f"[{qp.label}] "
f"{qp.proposal.tool} "
f"{qp.proposal.id}\n"
)
@@ -396,7 +393,7 @@ def _main_loop(stdscr: "curses._CursesWindow") -> None: # type: ignore # pragm
reason = _prompt(stdscr, "reject reason: ")
if reason:
reject(qp, reason=reason)
status_line = f"rejected {qp.proposal.tool} for [{qp.proposal.bottle_slug}]"
status_line = f"rejected {qp.proposal.tool} for [{qp.label}]"
else:
status_line = "reject aborted (empty reason)"
@@ -435,7 +432,7 @@ def _render(
cursor = "> " if i == selected else " "
line = (
f"{cursor}{ts_short} "
f"[{p.bottle_slug}] {p.tool:<18} {p.id[:8]}"
f"[{qp.label}] {p.tool:<18} {p.id[:8]}"
)
attr = curses.A_REVERSE if i == selected else curses.A_NORMAL
stdscr.addnstr(row, 0, line, w - 1, attr)
+10 -1
View File
@@ -32,6 +32,8 @@ if TYPE_CHECKING:
_SUPERVISE_MCP_NAME = "supervise"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
_IDENTITY_HEADER = "x-bot-bottle-identity"
def _skills_dir(guest_home: str) -> str:
@@ -91,6 +93,7 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="append_file",
bypass_args=("--dangerously-skip-permissions",),
resume_args=("--continue",),
smoke_test=("claude", "--version"),
)
@@ -300,9 +303,15 @@ class ClaudeAgentProvider(AgentProvider):
if plan.supervise_plan is None:
return
info(f"registering supervise MCP server in agent claude config → {supervise_url}")
# Deliver the identity token as an MCP request header — the supervise
# daemon requires it (mandatory (source_ip, token) attribution).
token = getattr(plan, "identity_token", "")
header = (
f" --header {shlex.quote(f'{_IDENTITY_HEADER}: {token}')}" if token else ""
)
r = bottle.exec(
f"claude mcp add --scope user --transport http "
f"{_SUPERVISE_MCP_NAME} {supervise_url}",
f"{_SUPERVISE_MCP_NAME} {supervise_url}{header}",
user="node",
)
if r.returncode != 0:
+67 -14
View File
@@ -9,6 +9,7 @@ invocation that registers the supervise daemon in Codex's
from __future__ import annotations
import base64
import os
import shlex
from pathlib import Path
@@ -26,7 +27,7 @@ from ...agent_provider import (
)
from .codex_auth import codex_host_access_token, write_codex_dummy_auth_file
from ...egress import CODEX_HOST_CREDENTIAL_TOKEN_REF, EgressRoute
from ...log import die, info, warn
from ...log import die, info
if TYPE_CHECKING:
@@ -34,6 +35,8 @@ if TYPE_CHECKING:
_SUPERVISE_MCP_NAME = "supervise"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
_IDENTITY_HEADER = "x-bot-bottle-identity"
_CODEX_CLI = "/home/node/.codex/packages/standalone/current/bin/codex"
_CODEX_CLI_PATH = (
"/home/node/.local/bin:"
@@ -42,6 +45,41 @@ _CODEX_CLI_PATH = (
)
def _toml_basic_string(value: str) -> str:
"""Quote `value` as a TOML basic (double-quoted) string."""
escaped = (
value.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\n", "\\n")
.replace("\t", "\\t")
)
return f'"{escaped}"'
def _supervise_mcp_config_toml(supervise_url: str, token: str) -> str:
"""Render the `[mcp_servers.supervise]` streamable-HTTP entry for
Codex's `config.toml`.
The Codex CLI has no `mcp add --header` flag; a static request
header on an HTTP MCP server is only expressible via the
`http_headers` config key (see `RawMcpServerConfig` /
`McpServerTransportConfig::StreamableHttp`). We deliver the
mandatory identity token (source_ip, token attribution) that way.
Only Codex-supported streamable-HTTP keys (`url`, `http_headers`)
are emitted."""
lines = [
"",
f"[mcp_servers.{_SUPERVISE_MCP_NAME}]",
f"url = {_toml_basic_string(supervise_url)}",
]
if token:
key = _toml_basic_string(_IDENTITY_HEADER)
val = _toml_basic_string(token)
lines.append(f"http_headers = {{ {key} = {val} }}")
lines.append("")
return "\n".join(lines)
def _skills_dir(guest_home: str) -> str:
# Codex agents still read skills from the claude-code convention
# (~/.claude/skills/) — the bot-bottle-codex image follows the
@@ -61,6 +99,7 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="read_prompt_file",
bypass_args=("--dangerously-bypass-approvals-and-sandbox",),
resume_args=("resume", "--last"),
smoke_test=(_CODEX_CLI, "--version"),
)
@@ -265,25 +304,39 @@ class CodexAgentProvider(AgentProvider):
bottle: "Bottle",
supervise_url: str,
) -> None:
"""Run `codex mcp add` inside the agent guest to register the
supervise daemon in Codex's user config (~/.codex/config.toml).
"""Register the supervise daemon as a streamable-HTTP MCP
server in Codex's user config (`~/.codex/config.toml`).
Mirrors the Claude provider's `claude mcp add` flow — failure
is logged but not fatal."""
We write the `[mcp_servers.supervise]` entry directly rather
than shelling out to `codex mcp add`: the CLI's `add` has no
way to attach a static request header, and the identity token
(mandatory (source_ip, token) attribution) MUST ride on the
MCP request as `http_headers`. Failure is FATAL when supervise
is enabled a silently-unregistered server leaves the agent
with no supervise access and, under mandatory attribution, no
way to recover from inside the bottle."""
if plan.supervise_plan is None:
return
info(f"registering supervise MCP server in agent codex config → {supervise_url}")
r = bottle.exec(
f"{shlex.quote(_CODEX_CLI)} mcp add {_SUPERVISE_MCP_NAME} --url "
f"{shlex.quote(supervise_url)}",
user="node",
token = getattr(plan, "identity_token", "")
block = _supervise_mcp_config_toml(supervise_url, token)
auth_dir = plan.agent_provision.guest_env.get("CODEX_HOME") \
or f"{plan.guest_home}/.codex"
config_path = f"{auth_dir}/config.toml"
# Append via base64 so the TOML payload never has to survive a
# shell-quoting round trip. node owns the config file, so append
# as node to preserve ownership/mode.
payload = base64.b64encode(block.encode()).decode()
script = (
f"printf %s {shlex.quote(payload)} | base64 -d "
f">> {shlex.quote(config_path)}"
)
r = bottle.exec(script, user="node")
if r.returncode != 0:
warn(
f"`codex mcp add supervise` failed (exit {r.returncode}): "
f"{(r.stderr or r.stdout or '').strip()}. Inside the bottle, "
f"register manually with: "
f"codex mcp add supervise --url {shlex.quote(supervise_url)}"
die(
"agent provider provisioning: could not register supervise "
f"MCP server in {config_path}: "
f"{(r.stderr or r.stdout or '').strip()}"
)
def headless_prompt(self, prompt: str) -> list[str]:
+62 -4
View File
@@ -6,6 +6,8 @@ egress container."""
from __future__ import annotations
import asyncio
import base64
import binascii
import json
import os
import signal
@@ -69,10 +71,28 @@ INTROSPECT_HOST = "_egress.local"
# → legacy per-bottle single-tenant mode (unchanged).
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
# App-layer identity token (defense-in-depth over the source-IP invariant);
# the agent injects it, the addon strips it so it never leaks upstream.
# App-layer identity token. Delivered as proxy credentials
# (`HTTPS_PROXY=http://<bottle_id>:<token>@gw`): clients honor it as part of
# the proxy protocol without app changes, and the addon reads + strips it so
# it never leaks upstream. The legacy `x-bot-bottle-identity` request header
# is still stripped defensively (git-http uses that header on its own port).
IDENTITY_HEADER = "x-bot-bottle-identity"
def _token_from_proxy_auth(header: str) -> str:
"""Extract the identity token (the password) from a `Proxy-Authorization:
Basic base64(<bottle_id>:<token>)` header. Empty on any malformed value
the mandatory `/resolve` then fail-closes on the empty token."""
scheme, _, encoded = header.partition(" ")
if scheme.lower() != "basic" or not encoded:
return ""
try:
decoded = base64.b64decode(encoded, validate=True).decode("utf-8")
except (binascii.Error, ValueError, UnicodeDecodeError):
return ""
_, _, password = decoded.partition(":")
return password
# Seconds the egress proxy holds a token-blocked request open waiting for the
# operator's supervisor decision (PRD 0062), overridable via env.
DEFAULT_TOKEN_ALLOW_TIMEOUT_SECONDS = 300.0
@@ -92,6 +112,10 @@ class EgressAddon:
# Class default so addons built via __new__ (e.g. in tests) default to
# single-tenant; __init__ sets the instance attribute for real runs.
_resolver: "PolicyResolver | None" = None
# Class default so __new__-built addons have it (real runs get a fresh
# per-instance dict in __init__; only http_connect mutates it, which the
# request-flow tests don't exercise).
_conn_tokens: "dict[str, str]" = {}
def __init__(self) -> None:
self.routes_path = os.environ.get("EGRESS_ROUTES", DEFAULT_ROUTES_PATH)
@@ -106,6 +130,10 @@ class EgressAddon:
# scan. In-memory only (a restart re-prompts); mutated only from the
# asyncio loop that runs the addon hooks, so no lock is needed.
self._safe_tokens: dict[str, set[str]] = {}
# Per-client-connection identity token captured from the CONNECT's
# `Proxy-Authorization` (HTTPS tunnels don't repeat it on the bumped
# inner requests). Keyed by client_conn.id; cleared on disconnect.
self._conn_tokens: dict[str, str] = {}
self._supervise_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "").strip()
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
self._reload(initial=True)
@@ -247,12 +275,42 @@ class EgressAddon:
return self.config, self._supervise_slug, os.environ
conn = flow.client_conn
client_ip = conn.peername[0] if conn and conn.peername else ""
token = flow.request.headers.get(IDENTITY_HEADER, "")
flow.request.headers.pop(IDENTITY_HEADER, None)
token = self._request_token(flow)
config, slug, tokens = resolve_client_context(self._resolver, client_ip, token)
env = {**os.environ, **tokens} if tokens else os.environ
return config, slug, env
def _request_token(self, flow: http.HTTPFlow) -> str:
"""The per-bottle identity token for this request, from the proxy
credentials the delivery mechanism (`HTTPS_PROXY=http://id:token@gw`)
that clients honor without app changes. Plain-HTTP requests carry
`Proxy-Authorization` directly; HTTPS bumped requests inherit the token
captured from their tunnel's CONNECT. Read then stripped so it never
leaks upstream (also strips the legacy header, if present)."""
token = _token_from_proxy_auth(
flow.request.headers.get("Proxy-Authorization", ""))
flow.request.headers.pop("Proxy-Authorization", None)
flow.request.headers.pop(IDENTITY_HEADER, None)
conn = flow.client_conn
if not token and conn is not None:
token = self._conn_tokens.get(getattr(conn, "id", ""), "")
return token
def http_connect(self, flow: http.HTTPFlow) -> None:
"""Capture the identity token from an HTTPS tunnel's CONNECT (the inner
bumped requests won't carry `Proxy-Authorization`), keyed by client
connection, and strip it so it never reaches upstream."""
token = _token_from_proxy_auth(
flow.request.headers.get("Proxy-Authorization", ""))
flow.request.headers.pop("Proxy-Authorization", None)
conn = flow.client_conn
if conn is not None and getattr(conn, "id", ""):
self._conn_tokens[conn.id] = token
def client_disconnected(self, client: typing.Any) -> None:
"""Drop the per-connection token when the client goes away."""
self._conn_tokens.pop(getattr(client, "id", ""), None)
async def request(self, flow: http.HTTPFlow) -> None:
request_path, _, query = flow.request.path.partition("?")
+13
View File
@@ -19,6 +19,9 @@ from .manifest import ManifestBottle, ManifestGitEntry
# Short network alias for git-gate inside the gateway. The
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
GIT_GATE_HOSTNAME = "git-gate"
# App-layer identity token header the agent's git sends to git-http and the
# gateway validates (mirrors egress_addon / git_http_backend IDENTITY_HEADER).
IDENTITY_HEADER = "x-bot-bottle-identity"
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
# git daemon (--timeout/--init-timeout), the access-hook subprocess in
# git_http_backend, and the git http-backend CGI subprocess.
@@ -75,6 +78,7 @@ def _gitconfig_validate_value(field: str, value: str) -> None:
def git_gate_render_gitconfig(
entries: tuple[ManifestGitEntry, ...], gate_host: str, *, scheme: str = "git",
identity_token: str = "",
) -> str:
"""Render the agent's ~/.gitconfig content for git-gate
`insteadOf` rewrites. Pure host-side, no docker / VM;
@@ -96,6 +100,15 @@ def git_gate_render_gitconfig(
"# the upstream bidirectionally (gitleaks-scanned push;\n",
"# fetch-from-upstream-before-every-upload-pack via access-hook).\n",
]
# Over the smart-HTTP transport (VM backends), attach the per-bottle
# identity token as a request header on requests to the gate, scoped to
# its URL so it never goes to any other remote. git-http requires it (the
# gateway's mandatory (source_ip, token) attribution). git:// (single-tenant
# docker) carries no header — attribution there is the network alias.
if identity_token and scheme == "http":
_gitconfig_validate_value("identity_token", identity_token)
out.append(f'[http "http://{gate_host}/"]\n')
out.append(f"\textraHeader = {IDENTITY_HEADER}: {identity_token}\n")
for entry in entries:
_gitconfig_validate_value(f"repos[{entry.Name!r}].url", entry.Upstream)
out.append(f'[url "{scheme}://{gate_host}/{entry.Name}.git"]\n')
+6
View File
@@ -16,6 +16,7 @@ import secrets
from pathlib import Path
from .. import log
from ..store_manager import StoreManager
from .broker import LaunchBroker, StubBroker
from .control_plane import make_server
from .docker_broker import DockerBroker
@@ -45,6 +46,11 @@ def main(argv: list[str] | None = None) -> int:
registry = RegistryStore(args.db)
registry.migrate()
# One DB per host: the supervise queue + audit tables live in the SAME
# SQLite file the registry owns, so the control plane is the single
# source of truth. The in-VM supervise daemon writes here; the host
# operator reaches it over HTTP (never a second, disconnected DB).
StoreManager(registry.db_path).migrate()
# An ephemeral signing secret ties the orchestrator (signer) to its
# broker (verifier). 'stub' records launches instead of starting
+61
View File
@@ -135,10 +135,71 @@ class OrchestratorClient:
bottles = payload.get("bottles")
return bottles if isinstance(bottles, list) else []
# --- supervise queue (operator TUI) ------------------------------------
def supervise_pending(self) -> list[dict[str, object]]:
"""Pending supervise proposals across all bottles
(`GET /supervise/proposals`)."""
payload = self._ok("GET", "/supervise/proposals")
proposals = payload.get("proposals")
return proposals if isinstance(proposals, list) else []
def supervise_respond(
self,
proposal_id: str,
*,
bottle_slug: str,
decision: str,
notes: str = "",
final_file: str | None = None,
) -> None:
"""Record an operator decision (`POST /supervise/respond`). `decision`
is approve/modify/reject. Raises `OrchestratorClientError` if the
proposal is gone or the bottle can no longer be applied to (409)."""
body: dict[str, object] = {
"proposal_id": proposal_id,
"bottle_slug": bottle_slug,
"decision": decision,
"notes": notes,
}
if final_file is not None:
body["final_file"] = final_file
self._ok("POST", "/supervise/respond", body)
def discover_orchestrator_url(*, timeout: float = 2.0) -> str:
"""The URL of the one running per-host orchestrator control plane, probing
the backends' well-known control-plane addresses (both on port 8099):
docker publishes it on loopback; the firecracker infra VM serves it on the
orchestrator TAP. Returns the first that answers `/health`; raises if none
do (no orchestrator up launch a bottle first)."""
candidates: list[str] = []
try: # docker: loopback-published control plane
from .lifecycle import DEFAULT_PORT as _DOCKER_PORT
candidates.append(f"http://127.0.0.1:{_DOCKER_PORT}")
except Exception: # noqa: BLE001 — backend optional
candidates.append("http://127.0.0.1:8099")
try: # firecracker: infra VM control plane on the orchestrator TAP
from ..backend.firecracker import netpool
from ..backend.firecracker.infra_vm import CONTROL_PLANE_PORT
candidates.append(
f"http://{netpool.orch_slot().guest_ip}:{CONTROL_PLANE_PORT}")
except Exception: # noqa: BLE001 — backend optional / not firecracker
pass
for url in candidates:
if OrchestratorClient(url, timeout=timeout).health():
return url
raise OrchestratorClientError(
"no running orchestrator control plane found (tried "
+ ", ".join(candidates)
+ "); launch a bottle first"
)
__all__ = [
"OrchestratorClient",
"OrchestratorClientError",
"RegisteredBottle",
"DEFAULT_TIMEOUT_SECONDS",
"discover_orchestrator_url",
]
+40 -2
View File
@@ -16,6 +16,10 @@ vsock / unix-socket portability caveats):
POST /attribute -> 200 {"bottle_id"} | 403
POST /resolve -> 200 {"bottle_id","policy"} | 403
body: {"source_ip","identity_token"}
GET /supervise/proposals -> 200 {"proposals": [ <proposal>, ...]}
POST /supervise/respond -> 200 {"responded": true} | 409 (operator)
body: {"proposal_id","bottle_slug",
"decision", ["notes"],["final_file"]}
`POST /bottles` / `DELETE` drive the full launch lifecycle: they mint (or
tear down) the bottle in the registry AND broker the backend-native launch
@@ -127,10 +131,44 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
return 403, {"error": "unattributed"}
return 200, {"bottle_id": rec.bottle_id}
if method == "GET" and route == "/supervise/proposals":
# Operator TUI: pending supervise proposals across all bottles.
return 200, {"proposals": orch.supervise_pending()}
if method == "POST" and route == "/supervise/respond":
# Operator decision: apply (approve/modify rewrites egress policy),
# write the queued response, audit — all server-side on the one DB.
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
proposal_id = data.get("proposal_id")
bottle_slug = data.get("bottle_slug")
decision = data.get("decision")
if not (isinstance(proposal_id, str) and proposal_id):
return 400, {"error": "proposal_id (string) is required"}
if not (isinstance(bottle_slug, str) and bottle_slug):
return 400, {"error": "bottle_slug (string) is required"}
if not (isinstance(decision, str) and decision):
return 400, {"error": "decision (string) is required"}
notes = data.get("notes")
final_file = data.get("final_file")
ok, err = orch.supervise_respond(
proposal_id,
bottle_slug=bottle_slug,
decision=decision,
notes=notes if isinstance(notes, str) else "",
final_file=final_file if isinstance(final_file, str) else None,
)
if ok:
return 200, {"responded": True}
return 409, {"error": err}
if method == "POST" and route == "/resolve":
# The per-request lookup the multi-tenant gateway makes: returns the
# bottle's policy. identity_token is OPTIONAL — absent means resolve
# by source IP alone (network-layer attribution).
# bottle's policy. Requires a matching (source_ip, identity_token)
# pair — a missing/empty/mismatched token fail-closes (403), no
# source-IP-only fallback.
try:
data = _parse_json_object(body)
except ValueError as e:
+27
View File
@@ -23,6 +23,13 @@ import time
from pathlib import Path
from ..docker_cmd import run_docker
from ..paths import host_db_path
from ..supervise import DB_PATH_IN_CONTAINER
# The host DB dir is bind-mounted here so the gateway's supervise daemon
# writes its queued proposals into the ONE host DB (the same file the
# orchestrator container opens and the operator reaches over HTTP).
_SUPERVISE_DB_DIR_IN_CONTAINER = os.path.dirname(DB_PATH_IN_CONTAINER)
# The gateway's mitmproxy writes its CA a beat after the container starts, so
# reads poll for it rather than assuming it's there on a fresh launch.
@@ -54,6 +61,14 @@ GATEWAY_DOCKERFILE = "Dockerfile.gateway"
_REPO_ROOT = Path(__file__).resolve().parents[2]
def _host_db_dir() -> str:
"""The host DB directory (created if missing), for the gateway's
supervise-DB bind-mount."""
db_dir = host_db_path().parent
db_dir.mkdir(parents=True, exist_ok=True)
return str(db_dir)
class GatewayError(Exception):
"""The shared gateway failed to build/start/stop (non-zero `docker` exit)."""
@@ -100,6 +115,7 @@ class DockerGateway(Gateway):
orchestrator_url: str = "",
build_context: Path | None = None,
dockerfile: str | None = GATEWAY_DOCKERFILE,
host_port_bindings: tuple[int, ...] = (),
) -> None:
self.image_ref = image_ref
self.name = name
@@ -110,6 +126,10 @@ class DockerGateway(Gateway):
self._orchestrator_url = orchestrator_url
self._build_context = build_context or _REPO_ROOT
self._dockerfile = dockerfile
# Ports published on the host (0.0.0.0). Used by the Firecracker
# backend's dev-harness gateway so VMs can reach it via their TAP link;
# Docker's DNAT + the nft `ct status dnat accept` rule handle the rest.
self._host_port_bindings = host_port_bindings
def image_exists(self) -> bool:
return run_docker(["docker", "image", "inspect", self.image_ref]).returncode == 0
@@ -187,7 +207,14 @@ class DockerGateway(Gateway):
# Persist the self-generated CA so it survives restarts (agents
# trust it) — see GATEWAY_CA_VOLUME.
"--volume", f"{GATEWAY_CA_VOLUME}:{MITMPROXY_HOME}",
# Share the one host DB: the supervise daemon queues proposals
# into the same file the orchestrator (and the operator, over
# HTTP) reads — no second, disconnected DB in the container.
"--volume", f"{_host_db_dir()}:{_SUPERVISE_DB_DIR_IN_CONTAINER}",
"--env", f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
]
for port in self._host_port_bindings:
argv += ["--publish", f"0.0.0.0:{port}:{port}"]
if self._orchestrator_url:
# Makes the gateway's egress / git / supervise daemons multi-tenant:
# each request resolves source-IP -> policy against the control plane.
+22 -9
View File
@@ -78,7 +78,13 @@ def _source_hash(repo_root: Path) -> str:
class OrchestratorService:
"""Manages the orchestrator control-plane container + the shared gateway.
Callers only need `ensure_running()` + `url`."""
Callers only need `ensure_running()` + `url`.
`orchestrator_name` / `orchestrator_label` let backends run independent
orchestrators on the same host without name collisions (e.g. the
Firecracker backend uses `bot-bottle-fc-orchestrator` alongside the Docker
backend's `bot-bottle-orchestrator`). Subclass and override `_gateway()`
to supply a backend-specific gateway variant."""
def __init__(
self,
@@ -89,6 +95,8 @@ class OrchestratorService:
gateway_image: str = GATEWAY_IMAGE,
repo_root: Path = _REPO_ROOT,
host_root: Path | None = None,
orchestrator_name: str = ORCHESTRATOR_NAME,
orchestrator_label: str = ORCHESTRATOR_LABEL,
) -> None:
self.port = port
self.network = network
@@ -100,6 +108,8 @@ class OrchestratorService:
self._gateway_image = gateway_image
self._repo_root = repo_root
self._host_root = host_root or bot_bottle_root()
self._orchestrator_name = orchestrator_name
self._orchestrator_label = orchestrator_label
@property
def url(self) -> str:
@@ -111,7 +121,7 @@ class OrchestratorService:
"""Control-plane URL as the gateway container reaches it — by name over
docker DNS on the shared network. This is the gateway's
BOT_BOTTLE_ORCHESTRATOR_URL."""
return f"http://{ORCHESTRATOR_NAME}:{self.port}"
return f"http://{self._orchestrator_name}:{self.port}"
def is_healthy(self, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS) -> bool:
try:
@@ -129,11 +139,11 @@ class OrchestratorService:
fixed-name container first). Register-only broker no docker socket.
Labels the container with `source_hash` so a later `ensure_running`
can detect a real code change (see `_source_hash`)."""
run_docker(["docker", "rm", "--force", ORCHESTRATOR_NAME])
run_docker(["docker", "rm", "--force", self._orchestrator_name])
proc = run_docker([
"docker", "run", "--detach",
"--name", ORCHESTRATOR_NAME,
"--label", ORCHESTRATOR_LABEL,
"--name", self._orchestrator_name,
"--label", self._orchestrator_label,
"--label", f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={source_hash}",
"--network", self.network,
# Host CLI reaches the control plane here; bound to loopback so it
@@ -185,12 +195,12 @@ class OrchestratorService:
*current* bind-mounted source. Mirrors `DockerGateway`'s
image-staleness check, but by content hash rather than image id since
the orchestrator runs bind-mounted source, not a built image."""
if not self._container_running(ORCHESTRATOR_NAME):
if not self._container_running(self._orchestrator_name):
return False
proc = run_docker([
"docker", "inspect", "--format",
"{{ index .Config.Labels \"" + ORCHESTRATOR_SOURCE_HASH_LABEL + "\" }}",
ORCHESTRATOR_NAME,
self._orchestrator_name,
])
if proc.returncode != 0:
return True # can't compare -> don't churn a working container
@@ -219,7 +229,10 @@ class OrchestratorService:
return self.url
self._ensure_orchestrator_image()
log.info("starting orchestrator container", context={"name": ORCHESTRATOR_NAME})
log.info(
"starting orchestrator container",
context={"name": self._orchestrator_name},
)
self._run_orchestrator_container(current_hash)
deadline = time.monotonic() + startup_timeout
@@ -234,7 +247,7 @@ class OrchestratorService:
def stop(self) -> None:
"""Remove the orchestrator + gateway containers (idempotent)."""
run_docker(["docker", "rm", "--force", ORCHESTRATOR_NAME])
run_docker(["docker", "rm", "--force", self._orchestrator_name])
self._gateway().stop()
+150 -10
View File
@@ -17,9 +17,37 @@ Launch lifecycle:
from __future__ import annotations
import json
from datetime import datetime, timezone
from .broker import LaunchBroker, LaunchRequest, sign_request
from .registry import BottleRecord, RegistryStore
from .gateway import Gateway
from ..supervise import (
AuditEntry,
COMPONENT_FOR_TOOL,
Response,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
list_all_pending_proposals,
read_proposal,
render_diff,
write_audit_entry,
write_response,
)
# Operator decision → Response.status. The apply half (egress tools) runs
# for approve/modify only.
_RESPOND_STATUS = {
"approve": STATUS_APPROVED,
"modify": STATUS_MODIFIED,
"reject": STATUS_REJECTED,
}
_APPLY_TOOLS = (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK)
class Orchestrator:
@@ -99,22 +127,134 @@ class Orchestrator:
"""Fail-closed attribution (delegates to the registry)."""
return self.registry.attribute(source_ip, identity_token)
def resolve(self, source_ip: str, identity_token: str = "") -> BottleRecord | None:
"""Resolve the bottle behind a request — the source-IP-keyed lookup
the multi-tenant gateway makes per request; the returned record
carries its `policy`. With a token, full attribution (source IP +
token); without, network-layer attribution by source IP alone
(valid where the IP is unspoofable and the control plane is
gateway-only)."""
if identity_token:
return self.registry.attribute(source_ip, identity_token)
return self.registry.by_source_ip(source_ip)
def resolve(self, source_ip: str, identity_token: str) -> BottleRecord | None:
"""Resolve the bottle behind a request — the per-request lookup the
multi-tenant gateway makes; the returned record carries its `policy`.
**Mandatory pair**: requires a matching `(source_ip, identity_token)`
(constant-time). There is no source-IP-only fallback the app-layer
token is delivered on every attributed data plane (egress proxy
credentials, git-gate/supervise headers), so a missing or mismatched
token fail-closes. This keeps a spoofed source IP (which the /31 TAP
alone does not prevent) from selecting another bottle's policy/tokens
without also holding that bottle's unguessable token."""
return self.registry.attribute(source_ip, identity_token)
def set_policy(self, bottle_id: str, policy: str) -> bool:
"""Update a bottle's gateway policy in place (live reload). False if
the bottle is unknown."""
return self.registry.set_policy(bottle_id, policy)
# --- supervise queue (operator approvals) ------------------------------
#
# The orchestrator owns the single DB *and* the live policy, so operator
# decisions are applied here, server-side, and reached over HTTP by the
# host TUI (no direct-DB access, one path for every backend).
def supervise_pending(self) -> list[dict[str, object]]:
"""All pending proposals across bottles, FIFO, as JSON dicts
(`Proposal.to_dict`, round-trippable via `Proposal.from_dict`).
Each dict carries an extra `bottle_label`: the bottle's human slug
resolved from the registry (the proposal itself is keyed by the
orchestrator-assigned bottle_id, which is opaque to an operator). The
CLI renders the label but still responds against `bottle_slug`."""
out: list[dict[str, object]] = []
for p in list_all_pending_proposals():
d = p.to_dict()
d["bottle_label"] = self._label_for(p.bottle_slug)
out.append(d)
return out
def _label_for(self, bottle_slug: str) -> str:
"""The human slug recorded in registry metadata for a proposal's
bottle, or the bottle_slug unchanged when the bottle is gone or has no
recorded slug so the label is always non-empty."""
rec = self.registry.get(bottle_slug)
if rec is None:
return bottle_slug
try:
meta = json.loads(rec.metadata) if rec.metadata else {}
except ValueError:
meta = {}
slug = meta.get("slug") if isinstance(meta, dict) else None
return slug if isinstance(slug, str) and slug else bottle_slug
def _record_for_slug(self, slug: str) -> BottleRecord | None:
"""The live registry record for a proposal's bottle, or None (e.g. the
bottle was torn down before the operator responded).
In consolidated mode the supervise server attributes each proposal to
the orchestrator-assigned bottle_id and stores that as the proposal's
`bottle_slug` (see supervise_server `_attributed_config`), so the fast
path is a direct bottle_id lookup. The metadata-slug scan is the
fallback for legacy single-tenant proposals keyed by the human slug."""
rec = self.registry.get(slug)
if rec is not None:
return rec
for rec in self.registry.all():
try:
meta = json.loads(rec.metadata) if rec.metadata else {}
except ValueError:
meta = {}
if isinstance(meta, dict) and meta.get("slug") == slug:
return rec
return None
def supervise_respond(
self,
proposal_id: str,
*,
bottle_slug: str,
decision: str,
notes: str = "",
final_file: str | None = None,
) -> tuple[bool, str]:
"""Record an operator decision on a queued proposal, applying it
server-side. `decision` is approve/modify/reject.
Approve/modify on an egress tool rewrites the bottle's policy so the
gateway serves the new routes on its next `/resolve` (the live apply);
then the queued Response is written (unblocking the agent's MCP call)
and an audit entry recorded all against the one DB. Returns
(ok, error): ok=False with a message when the proposal or decision is
unknown, or the bottle is gone so an approval can't be applied."""
status = _RESPOND_STATUS.get(decision)
if status is None:
return False, f"unknown decision {decision!r}"
try:
proposal = read_proposal(bottle_slug, proposal_id)
except FileNotFoundError:
return False, "no such proposal"
diff_before, diff_after = "", ""
if status in (STATUS_APPROVED, STATUS_MODIFIED) and proposal.tool in _APPLY_TOOLS:
new_policy = final_file if final_file is not None else proposal.proposed_file
rec = self._record_for_slug(bottle_slug)
if rec is None:
return False, (
f"bottle {bottle_slug!r} is no longer registered; "
"cannot apply the route change"
)
diff_before, diff_after = rec.policy, new_policy
self.set_policy(rec.bottle_id, new_policy)
write_response(bottle_slug, Response(
proposal_id=proposal_id, status=status, notes=notes, final_file=final_file,
))
component = COMPONENT_FOR_TOOL.get(proposal.tool)
if component is not None:
write_audit_entry(AuditEntry(
timestamp=datetime.now(timezone.utc).isoformat(),
bottle_slug=bottle_slug,
component=component,
operator_action=status,
operator_notes=notes,
justification=proposal.justification,
diff=render_diff(diff_before, diff_after, label=component),
))
return True, ""
# --- consolidated gateway ----------------------------------------------
def ensure_gateway(self) -> None:
+47 -3
View File
@@ -51,12 +51,16 @@ from dataclasses import dataclass, replace
try:
# Same-directory imports inside the bundle container; these files are
# COPYed flat under /app by Dockerfile.gateway.
from egress_addon_core import LOG_OFF, load_config
from egress_addon_core import (
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
)
from policy_resolver import PolicyResolveError, PolicyResolver
import supervise as _sv
except ModuleNotFoundError:
# Package imports for host-side tests and tooling.
from .egress_addon_core import LOG_OFF, load_config
from .egress_addon_core import (
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
)
from .policy_resolver import PolicyResolveError, PolicyResolver
from . import supervise as _sv
@@ -65,6 +69,8 @@ except ModuleNotFoundError:
MCP_PROTOCOL_VERSION = "2024-11-05"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
IDENTITY_HEADER = "x-bot-bottle-identity"
SERVER_NAME = "bot-bottle-supervise"
SERVER_VERSION = "0.1.0"
@@ -525,6 +531,17 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
if method == "tools/list":
return handle_tools_list(req.params)
if method == "tools/call":
# `list-egress-routes` is read-only introspection. In consolidated
# mode the gateway's *static* route table is empty (routes are
# resolved per request by source IP), so answer it from the calling
# bottle's resolved policy. Otherwise the agent sees an empty
# allowlist and composes an egress proposal that *replaces* the live
# routes instead of extending them — silently dropping base routes
# like api.anthropic.com when the operator approves it.
if req.params.get("name") == _sv.TOOL_LIST_EGRESS_ROUTES:
resolved = self._resolved_routes_payload()
if resolved is not None:
return resolved
# Attribute the proposal to the calling bottle. Single-tenant → the
# env slug on `config`; consolidated → the source-IP-resolved
# bottle id, so one shared server queues each bottle's proposal
@@ -532,6 +549,28 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
return handle_tools_call(req.params, self._attributed_config(config))
raise _RpcClientError(ERR_METHOD_NOT_FOUND, f"method not found: {method}")
def _resolved_routes_payload(self) -> dict[str, object] | None:
"""The calling bottle's live egress routes as the `list-egress-routes`
JSON payload, resolved by (source_ip, identity token) the same shape
the single-tenant introspection endpoint returns. None when there is no
resolver (single-tenant), so the caller falls back to that endpoint.
Fail-closed like `_attributed_config`: an unattributed source or an
unreachable orchestrator yields an empty route list (never another
bottle's), courtesy of `resolve_client_context`."""
resolver = getattr(self.server, "policy_resolver", None)
if resolver is None:
return None
headers = getattr(self, "headers", None)
token = headers.get(IDENTITY_HEADER, "") if headers is not None else ""
conf, _slug, _tokens = resolve_client_context(
resolver, self.client_address[0], token,
)
body = json.dumps(
{"routes": [route_to_yaml_dict(r) for r in conf.routes]}, indent=2,
)
return {"content": [{"type": "text", "text": body}], "isError": False}
def _attributed_config(self, config: ServerConfig) -> ServerConfig:
"""The ServerConfig with `bottle_slug` bound to *this request's* bottle.
Single-tenant (no resolver): unchanged. Consolidated: the bottle id
@@ -541,8 +580,13 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
resolver = getattr(self.server, "policy_resolver", None)
if resolver is None:
return config
# The agent's MCP client sends the identity token as a request header
# (provisioned via `mcp add --header`); the orchestrator requires the
# (source_ip, token) pair, so a missing/wrong token fail-closes below.
headers = getattr(self, "headers", None)
token = headers.get(IDENTITY_HEADER, "") if headers is not None else ""
try:
bottle_id = resolver.resolve_bottle_id(self.client_address[0])
bottle_id = resolver.resolve_bottle_id(self.client_address[0], token)
except PolicyResolveError as e:
raise _RpcInternalError(f"orchestrator unreachable, cannot attribute: {e}") from e
if not bottle_id:
+21 -4
View File
@@ -49,10 +49,12 @@ let
# /31 alignment == an even final octet (only bit 0 matters for base+2i).
lastOctet = lib.toInt (lib.last (lib.splitString "." cfg.ipBase));
# The script needs ip/nft/sysctl + the usual coreutils. It gets every
# pool value via the unit's Environment=, so it never reads the shared
# defaults file (which isn't beside it once copied to the store).
runtimePath = with pkgs; [ iproute2 nftables procps coreutils gnused ];
# The script needs ip/nft/sysctl + the usual coreutils, plus iptables
# for the orchestrator link's DOCKER-USER accept (best-effort; skipped
# when Docker is absent). It gets every pool value via the unit's
# Environment=, so it never reads the shared defaults file (which isn't
# beside it once copied to the store).
runtimePath = with pkgs; [ iproute2 nftables iptables procps coreutils gnused ];
ownEnv =
if cfg.group != null
@@ -64,6 +66,7 @@ let
BOT_BOTTLE_FC_IP_BASE = cfg.ipBase;
BOT_BOTTLE_FC_IFACE_PREFIX = cfg.ifacePrefix;
BOT_BOTTLE_FC_NFT_TABLE = cfg.tableName;
BOT_BOTTLE_FC_ORCH_IFACE = cfg.orchIface;
} // ownEnv;
in
{
@@ -127,6 +130,19 @@ in
description = "nftables table name for the isolation boundary. Must match netpool.NFT_TABLE.";
};
orchIface = lib.mkOption {
type = lib.types.str;
default = defaults.BOT_BOTTLE_FC_ORCH_IFACE;
defaultText = lib.literalMD "the shared `netpool.defaults.env` value";
description = ''
TAP name for the orchestrator/gateway VM's dedicated link. Unlike
the isolated bbfc* agent pool, this link is NAT'd to the internet
(the orchestrator is trusted infra that builds agent images in-VM
and forwards agent egress upstream). Must match
BOT_BOTTLE_FC_ORCH_IFACE / netpool.ORCH_IFACE.
'';
};
writeEnvFile = lib.mkOption {
type = lib.types.bool;
default = false;
@@ -176,6 +192,7 @@ in
BOT_BOTTLE_FC_IP_BASE=${cfg.ipBase}
BOT_BOTTLE_FC_IFACE_PREFIX=${cfg.ifacePrefix}
BOT_BOTTLE_FC_NFT_TABLE=${cfg.tableName}
BOT_BOTTLE_FC_ORCH_IFACE=${cfg.orchIface}
'';
};
};
+127 -2
View File
@@ -63,12 +63,13 @@ POOL_SIZE="${BOT_BOTTLE_FC_POOL_SIZE:-$(_default BOT_BOTTLE_FC_POOL_SIZE)}"
IP_BASE="${BOT_BOTTLE_FC_IP_BASE:-$(_default BOT_BOTTLE_FC_IP_BASE)}"
PREFIX="${BOT_BOTTLE_FC_IFACE_PREFIX:-$(_default BOT_BOTTLE_FC_IFACE_PREFIX)}"
TABLE="${BOT_BOTTLE_FC_NFT_TABLE:-$(_default BOT_BOTTLE_FC_NFT_TABLE)}"
ORCH_IFACE="${BOT_BOTTLE_FC_ORCH_IFACE:-$(_default BOT_BOTTLE_FC_ORCH_IFACE)}"
OWNER="${BOT_BOTTLE_FC_OWNER:-${SUDO_USER:-$USER}}"
GROUP="${BOT_BOTTLE_FC_GROUP:-}"
# Fail loudly rather than provisioning a half/empty range if a value
# resolved to nothing (env unset AND the shared file unreadable).
for _v in POOL_SIZE IP_BASE PREFIX TABLE; do
for _v in POOL_SIZE IP_BASE PREFIX TABLE ORCH_IFACE; do
[ -n "${!_v}" ] || { echo "error: $_v unresolved (set BOT_BOTTLE_FC_* or fix $_DEFAULTS)" >&2; exit 1; }
done
@@ -89,6 +90,13 @@ host_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 )); }
guest_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 + 1 )); }
iface() { echo "${PREFIX}$1"; }
# Orchestrator/gateway VM link: a /31 at the TOP of the IP_BASE /16
# (host x.y.255.0, guest x.y.255.1), well clear of the agent pool near
# the bottom of the block. Must match netpool.py:orch_slot().
_orch_base() { echo $(( ($(_ip_to_int "$IP_BASE") & 0xFFFF0000) + 0xFF00 )); }
orch_host() { _int_to_ip "$(_orch_base)"; }
orch_guest() { _int_to_ip $(( $(_orch_base) + 1 )); }
require_root() {
if [ "$(id -u)" -ne 0 ]; then
echo "error: '$1' needs root (run under sudo)" >&2
@@ -125,8 +133,21 @@ cmd_up() {
echo " $dev host=$host guest=$(guest_ip "$i") $own_desc"
done
# The orchestrator/gateway VM's dedicated link. Same rootless-open
# ownership as the pool, but NAT'd to the internet (below) — it is
# trusted infra, not an isolated agent slot.
ip link show "$ORCH_IFACE" >/dev/null 2>&1 \
|| ip tuntap add dev "$ORCH_IFACE" mode tap "${own_args[@]}"
ip addr replace "$(orch_host)/31" dev "$ORCH_IFACE"
ip link set "$ORCH_IFACE" up
echo " $ORCH_IFACE host=$(orch_host) guest=$(orch_guest) (NAT'd egress) $own_desc"
_install_nft
echo "nftables table inet $TABLE installed (fail-closed boundary)"
_install_orch_egress
echo "orchestrator egress installed ($ORCH_IFACE -> NAT out)"
_install_gateway_route
echo "agent->gateway route installed (${PREFIX}* :$GATEWAY_PORTS -> $(orch_guest))"
echo "done."
}
@@ -142,12 +163,30 @@ _install_nft() {
# input: a VM never needs host-local delivery (its gateway is
# reached via DNAT->forward), so drop all direct input from VMs
# -> host services bound on 0.0.0.0 are unreachable from the VM.
# Delete-first (create empty, delete, recreate) so a re-applied `up`
# lands identical state instead of appending rules / erroring on the
# existing base chains — the setup is idempotent regardless of history.
# Anti-spoof: bind each TAP to its assigned guest IP. The /31 alone
# does NOT make the source address unspoofable — root in an agent VM
# can source another bottle's guest IP on its own bbfc TAP, and the
# gateway attributes egress/policy/tokens by source IP. So drop any
# packet whose source isn't the guest address assigned to the exact
# TAP it arrived on, before it can be attributed. One rule per slot.
local antispoof="" i
for i in $(seq 0 $((POOL_SIZE-1))); do
antispoof="${antispoof} iifname \"$(iface "$i")\" ip saddr != $(guest_ip "$i") drop
"
done
nft -f - <<EOF
table inet $TABLE {}
delete table inet $TABLE
table inet $TABLE {
chain forward {
type filter hook forward priority -10; policy accept;
iifname != "${PREFIX}*" return
ct state established,related accept
${antispoof} ct state established,related accept
ct status dnat accept
drop
}
@@ -161,8 +200,87 @@ table inet $TABLE {
EOF
}
# Give the orchestrator/gateway VM real internet egress (agent VMs get
# none — that's the isolation table above). Three parts, because the
# path must work both during bootstrap (Docker still present) and after
# Docker is removed:
# * masquerade — SNAT the orch guest /31 out the host uplink so its
# RFC-1918 address can reach the internet.
# * nft forward — accept the orch link's forward path (load-bearing
# on a pure-nft host whose FORWARD policy drops; a
# harmless no-op where forwarding is already open).
# It never drops, so it can't weaken the isolation
# table's agent drops.
# * DOCKER-USER — during bootstrap Docker's FORWARD chain policy is
# DROP; its sanctioned DOCKER-USER hook is the only
# place a user ACCEPT survives. Best-effort + guarded
# (skipped once Docker is gone).
_install_orch_egress() {
nft -f - <<EOF
table inet ${TABLE}_nat {}
delete table inet ${TABLE}_nat
table inet ${TABLE}_nat {
chain forward {
type filter hook forward priority -10; policy accept;
iifname "$ORCH_IFACE" accept
oifname "$ORCH_IFACE" ct state established,related accept
}
chain postrouting {
type nat hook postrouting priority 100; policy accept;
ip saddr $(orch_guest) oifname != "$ORCH_IFACE" masquerade
}
}
EOF
_docker_user_orch add
}
# Insert (add) or delete (del) the DOCKER-USER ACCEPT rules for the
# orchestrator link, idempotently, only when the chain exists.
_docker_user_orch() {
local op="$1" flag
command -v iptables >/dev/null 2>&1 || return 0
iptables -t filter -L DOCKER-USER >/dev/null 2>&1 || return 0
for flag in "-i" "-o"; do
if [ "$op" = add ]; then
iptables -C DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT 2>/dev/null \
|| iptables -I DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT
else
iptables -D DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT 2>/dev/null || true
fi
done
}
# Agent -> gateway VM routing. The shared gateway (egress / supervise /
# git-http) runs in the orchestrator/infra VM at $(orch_guest). Agents keep
# addressing their own host-side TAP IP on the gateway ports; a PREROUTING
# DNAT redirects that to the infra VM, and the isolation table's
# `ct status dnat accept` forward rule lets it through — every other agent
# egress stays dropped. Source IP is deliberately NOT masqueraded: the
# gateway attributes each request to the originating bottle by its (nft +
# /31 unspoofable) guest IP.
_install_gateway_route() {
nft -f - <<EOF
table ip ${TABLE}_gw {}
delete table ip ${TABLE}_gw
table ip ${TABLE}_gw {
chain prerouting {
type nat hook prerouting priority -100; policy accept;
iifname "${PREFIX}*" tcp dport { $GATEWAY_PORTS } dnat to $(orch_guest)
}
}
EOF
}
cmd_down() {
require_root down
_docker_user_orch del
nft delete table ip "${TABLE}_gw" 2>/dev/null || true
nft delete table inet "${TABLE}_nat" 2>/dev/null || true
if ip link show "$ORCH_IFACE" >/dev/null 2>&1; then
ip link set "$ORCH_IFACE" down 2>/dev/null || true
ip tuntap del dev "$ORCH_IFACE" mode tap 2>/dev/null || true
echo " removed $ORCH_IFACE"
fi
nft delete table inet "$TABLE" 2>/dev/null || true
for i in $(seq 0 $((POOL_SIZE-1))); do
local dev ; dev="$(iface "$i")"
@@ -178,6 +296,10 @@ cmd_down() {
cmd_status() {
echo "table inet $TABLE:"
nft list table inet "$TABLE" 2>/dev/null || echo " (absent)"
echo "table inet ${TABLE}_nat (orchestrator egress):"
nft list table inet "${TABLE}_nat" 2>/dev/null || echo " (absent)"
echo "table ip ${TABLE}_gw (agent->gateway route):"
nft list table ip "${TABLE}_gw" 2>/dev/null || echo " (absent)"
echo "taps:"
for i in $(seq 0 $((POOL_SIZE-1))); do
local dev ; dev="$(iface "$i")"
@@ -185,6 +307,9 @@ cmd_status() {
ip -brief addr show "$dev" | sed 's/^/ /'
fi
done
if ip -brief addr show "$ORCH_IFACE" >/dev/null 2>&1; then
ip -brief addr show "$ORCH_IFACE" | sed 's/^/ /'
fi
}
case "${1:-}" in
+35
View File
@@ -244,5 +244,40 @@ class TestHasBackend(unittest.TestCase):
self.assertFalse(has_backend("nonexistent"))
class TestEnsureOrchestrator(unittest.TestCase):
"""The backend-agnostic orchestrator bring-up entry point. Docker starts
the orchestrator + gateway containers; firecracker boots the infra VM;
backends without one (macos-container) die with a pointer."""
def test_docker_delegates_to_orchestrator_service(self):
b = get_bottle_backend("docker")
with patch(
"bot_bottle.orchestrator.lifecycle.OrchestratorService"
) as service_cls:
service_cls.return_value.ensure_running.return_value = (
"http://127.0.0.1:8099"
)
url = b.ensure_orchestrator()
self.assertEqual(url, "http://127.0.0.1:8099")
service_cls.return_value.ensure_running.assert_called_once_with()
def test_firecracker_delegates_to_infra_vm(self):
b = get_bottle_backend("firecracker")
with patch(
"bot_bottle.backend.firecracker.infra_vm.ensure_running"
) as ensure_running:
ensure_running.return_value.control_plane_url = (
"http://10.243.255.1:8099"
)
url = b.ensure_orchestrator()
self.assertEqual(url, "http://10.243.255.1:8099")
def test_macos_default_dies(self):
from bot_bottle.log import Die
b = get_bottle_backend("macos-container")
with self.assertRaises(Die):
b.ensure_orchestrator()
if __name__ == "__main__":
unittest.main()
+81 -13
View File
@@ -54,6 +54,7 @@ def _plan(
skills: list[str] | None = None,
agent_provision: AgentProvisionPlan | None = None,
supervise: bool = False,
identity_token: str = "",
) -> DockerBottlePlan:
bottle_json: dict = {"agent_provider": {"template": "codex"}} # type: ignore
if supervise:
@@ -100,6 +101,7 @@ def _plan(
),
supervise_plan=supervise_plan,
use_runsc=False,
identity_token=identity_token,
agent_provision=agent_provision or AgentProvisionPlan(
template="codex", command="codex", prompt_mode="read_prompt_file",
image="bot-bottle-codex:latest", dockerfile="",
@@ -314,6 +316,31 @@ class TestCodexDockerfile(unittest.TestCase):
self.assertIn("procps", dockerfile)
# Codex-supported streamable-HTTP MCP config keys (RawMcpServerConfig in
# codex-rs/config/src/mcp_types.rs). config.toml uses deny_unknown_fields,
# so an entry that names anything outside this set is rejected by the CLI.
_CODEX_HTTP_MCP_KEYS = frozenset({
"url", "http_headers", "env_http_headers", "bearer_token_env_var",
# shared (transport-agnostic) keys
"environment_id", "auth", "startup_timeout_sec", "startup_timeout_ms",
"tool_timeout_sec", "enabled", "required", "supports_parallel_tool_calls",
"default_tools_approval_mode", "enabled_tools", "disabled_tools",
"scopes", "oauth", "oauth_resource", "name", "tools",
})
def _append_target(bottle: MagicMock) -> tuple[str, str]:
"""Reconstruct (config_path, appended_toml) from the base64 append
script the provider ran."""
import base64 as _b64
script = bottle.exec.call_args.args[0]
# printf %s '<b64>' | base64 -d >> '<path>'
b64 = script.split("printf %s ", 1)[1].split(" |", 1)[0].strip("'")
path = script.rsplit(">> ", 1)[1].strip().strip("'")
return path, _b64.b64decode(b64).decode()
class TestCodexSuperviseMcp(unittest.TestCase):
def test_noop_when_supervise_disabled(self):
bottle = _make_bottle()
@@ -322,27 +349,68 @@ class TestCodexSuperviseMcp(unittest.TestCase):
)
bottle.exec.assert_not_called()
def test_runs_codex_mcp_add_as_node(self):
def test_appends_streamable_http_entry_as_node(self):
import tomllib
bottle = _make_bottle()
plan = _plan(supervise=True, identity_token="tok-abc123")
CodexAgentProvider().provision_supervise_mcp(plan, bottle, _URL)
bottle.exec.assert_called_once()
self.assertEqual("node", bottle.exec.call_args.kwargs.get("user"))
config_path, appended = _append_target(bottle)
self.assertEqual("/home/node/.codex/config.toml", config_path)
# The appended block must be valid TOML and parse to a streamable
# HTTP server carrying the identity token as a static http header.
parsed = tomllib.loads(appended)
server = parsed["mcp_servers"]["supervise"]
self.assertEqual(_URL, server["url"])
self.assertEqual(
"tok-abc123", server["http_headers"]["x-bot-bottle-identity"],
)
# Never emit an unsupported key (config.toml is deny_unknown_fields);
# in particular there is no `--header` / `header` surface.
self.assertTrue(
set(server).issubset(_CODEX_HTTP_MCP_KEYS),
f"unsupported codex mcp keys: {set(server) - _CODEX_HTTP_MCP_KEYS}",
)
self.assertNotIn("mcp add", bottle.exec.call_args.args[0])
def test_appends_to_custom_codex_home(self):
bottle = _make_bottle()
provision = AgentProvisionPlan(
template="codex", command="codex", prompt_mode="read_prompt_file",
image="", dockerfile="", guest_home="/home/node",
instance_name="bot-bottle-demo-abc12",
prompt_file=Path("/tmp/prompt.txt"),
guest_env={"CODEX_HOME": "/home/node/alt-codex"},
)
plan = _plan(
supervise=True, agent_provision=provision, identity_token="tok",
)
CodexAgentProvider().provision_supervise_mcp(plan, bottle, _URL)
config_path, _ = _append_target(bottle)
self.assertEqual("/home/node/alt-codex/config.toml", config_path)
def test_omits_http_headers_when_no_token(self):
import tomllib
bottle = _make_bottle()
CodexAgentProvider().provision_supervise_mcp(
_plan(supervise=True), bottle, _URL,
)
bottle.exec.assert_called_once()
script = bottle.exec.call_args.args[0]
self.assertEqual("node", bottle.exec.call_args.kwargs.get("user"))
self.assertEqual(
"/home/node/.codex/packages/standalone/current/bin/codex "
f"mcp add supervise --url {_URL}",
script,
)
_, appended = _append_target(bottle)
server = tomllib.loads(appended)["mcp_servers"]["supervise"]
self.assertNotIn("http_headers", server)
def test_logs_warning_on_failure_but_does_not_raise(self):
def test_registration_failure_is_fatal(self):
bottle = _make_bottle(
exec_result=ExecResult(returncode=1, stdout="", stderr="boom"),
)
CodexAgentProvider().provision_supervise_mcp(
_plan(supervise=True), bottle, _URL,
)
with self.assertRaises(SystemExit):
CodexAgentProvider().provision_supervise_mcp(
_plan(supervise=True), bottle, _URL,
)
class TestCodexHeadlessPrompt(unittest.TestCase):
@@ -91,6 +91,7 @@ class TestLaunchCommittedImage(unittest.TestCase):
with mock.patch.object(launch_mod, "read_committed_image", return_value=committed_tag), \
mock.patch.object(launch_mod.docker_mod, "image_exists", return_value=image_present), \
mock.patch.object(launch_mod.docker_mod, "build_image", side_effect=_build), \
mock.patch.object(launch_mod.docker_mod, "verify_agent_image"), \
mock.patch.object(launch_mod, "launch_consolidated", return_value=_CTX), \
mock.patch.object(launch_mod, "teardown_consolidated"), \
mock.patch.object(launch_mod, "DockerGateway", return_value=gw), \
@@ -94,6 +94,7 @@ class TestTeardownWarning(unittest.TestCase):
)
with mock.patch.object(launch_mod.docker_mod, "build_image"), \
mock.patch.object(launch_mod.docker_mod, "verify_agent_image"), \
mock.patch.object(launch_mod, "launch_consolidated", return_value=ctx), \
mock.patch.object(launch_mod, "teardown_consolidated"), \
mock.patch.object(launch_mod, "DockerGateway", return_value=gw), \
+36
View File
@@ -255,6 +255,30 @@ class TestBottleAgentArgv(unittest.TestCase):
argv[idx:],
)
def test_codex_multiword_prompt_survives_ssh_reparse(self):
# codex's read_prompt_file mode passes a single positional with
# spaces ("Read and follow the instructions in <path>."). ssh
# space-joins the remote argv and the guest shell re-splits it, so
# the token MUST be quoted or codex sees "and" as a subcommand
# (regression). Each remote token is shlex.quote'd; round-tripping
# the joined remote command back through shlex.split must recover
# the prompt as ONE argument.
import shlex
argv = _bottle(
agent_command="codex",
agent_prompt_mode="read_prompt_file",
agent_provider_template="codex",
prompt_path_in_guest="/home/node/.bot-bottle-prompt.txt",
).agent_argv([], tty=False)
idx = argv.index("--")
remote_line = " ".join(argv[idx + 1:]) # what ssh sends to the guest
reparsed = shlex.split(remote_line) # what the guest shell sees
prompt = "Read and follow the instructions in /home/node/.bot-bottle-prompt.txt."
self.assertIn(prompt, reparsed)
# codex is the last simple token before the (single) prompt arg.
self.assertEqual([*reparsed[reparsed.index("codex"):]], ["codex", prompt])
def test_workdir_sets_chdir(self):
# The agent runs from its workdir via `env --chdir` (ssh-safe;
# not a `sh -c 'cd …'` wrapper, which the ssh arg-join mangles).
@@ -334,6 +358,18 @@ class TestBootArgs(unittest.TestCase):
self.assertEqual("/run/rootfs.ext4", cfg["drives"][0]["path_on_host"])
self.assertFalse(cfg["drives"][0]["is_read_only"])
self.assertEqual("bbfc0", cfg["network-interfaces"][0]["host_dev_name"])
self.assertEqual(1, len(cfg["drives"])) # no data drive by default
def test_config_adds_data_drive(self):
cfg = cast(Any, firecracker_vm._config(
rootfs=Path("/run/rootfs.ext4"), tap="bbfc0",
guest_ip="100.64.0.1", host_ip="100.64.0.0", pubkey="k",
vcpus=2, mem_mib=2048, guest_mac="06:00:AC:10:00:02",
data_drive=Path("/run/registry.ext4"),
))
self.assertEqual(2, len(cfg["drives"]))
self.assertFalse(cfg["drives"][1]["is_root_device"])
self.assertEqual("/run/registry.ext4", cfg["drives"][1]["path_on_host"])
class TestBottleExecClose(unittest.TestCase):
+15
View File
@@ -64,6 +64,21 @@ class TestNetpoolProbes(unittest.TestCase):
patch.object(netpool, "tap_present", side_effect=[True, False]):
self.assertEqual(["bbfc1"], netpool.missing_taps())
def test_orch_slot_is_top_of_ip_base_16(self):
# Dedicated orchestrator link: /31 at the top of the IP_BASE /16,
# clear of the pool (bottom of the block). Must match the shell
# script's orch_host()/orch_guest() and be a non-pool index.
with patch.dict("os.environ", {"BOT_BOTTLE_FC_IP_BASE": "10.243.0.0"}):
s = netpool.orch_slot()
self.assertEqual(netpool.ORCH_IFACE, s.iface)
self.assertEqual("10.243.255.0", s.host_ip)
self.assertEqual("10.243.255.1", s.guest_ip)
self.assertEqual(-1, s.index)
# Never collides with a pool slot's guest address.
with patch.dict("os.environ", {"BOT_BOTTLE_FC_IP_BASE": "10.243.0.0"}):
pool_guests = {sl.guest_ip for sl in netpool.all_slots()}
self.assertNotIn(s.guest_ip, pool_guests)
class TestConsoleTail(unittest.TestCase):
def test_reads_tail(self):
@@ -0,0 +1,74 @@
"""Unit tests for the docker-free Firecracker agent-image builder.
The VM boot / SSH / buildah plumbing (`_build_in_infra`) is integration-tested
on a KVM host; here we cover the cache decision, the boot-bit injection, and
the smoke-test no-op the logic that must hold without a VM.
"""
from __future__ import annotations
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from bot_bottle.backend.firecracker import image_builder
class TestBuildAgentRootfsDir(unittest.TestCase):
def setUp(self):
self._tmp = tempfile.TemporaryDirectory()
self.cache = Path(self._tmp.name)
self.dockerfile = self.cache / "Dockerfile"
self.dockerfile.write_text("FROM node:22-slim\n")
self.addCleanup(self._tmp.cleanup)
def test_cache_hit_skips_rebuild(self):
digest = image_builder._dockerfile_hash(self.dockerfile)
base = self.cache / "rootfs" / f"agent-{digest}"
base.mkdir(parents=True)
(base / ".bb-ready").write_text("ok\n")
with patch.object(image_builder.util, "cache_dir", return_value=self.cache), \
patch.object(image_builder, "_build_in_infra") as build:
out = image_builder.build_agent_rootfs_dir(
self.dockerfile, image_tag="t:latest")
build.assert_not_called()
self.assertEqual(base, out)
def test_cache_miss_builds_injects_and_marks_ready(self):
with patch.object(image_builder.util, "cache_dir", return_value=self.cache), \
patch.object(image_builder, "_build_in_infra") as build, \
patch.object(image_builder.util, "inject_guest_boot") as inject:
out = image_builder.build_agent_rootfs_dir(
self.dockerfile, image_tag="t:latest", smoke_test=("claude", "--version"))
build.assert_called_once()
# smoke_test threads through to the VM build.
self.assertEqual(("claude", "--version"), build.call_args.args[2])
inject.assert_called_once()
self.assertTrue((out / ".bb-ready").is_file())
def test_content_addressed_cache_key(self):
other = self.cache / "Dockerfile2"
other.write_text("FROM python:3.12-slim\n")
self.assertNotEqual(
image_builder._dockerfile_hash(self.dockerfile),
image_builder._dockerfile_hash(other),
)
class TestSmokeTest(unittest.TestCase):
def test_empty_argv_is_noop(self):
with patch.object(image_builder, "_ssh") as ssh:
image_builder._smoke_test(Path("/k"), "10.0.0.1", "tag", "ctr", ())
ssh.assert_not_called()
def test_failed_smoke_dies(self):
import subprocess
result = subprocess.CompletedProcess([], 1, stdout="broken", stderr="")
with patch.object(image_builder, "_ssh", return_value=result), \
self.assertRaises(SystemExit):
image_builder._smoke_test(Path("/k"), "10.0.0.1", "tag", "ctr", ("claude", "--version"))
if __name__ == "__main__":
unittest.main()
+179
View File
@@ -0,0 +1,179 @@
"""Unit tests for the Firecracker infra VM (control-plane VM boot).
The KVM boot / HTTP reachability is integration-tested on a KVM host; here
we cover the URL shape, the rootfs-variant wiring, and the health-poll
decisions that must hold without a VM.
"""
from __future__ import annotations
import unittest
from pathlib import Path
from unittest.mock import MagicMock, patch
from bot_bottle.backend.firecracker import infra_vm
class TestControlPlaneUrl(unittest.TestCase):
def test_url_uses_guest_ip_and_port(self):
infra = infra_vm.InfraVm(
vm=MagicMock(), guest_ip="10.243.255.1", private_key=Path("/k"))
self.assertEqual(
f"http://10.243.255.1:{infra_vm.CONTROL_PLANE_PORT}",
infra.control_plane_url,
)
class TestBuildInfraRootfs(unittest.TestCase):
def test_uses_infra_variant_and_init(self):
with patch.object(infra_vm.util, "build_base_rootfs_dir") as build:
build.return_value = Path("/cache/rootfs/x-infra")
infra_vm.build_infra_rootfs_dir()
build.assert_called_once()
self.assertEqual(infra_vm._INFRA_IMAGE, build.call_args.args[0])
# variant is "-infra-<init-hash>" so an init change rebuilds the rootfs.
self.assertTrue(build.call_args.kwargs["variant"].startswith("-infra-"))
# The init runs BOTH the control plane and the gateway data plane,
# and exports PATH so gateway_init's subprocess daemons find python3.
init = build.call_args.kwargs["init_script"]
self.assertIn("bot_bottle.orchestrator", init)
self.assertIn("gateway_init.py", init)
self.assertIn("export PATH=", init)
# Persistent registry volume mounted at the DB dir before the CP starts.
self.assertIn("/dev/vdb", init)
# VM backend uses git-http (9420); the git:// daemon is left out.
self.assertIn("BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise", init)
class TestSshGatewayTransport(unittest.TestCase):
def test_cp_into_preserves_source_mode(self):
import os
import tempfile
from subprocess import CompletedProcess
with tempfile.NamedTemporaryFile() as f:
os.chmod(f.name, 0o700) # like the staged access-hook
t = infra_vm.SshGatewayTransport(Path("/k"), "10.0.0.1")
with patch.object(infra_vm.subprocess, "run",
return_value=CompletedProcess([], 0)) as run:
t.cp_into(f.name, "/etc/git-gate/access-hook")
remote_cmd = run.call_args.args[0][-1]
self.assertIn("chmod 700", remote_cmd) # exec bit preserved over SSH
def test_exec_raises_on_failure(self):
from subprocess import CompletedProcess
t = infra_vm.SshGatewayTransport(Path("/k"), "10.0.0.1")
with patch.object(infra_vm.subprocess, "run",
return_value=CompletedProcess([], 1, stderr="nope")), \
self.assertRaises(infra_vm.GatewayProvisionError):
t.exec(["mkdir", "-p", "/git-gate"])
class TestRegistryVolume(unittest.TestCase):
def test_reuses_existing_volume(self):
import tempfile
with tempfile.TemporaryDirectory() as td:
vol = Path(td) / "registry.ext4"
vol.write_bytes(b"") # already present
with patch.object(infra_vm, "registry_volume_path", return_value=vol), \
patch.object(infra_vm.subprocess, "run") as run:
out = infra_vm._ensure_registry_volume()
run.assert_not_called() # no mke2fs when it exists
self.assertEqual(vol, out)
def test_creates_volume_when_missing(self):
import tempfile
from subprocess import CompletedProcess
with tempfile.TemporaryDirectory() as td:
vol = Path(td) / "registry.ext4"
with patch.object(infra_vm, "registry_volume_path", return_value=vol), \
patch.object(infra_vm.subprocess, "run",
return_value=CompletedProcess([], 0)) as run:
infra_vm._ensure_registry_volume()
argv = run.call_args.args[0]
self.assertIn("mke2fs", argv)
self.assertIn(str(vol), argv)
class TestEnsureBuilt(unittest.TestCase):
def test_builds_deps_before_infra(self):
with patch.object(infra_vm.docker_mod, "build_image") as build:
infra_vm.ensure_built()
tags = [c.args[0] for c in build.call_args_list]
# infra is FROM gateway and COPY --from orchestrator, so both first.
self.assertEqual(infra_vm._INFRA_IMAGE, tags[-1])
self.assertIn(infra_vm._ORCHESTRATOR_IMAGE, tags[:-1])
self.assertIn(infra_vm._GATEWAY_IMAGE, tags[:-1])
class TestWaitForHealth(unittest.TestCase):
def _infra(self, alive: bool = True) -> infra_vm.InfraVm:
vm = MagicMock()
vm.is_alive.return_value = alive
return infra_vm.InfraVm(vm=vm, guest_ip="10.0.0.1", private_key=Path("/k"))
def test_returns_on_200(self):
infra = self._infra()
cm = MagicMock()
cm.__enter__.return_value.status = 200
with patch.object(infra_vm.urllib.request, "urlopen", return_value=cm):
infra_vm.wait_for_health(infra, timeout=5) # must not raise
def test_dies_when_vm_exits(self):
infra = self._infra(alive=False)
assert infra.vm is not None # narrow for the type checker (it's a mock)
infra.vm.process.returncode = 1
with patch.object(infra_vm.firecracker_vm, "_console_tail", return_value=""), \
self.assertRaises(SystemExit):
infra_vm.wait_for_health(infra, timeout=5)
class TestEnsureRunningSingleton(unittest.TestCase):
def test_adopts_when_healthy(self):
# A healthy control plane + existing key -> adopt (no boot), vm=None.
with patch.object(infra_vm, "_health_ok", return_value=True), \
patch.object(infra_vm, "_infra_dir") as d, \
patch.object(infra_vm, "boot") as boot:
keydir = MagicMock()
(keydir / "id_ed25519").exists.return_value = True
d.return_value = keydir
infra = infra_vm.ensure_running()
boot.assert_not_called()
self.assertIsNone(infra.vm)
def test_boots_when_unhealthy(self):
with patch.object(infra_vm, "_health_ok", return_value=False), \
patch.object(infra_vm, "stop") as stop, \
patch.object(infra_vm, "ensure_built") as built, \
patch.object(infra_vm, "boot") as boot, \
patch.object(infra_vm, "wait_for_health") as wait:
boot.return_value = infra_vm.InfraVm(
guest_ip="10.243.255.1", private_key=Path("/k"), vm=MagicMock())
infra_vm.ensure_running()
stop.assert_called_once() # clear a stale VM first
built.assert_called_once()
boot.assert_called_once()
wait.assert_called_once()
class TestKillPidfile(unittest.TestCase):
def test_noop_when_no_pidfile(self):
import tempfile
with tempfile.TemporaryDirectory() as td:
with patch.object(infra_vm, "_pid_file", return_value=Path(td) / "vm.pid"), \
patch.object(infra_vm.os, "kill") as kill:
infra_vm._kill_pidfile() # must not raise
kill.assert_not_called()
def test_skips_dead_or_recycled_pid(self):
import tempfile
with tempfile.TemporaryDirectory() as td:
pidf = Path(td) / "vm.pid"
pidf.write_text("999999") # a PID that isn't a live firecracker
with patch.object(infra_vm, "_pid_file", return_value=pidf), \
patch.object(infra_vm.os, "kill") as kill:
infra_vm._kill_pidfile()
kill.assert_not_called()
if __name__ == "__main__":
unittest.main()
+8 -7
View File
@@ -8,6 +8,7 @@ from unittest.mock import Mock, patch
from bot_bottle.backend.docker.gateway_provision import (
GatewayProvisionError,
DockerGatewayTransport,
deprovision_git_gate,
provision_git_gate,
)
@@ -54,7 +55,7 @@ class TestProvisionGitGate(unittest.TestCase):
def test_copies_creds_and_runs_namespaced_init(self) -> None:
calls: list[list[str]] = []
with patch(_RUN, side_effect=_recorder(calls)):
provision_git_gate("gw", "bottle1", _plan(_up("foo", known_hosts="/host/kh")))
provision_git_gate(DockerGatewayTransport("gw"), "bottle1", _plan(_up("foo", known_hosts="/host/kh")))
cps = [c for c in calls if c[:2] == ["docker", "cp"]]
self.assertIn(["docker", "cp", "/host/keys/id", "gw:/git-gate/creds/bottle1/foo-key"], cps)
@@ -71,32 +72,32 @@ class TestProvisionGitGate(unittest.TestCase):
def test_omits_known_hosts_copy_when_absent(self) -> None:
calls: list[list[str]] = []
with patch(_RUN, side_effect=_recorder(calls)):
provision_git_gate("gw", "b1", _plan(_up("foo"))) # no known_hosts
provision_git_gate(DockerGatewayTransport("gw"), "b1", _plan(_up("foo"))) # no known_hosts
creds_cps = [c for c in calls if c[:2] == ["docker", "cp"] and "/git-gate/creds/" in c[3]]
self.assertEqual(1, len(creds_cps)) # only the key, not known_hosts
self.assertTrue(creds_cps[0][3].endswith("/foo-key"))
def test_no_upstreams_is_noop(self) -> None:
with patch(_RUN) as m:
provision_git_gate("gw", "b1", _plan())
provision_git_gate(DockerGatewayTransport("gw"), "b1", _plan())
m.assert_not_called()
def test_raises_on_docker_failure(self) -> None:
with patch(_RUN, return_value=_proc(returncode=1, stderr="boom")):
with self.assertRaises(GatewayProvisionError):
provision_git_gate("gw", "b1", _plan(_up("foo")))
provision_git_gate(DockerGatewayTransport("gw"), "b1", _plan(_up("foo")))
def test_rejects_unsafe_bottle_id_before_any_docker(self) -> None:
with patch(_RUN) as m:
with self.assertRaises(GatewayProvisionError):
provision_git_gate("gw", "../etc", _plan(_up("foo")))
provision_git_gate(DockerGatewayTransport("gw"), "../etc", _plan(_up("foo")))
m.assert_not_called() # rejected before a single docker call
class TestDeprovision(unittest.TestCase):
def test_removes_repo_and_creds(self) -> None:
with patch(_RUN, return_value=_proc()) as m:
deprovision_git_gate("gw", "b1")
deprovision_git_gate(DockerGatewayTransport("gw"), "b1")
argv = m.call_args.args[0]
self.assertEqual(["docker", "exec", "gw", "rm", "-rf"], argv[:5])
self.assertIn("/git/b1", argv)
@@ -105,7 +106,7 @@ class TestDeprovision(unittest.TestCase):
def test_rejects_unsafe_bottle_id(self) -> None:
with patch(_RUN) as m:
with self.assertRaises(GatewayProvisionError):
deprovision_git_gate("gw", "a/b")
deprovision_git_gate(DockerGatewayTransport("gw"), "a/b")
m.assert_not_called()
@@ -75,6 +75,19 @@ class TestRenderGitconfig(unittest.TestCase):
out = git_gate_render_gitconfig((_entry(),), "1.2.3.4:9418", scheme="http")
self.assertIn('[url "http://1.2.3.4:9418/repo.git"]', out)
def test_identity_token_extraheader_over_http(self) -> None:
# Delivered as a URL-scoped http.extraHeader so git-http can enforce
# the mandatory (source_ip, token) pair; only over the http transport.
out = git_gate_render_gitconfig(
(_entry(),), "1.2.3.4:9420", scheme="http", identity_token="TOK123")
self.assertIn('[http "http://1.2.3.4:9420/"]', out)
self.assertIn("extraHeader = x-bot-bottle-identity: TOK123", out)
def test_identity_token_omitted_over_git_scheme(self) -> None:
out = git_gate_render_gitconfig(
(_entry(),), "git-gate", scheme="git", identity_token="TOK123")
self.assertNotIn("extraHeader", out)
def test_remote_key_alias_with_nondefault_port(self) -> None:
out = git_gate_render_gitconfig(
(_entry(RemoteKey="10.0.0.5", UpstreamPort="2222"),), "git-gate",
+88 -3
View File
@@ -13,11 +13,19 @@ import threading
import unittest
import urllib.request
from pathlib import Path
from unittest.mock import patch
from bot_bottle.orchestrator.broker import StubBroker
from bot_bottle.orchestrator.control_plane import dispatch, make_server
from bot_bottle.orchestrator.registry import RegistryStore
from bot_bottle.orchestrator.service import Orchestrator
from bot_bottle.store_manager import StoreManager
from bot_bottle.supervise import (
Proposal,
TOOL_EGRESS_ALLOW,
sha256_hex,
write_proposal,
)
def _body(obj: object) -> bytes:
@@ -171,14 +179,26 @@ class TestDispatch(unittest.TestCase):
)
self.assertEqual(400, status)
def test_resolve_without_token_by_source_ip(self) -> None:
_, reg = dispatch(
def test_resolve_without_token_denies(self) -> None:
# Mandatory token: source-IP alone no longer resolves (fail-closed 403).
dispatch(
self.orch, "POST", "/bottles",
_body({"source_ip": "10.243.0.5", "policy": "P"}),
)
status, payload = dispatch(
status, _ = dispatch(
self.orch, "POST", "/resolve", _body({"source_ip": "10.243.0.5"})
)
self.assertEqual(403, status)
def test_resolve_with_matching_token(self) -> None:
_, reg = dispatch(
self.orch, "POST", "/bottles",
_body({"source_ip": "10.243.0.6", "policy": "P"}),
)
status, payload = dispatch(
self.orch, "POST", "/resolve",
_body({"source_ip": "10.243.0.6", "identity_token": reg["identity_token"]}),
)
self.assertEqual(200, status)
self.assertEqual(reg["bottle_id"], payload["bottle_id"])
self.assertEqual("P", payload["policy"])
@@ -223,5 +243,70 @@ class TestServerRoundTrip(unittest.TestCase):
self.assertEqual(reg["bottle_id"], attr["bottle_id"])
class TestDispatchSupervise(unittest.TestCase):
"""The /supervise/* routes over the pure dispatch()."""
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
root = Path(self._tmp.name)
db = root / "db" / "bot-bottle.db"
db.parent.mkdir(parents=True)
self._env = patch.dict("os.environ", {
"BOT_BOTTLE_ROOT": str(root),
"SUPERVISE_DB_PATH": str(db),
})
self._env.start()
self.store = RegistryStore(db)
self.store.migrate()
StoreManager(db).migrate()
secret = secrets.token_bytes(16)
self.orch = Orchestrator(self.store, StubBroker(secret), secret)
def tearDown(self) -> None:
self._env.stop()
self._tmp.cleanup()
def _queue(self, slug: str, proposed: str) -> str:
self.store.register(
"10.243.0.1", metadata=json.dumps({"slug": slug}), policy="routes: []\n")
p = Proposal.new(
bottle_slug=slug, tool=TOOL_EGRESS_ALLOW, proposed_file=proposed,
justification="need it", current_file_hash=sha256_hex(proposed))
write_proposal(p)
return p.id
def test_list_pending(self) -> None:
pid = self._queue("demo", "routes:\n - host: google.com\n")
status, payload = dispatch(self.orch, "GET", "/supervise/proposals", b"")
self.assertEqual(200, status)
proposals = payload["proposals"]
assert isinstance(proposals, list)
self.assertEqual(pid, proposals[0]["id"])
def test_respond_approve_applies_and_clears(self) -> None:
pid = self._queue("demo", "routes:\n - host: google.com\n")
status, payload = dispatch(
self.orch, "POST", "/supervise/respond",
_body({"proposal_id": pid, "bottle_slug": "demo", "decision": "approve"}),
)
self.assertEqual(200, status)
self.assertTrue(payload["responded"])
_, listing = dispatch(self.orch, "GET", "/supervise/proposals", b"")
self.assertEqual([], listing["proposals"])
def test_respond_requires_fields(self) -> None:
status, _ = dispatch(
self.orch, "POST", "/supervise/respond", _body({"decision": "approve"}))
self.assertEqual(400, status)
def test_respond_unknown_proposal_conflicts(self) -> None:
status, payload = dispatch(
self.orch, "POST", "/supervise/respond",
_body({"proposal_id": "ghost", "bottle_slug": "demo", "decision": "approve"}),
)
self.assertEqual(409, status)
self.assertIn("no such proposal", str(payload["error"]))
if __name__ == "__main__":
unittest.main()
+7
View File
@@ -91,6 +91,13 @@ class TestDockerGateway(unittest.TestCase):
self.assertEqual(self.sc.network, runs[0][runs[0].index("--network") + 1])
# Persists its CA on a named volume so agents keep trusting it.
self.assertTrue(any("mitmproxy" in a for a in runs[0]))
# Shares the ONE host DB: the supervise daemon queues into the same
# file the orchestrator + operator (over HTTP) use.
self.assertTrue(any(
a.startswith("SUPERVISE_DB_PATH=") and a.endswith("/run/supervise/bot-bottle.db")
for a in runs[0]))
self.assertTrue(any(
a.endswith(":/run/supervise") for a in runs[0]))
def test_ensure_running_creates_network_when_missing(self) -> None:
calls: list[list[str]] = []
+152 -2
View File
@@ -2,15 +2,26 @@
from __future__ import annotations
import json
import secrets
import tempfile
import unittest
from pathlib import Path
from unittest.mock import patch
from bot_bottle.orchestrator.broker import LaunchBroker, LaunchRequest, StubBroker
from bot_bottle.orchestrator.registry import RegistryStore
from bot_bottle.orchestrator.service import Orchestrator
from bot_bottle.orchestrator.gateway import Gateway
from bot_bottle.store_manager import StoreManager
from bot_bottle.supervise import (
Proposal,
STATUS_APPROVED,
TOOL_EGRESS_ALLOW,
read_response,
sha256_hex,
write_proposal,
)
class _FailingBroker(LaunchBroker):
@@ -114,9 +125,12 @@ class TestOrchestrator(unittest.TestCase):
def test_set_policy_unknown_is_false(self) -> None:
self.assertFalse(self.orch.set_policy("ghost", "{}"))
def test_resolve_by_source_ip_without_token(self) -> None:
def test_resolve_requires_matching_token(self) -> None:
# Mandatory (source_ip, token) pair — no source-IP-only fallback.
rec = self.orch.launch_bottle("10.243.0.1", policy="P")
got = self.orch.resolve("10.243.0.1") # network-layer, no token
self.assertIsNone(self.orch.resolve("10.243.0.1", "")) # empty token denies
self.assertIsNone(self.orch.resolve("10.243.0.1", "wrong")) # mismatch denies
got = self.orch.resolve("10.243.0.1", rec.identity_token) # exact pair
assert got is not None
self.assertEqual(rec.bottle_id, got.bottle_id)
self.assertEqual("P", got.policy)
@@ -152,5 +166,141 @@ class TestOrchestrator(unittest.TestCase):
)
class TestOrchestratorSupervise(unittest.TestCase):
"""Operator-approval flow: the orchestrator applies the decision
server-side against the single DB (queue + policy + audit)."""
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
root = Path(self._tmp.name)
db = root / "db" / "bot-bottle.db"
db.parent.mkdir(parents=True)
# One DB for registry + supervise queue + audit (as in the VM).
self._env = patch.dict("os.environ", {
"BOT_BOTTLE_ROOT": str(root),
"SUPERVISE_DB_PATH": str(db),
})
self._env.start()
self.store = RegistryStore(db)
self.store.migrate()
StoreManager(db).migrate()
secret = secrets.token_bytes(16)
self.orch = Orchestrator(self.store, StubBroker(secret), secret)
def tearDown(self) -> None:
self._env.stop()
self._tmp.cleanup()
def _register(self, slug: str, policy: str) -> str:
rec = self.store.register(
"10.243.0.1", metadata=json.dumps({"slug": slug}), policy=policy)
return rec.bottle_id
def _queue(self, slug: str, proposed: str) -> str:
p = Proposal.new(
bottle_slug=slug, tool=TOOL_EGRESS_ALLOW, proposed_file=proposed,
justification="need it", current_file_hash=sha256_hex(proposed))
write_proposal(p)
return p.id
def test_pending_lists_queued_proposal(self) -> None:
self._register("demo", "routes: []\n")
pid = self._queue("demo", "routes:\n - host: google.com\n")
pending = self.orch.supervise_pending()
self.assertEqual(1, len(pending))
self.assertEqual(pid, pending[0]["id"])
self.assertEqual("demo", pending[0]["bottle_slug"])
def test_approve_applies_policy_writes_response_and_clears_pending(self) -> None:
bottle_id = self._register("demo", "routes:\n - host: existing.com\n")
new_routes = "routes:\n - host: google.com\n"
pid = self._queue("demo", new_routes)
ok, err = self.orch.supervise_respond(
pid, bottle_slug="demo", decision="approve")
self.assertTrue(ok, err)
# policy live-applied so /resolve serves the new routes
rec = self.store.get(bottle_id)
assert rec is not None
self.assertEqual(new_routes, rec.policy)
# response written -> agent unblocks, proposal no longer pending
self.assertEqual(STATUS_APPROVED, read_response("demo", pid).status)
self.assertEqual([], self.orch.supervise_pending())
def test_pending_carries_human_label(self) -> None:
# The proposal is keyed by bottle_id, but pending dicts also expose the
# bottle's human slug so the operator sees a name, not a hex id.
bottle_id = self._register("codex-dev-a1b2c", "routes: []\n")
self._queue(bottle_id, "routes:\n - host: google.com\n")
pending = self.orch.supervise_pending()
self.assertEqual(bottle_id, pending[0]["bottle_slug"])
self.assertEqual("codex-dev-a1b2c", pending[0]["bottle_label"])
def test_pending_label_falls_back_to_slug_when_bottle_gone(self) -> None:
# No registry record (torn down): label is the id, never empty.
self._queue("ghost-id", "routes:\n - host: google.com\n")
pending = self.orch.supervise_pending()
self.assertEqual("ghost-id", pending[0]["bottle_label"])
def test_approve_by_bottle_id_applies_policy(self) -> None:
# Consolidated reality: the supervise server keys each proposal by the
# orchestrator-assigned bottle_id, not the human slug. Approval must
# resolve the record by that id and apply the policy (regression for
# the "bottle <id> is no longer registered" 409).
bottle_id = self._register("codex-dev-a1b2c", "routes: []\n")
new_routes = "routes:\n - host: google.com\n"
pid = self._queue(bottle_id, new_routes)
ok, err = self.orch.supervise_respond(
pid, bottle_slug=bottle_id, decision="approve")
self.assertTrue(ok, err)
rec = self.store.get(bottle_id)
assert rec is not None
self.assertEqual(new_routes, rec.policy)
def test_modify_applies_final_file_not_proposed(self) -> None:
bottle_id = self._register("demo", "routes: []\n")
pid = self._queue("demo", "routes:\n - host: google.com\n")
edited = "routes:\n - host: example.com\n"
ok, _ = self.orch.supervise_respond(
pid, bottle_slug="demo", decision="modify", final_file=edited)
self.assertTrue(ok)
rec = self.store.get(bottle_id)
assert rec is not None
self.assertEqual(edited, rec.policy)
def test_reject_leaves_policy_unchanged(self) -> None:
bottle_id = self._register("demo", "routes:\n - host: existing.com\n")
pid = self._queue("demo", "routes:\n - host: google.com\n")
ok, _ = self.orch.supervise_respond(
pid, bottle_slug="demo", decision="reject", notes="no")
self.assertTrue(ok)
rec = self.store.get(bottle_id)
assert rec is not None
self.assertEqual("routes:\n - host: existing.com\n", rec.policy)
self.assertEqual("rejected", read_response("demo", pid).status)
def test_unknown_proposal_is_error(self) -> None:
ok, err = self.orch.supervise_respond(
"ghost", bottle_slug="demo", decision="approve")
self.assertFalse(ok)
self.assertIn("no such proposal", err)
def test_unknown_decision_is_error(self) -> None:
self._register("demo", "routes: []\n")
pid = self._queue("demo", "routes:\n - host: google.com\n")
ok, err = self.orch.supervise_respond(
pid, bottle_slug="demo", decision="bogus")
self.assertFalse(ok)
self.assertIn("unknown decision", err)
def test_approve_when_bottle_gone_cannot_apply(self) -> None:
# Proposal queued but the bottle was torn down before the operator
# acted: an egress apply has no target, so respond fails closed.
pid = self._queue("ghost-bottle", "routes:\n - host: google.com\n")
ok, err = self.orch.supervise_respond(
pid, bottle_slug="ghost-bottle", decision="approve")
self.assertFalse(ok)
self.assertIn("no longer registered", err)
if __name__ == "__main__":
unittest.main()
+123 -171
View File
@@ -1,31 +1,24 @@
"""Unit: supervise headless paths (PRD 0013 phase 4, PRD 0016).
"""Unit: supervise headless paths — the discovery + approve/reject that the
TUI key handlers call into.
The curses TUI itself isn't exercised here — these tests cover the
discovery + approve/reject paths that the TUI's key handlers call into.
These go through the orchestrator HTTP client now (the operator never
touches the DB directly), so the client is mocked here; the server-side
apply / response / audit is covered in test_orchestrator_service.
"""
import os
import tempfile
import unittest
from datetime import datetime, timezone
from pathlib import Path
from unittest.mock import patch
from unittest.mock import MagicMock, patch
from bot_bottle import supervise
from tests.unit import use_bottle_root
from bot_bottle.audit_store import AuditStore
from bot_bottle.cli import supervise as supervise_cli
from bot_bottle.queue_store import QueueStore
from bot_bottle.supervise import (
Proposal,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
read_audit_entries,
read_response,
sha256_hex,
)
@@ -33,198 +26,131 @@ from bot_bottle.supervise import (
FIXED = datetime(2026, 5, 25, 12, 0, 0, tzinfo=timezone.utc)
def _proposal(slug: str = "dev", tool: str = TOOL_EGRESS_ALLOW) -> Proposal:
def _proposal(slug: str = "dev", tool: str = TOOL_EGRESS_ALLOW,
*, now: datetime = FIXED) -> Proposal:
payloads = {
supervise.TOOL_EGRESS_ALLOW: "routes:\n - host: example.com\n",
supervise.TOOL_EGRESS_BLOCK: "routes:\n - host: example.com\n",
TOOL_EGRESS_ALLOW: "routes:\n - host: example.com\n",
TOOL_EGRESS_BLOCK: "routes:\n - host: example.com\n",
TOOL_GITLEAKS_ALLOW: "file: tests/test_fixture.py\nline: 3\n",
TOOL_EGRESS_TOKEN_ALLOW: "host: api.example.com\ndetector: token\n",
}
payload = payloads.get(tool, "")
return Proposal.new(
bottle_slug=slug, tool=tool,
proposed_file=payload,
justification=f"needed for {slug}",
current_file_hash=sha256_hex(payload),
now=FIXED,
bottle_slug=slug, tool=tool, proposed_file=payload,
justification=f"needed for {slug}", current_file_hash=sha256_hex(payload),
now=now,
)
class _FakeHomeMixin:
"""Point bot_bottle_root at a temp dir (via BOT_BOTTLE_ROOT) for the test."""
class _ClientMixin:
"""Install a mock orchestrator client as the CLI-session singleton."""
def _setup_fake_home(self):
self._tmp = tempfile.TemporaryDirectory(prefix="supervise-test.")
self._restore_home = use_bottle_root(Path(self._tmp.name) / ".bot-bottle")
QueueStore("").migrate()
AuditStore().migrate()
def _teardown_fake_home(self):
self._restore_home()
self._tmp.cleanup()
def _install_client(self, pending: "list[Proposal] | None" = None) -> MagicMock:
client = MagicMock()
client.supervise_pending.return_value = [
p.to_dict() for p in (pending or [])
]
patcher = patch.object(supervise_cli, "_client", return_value=client)
patcher.start()
self.addCleanup(patcher.stop) # type: ignore[attr-defined]
self.addCleanup( # type: ignore[attr-defined]
lambda: setattr(supervise_cli, "_client_instance", None))
return client
class TestDiscoverPending(_FakeHomeMixin, unittest.TestCase):
def setUp(self):
self._setup_fake_home()
def tearDown(self):
self._teardown_fake_home()
def test_empty_when_no_queues(self):
class TestDiscoverPending(_ClientMixin, unittest.TestCase):
def test_empty(self) -> None:
self._install_client([])
self.assertEqual([], supervise_cli.discover_pending())
def test_walks_all_slug_subdirs(self):
for slug in ("dev", "api"):
supervise.write_proposal(_proposal(slug=slug))
def test_lists_all_bottles(self) -> None:
self._install_client([_proposal("dev"), _proposal("api")])
pending = supervise_cli.discover_pending()
self.assertEqual({"dev", "api"}, {qp.proposal.bottle_slug for qp in pending})
self.assertEqual(
{"dev", "api"}, {qp.proposal.bottle_slug for qp in pending})
def test_sorted_by_arrival_across_bottles(self):
early = Proposal.new(
bottle_slug="api", tool=TOOL_EGRESS_ALLOW,
proposed_file="routes:\n - host: early.example.com\n", justification="early",
current_file_hash="h",
now=datetime(2026, 5, 25, 10, 0, 0, tzinfo=timezone.utc),
)
late = Proposal.new(
bottle_slug="dev", tool=TOOL_EGRESS_ALLOW,
proposed_file="routes:\n - host: late.example.com\n", justification="late",
current_file_hash="h",
now=datetime(2026, 5, 25, 14, 0, 0, tzinfo=timezone.utc),
)
for p in (late, early):
supervise.write_proposal(p)
def test_sorted_by_arrival(self) -> None:
early = _proposal(
"api", now=datetime(2026, 5, 25, 10, 0, 0, tzinfo=timezone.utc))
late = _proposal(
"dev", now=datetime(2026, 5, 25, 14, 0, 0, tzinfo=timezone.utc))
self._install_client([late, early])
pending = supervise_cli.discover_pending()
self.assertEqual([early.id, late.id], [qp.proposal.id for qp in pending])
def test_excludes_already_responded(self):
p = _proposal()
supervise.write_proposal(p)
supervise.write_response("dev", supervise.Response(
proposal_id=p.id, status=STATUS_APPROVED, notes="",
))
self.assertEqual([], supervise_cli.discover_pending())
def test_label_comes_from_bottle_label(self) -> None:
# The server tags each dict with the human slug; the CLI displays it
# while the proposal stays keyed by the opaque bottle_id.
client = MagicMock()
d = _proposal("3601cbe883c2786d").to_dict()
d["bottle_label"] = "codex-dev-a1b2c"
client.supervise_pending.return_value = [d]
with patch.object(supervise_cli, "_client", return_value=client):
pending = supervise_cli.discover_pending()
self.assertEqual("codex-dev-a1b2c", pending[0].label)
self.assertEqual("3601cbe883c2786d", pending[0].proposal.bottle_slug)
def test_label_falls_back_to_slug_when_absent(self) -> None:
# Legacy dicts without bottle_label (e.g. an older orchestrator).
self._install_client([_proposal("dev")])
self.assertEqual("dev", supervise_cli.discover_pending()[0].label)
class TestApproveReject(_FakeHomeMixin, unittest.TestCase):
def setUp(self):
self._setup_fake_home()
class TestApproveReject(_ClientMixin, unittest.TestCase):
def _qp(self, tool: str = TOOL_EGRESS_ALLOW) -> "supervise_cli.QueuedProposal":
return supervise_cli.QueuedProposal(proposal=_proposal(tool=tool))
def tearDown(self):
self._teardown_fake_home()
def _enqueue(self, tool: str = TOOL_EGRESS_ALLOW):
p = _proposal(tool=tool)
supervise.write_proposal(p)
return supervise_cli.QueuedProposal(proposal=p)
def test_approve_writes_response(self):
qp = self._enqueue()
with patch(
"bot_bottle.cli.supervise.apply_routes_change",
return_value=("routes: []\n", "routes:\n - host: example.com\n"),
):
supervise_cli.approve(qp)
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
self.assertEqual(STATUS_APPROVED, resp.status)
self.assertIsNone(resp.final_file)
def test_approve_with_final_file_marks_modified(self):
qp = self._enqueue()
with patch(
"bot_bottle.cli.supervise.apply_routes_change",
return_value=("routes: []\n", "routes:\n - host: edited.example.com\n"),
):
supervise_cli.approve(
qp,
final_file="routes:\n - host: edited.example.com\n",
notes="tweaked",
)
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
self.assertEqual(STATUS_MODIFIED, resp.status)
self.assertEqual("routes:\n - host: edited.example.com\n", resp.final_file)
self.assertEqual("tweaked", resp.notes)
def test_reject_writes_rejection(self):
qp = self._enqueue()
supervise_cli.reject(qp, reason="nope")
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
self.assertEqual(STATUS_REJECTED, resp.status)
self.assertEqual("nope", resp.notes)
def test_approve_egress_block_writes_audit_log(self):
qp = self._enqueue(tool=supervise.TOOL_EGRESS_BLOCK)
with patch(
"bot_bottle.cli.supervise.apply_routes_change",
return_value=("routes: []\n", "routes:\n - host: example.com\n"),
) as apply_routes_change:
supervise_cli.approve(qp)
apply_routes_change.assert_called_once_with(
"dev",
"routes:\n - host: example.com\n",
def test_approve_calls_respond(self) -> None:
client = self._install_client()
qp = self._qp()
supervise_cli.approve(qp)
client.supervise_respond.assert_called_once_with(
qp.proposal.id, bottle_slug="dev", decision="approve",
notes="", final_file=None,
)
entries = read_audit_entries("egress", "dev")
self.assertEqual(1, len(entries))
self.assertEqual(STATUS_APPROVED, entries[0].operator_action)
self.assertEqual("needed for dev", entries[0].justification)
def test_approve_gitleaks_allow_leaves_response_for_gate(self):
qp = self._enqueue(tool=TOOL_GITLEAKS_ALLOW)
supervise_cli.approve(qp, notes="dummy fixture")
# Gate polls the DB for the response; TUI must not archive it.
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
self.assertEqual(STATUS_APPROVED, resp.status)
self.assertEqual("dummy fixture", resp.notes)
def test_modify_sets_decision_and_final_file(self) -> None:
client = self._install_client()
qp = self._qp()
edited = "routes:\n - host: edited.example.com\n"
supervise_cli.approve(qp, final_file=edited, notes="tweaked")
client.supervise_respond.assert_called_once_with(
qp.proposal.id, bottle_slug="dev", decision="modify",
notes="tweaked", final_file=edited,
)
def test_tui_gitleaks_allow_requires_reason(self):
qp = self._enqueue(tool=TOOL_GITLEAKS_ALLOW)
def test_reject_calls_respond(self) -> None:
client = self._install_client()
qp = self._qp()
supervise_cli.reject(qp, reason="nope")
client.supervise_respond.assert_called_once_with(
qp.proposal.id, bottle_slug="dev", decision="reject", notes="nope",
)
def test_tui_report_only_requires_reason(self) -> None:
self._install_client()
qp = self._qp(tool=TOOL_GITLEAKS_ALLOW)
with patch.object(supervise_cli, "_prompt", return_value=""):
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
self.assertEqual("approve aborted (empty reason)", status)
def test_tui_gitleaks_allow_writes_reason(self):
qp = self._enqueue(tool=TOOL_GITLEAKS_ALLOW)
def test_tui_report_only_writes_reason(self) -> None:
client = self._install_client()
qp = self._qp(tool=TOOL_GITLEAKS_ALLOW)
with patch.object(supervise_cli, "_prompt", return_value="test fixture"):
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
self.assertIn("approved gitleaks-allow", status)
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
self.assertEqual("test fixture", resp.notes)
client.supervise_respond.assert_called_once()
self.assertEqual(
"test fixture", client.supervise_respond.call_args.kwargs["notes"])
def test_approve_token_allow_leaves_response_for_egress(self):
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
supervise_cli.approve(qp, notes="false positive")
# The egress addon polls the DB for the response; the TUI must
# not archive it (the addon archives after reading).
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
self.assertEqual(STATUS_APPROVED, resp.status)
self.assertEqual("false positive", resp.notes)
def test_token_allow_writes_no_audit_log(self):
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
supervise_cli.approve(qp, notes="false positive")
self.assertEqual([], read_audit_entries("egress", "dev"))
def test_tui_token_allow_requires_reason(self):
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
with patch.object(supervise_cli, "_prompt", return_value=""):
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
self.assertEqual("approve aborted (empty reason)", status)
def test_tui_token_allow_writes_reason(self):
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
with patch.object(supervise_cli, "_prompt", return_value="legit"):
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
self.assertIn("approved egress-token-allow", status)
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
self.assertEqual("legit", resp.notes)
def test_suffix_for_token_allow_is_txt(self):
self.assertEqual(".txt", supervise_cli._suffix_for_tool(TOOL_EGRESS_TOKEN_ALLOW))
def test_suffix_for_token_allow_is_txt(self) -> None:
self.assertEqual(
".txt", supervise_cli._suffix_for_tool(TOOL_EGRESS_TOKEN_ALLOW))
class TestEditInEditor(unittest.TestCase):
def test_runs_editor_returns_edited_content(self):
def test_runs_editor_returns_edited_content(self) -> None:
original_editor = os.environ.get("EDITOR")
try:
with tempfile.NamedTemporaryFile(
@@ -245,7 +171,7 @@ class TestEditInEditor(unittest.TestCase):
else:
os.environ["EDITOR"] = original_editor
def test_returns_none_when_unchanged(self):
def test_returns_none_when_unchanged(self) -> None:
original_editor = os.environ.get("EDITOR")
try:
with tempfile.NamedTemporaryFile(
@@ -267,5 +193,31 @@ class TestEditInEditor(unittest.TestCase):
os.environ["EDITOR"] = original_editor
class TestResolveOrchestratorUrl(unittest.TestCase):
"""`_resolve_orchestrator_url` starts the backend orchestrator on demand
when discovery finds nothing supervise is often the first thing run."""
def test_returns_discovered_url_without_starting(self) -> None:
with patch.object(
supervise_cli, "discover_orchestrator_url",
return_value="http://127.0.0.1:8099",
), patch("bot_bottle.backend.get_bottle_backend") as get_backend:
url = supervise_cli._resolve_orchestrator_url()
self.assertEqual(url, "http://127.0.0.1:8099")
get_backend.assert_not_called() # nothing to start; discovery won
def test_starts_backend_orchestrator_when_none_running(self) -> None:
backend = MagicMock()
backend.name = "firecracker"
backend.ensure_orchestrator.return_value = "http://10.243.255.1:8099"
with patch.object(
supervise_cli, "discover_orchestrator_url",
side_effect=supervise_cli.OrchestratorClientError("none"),
), patch("bot_bottle.backend.get_bottle_backend", return_value=backend):
url = supervise_cli._resolve_orchestrator_url()
self.assertEqual(url, "http://10.243.255.1:8099")
backend.ensure_orchestrator.assert_called_once_with()
if __name__ == "__main__":
unittest.main()
@@ -56,6 +56,15 @@ class _FakeHomeMixin:
class TestCmdSuperviseErrorPaths(_FakeHomeMixin, unittest.TestCase):
def setUp(self):
self._setup_fake_home()
# `cmd_supervise` establishes the orchestrator client up front; these
# tests exercise the curses / crash-logging paths that run *after*
# that, so stub the client. Otherwise the outcome depends on whether a
# live orchestrator happens to be reachable (CI has none, so the
# up-front connect would error and short-circuit before curses).
client_patch = mock.patch.object(
supervise_cli, "_client", return_value=mock.MagicMock())
client_patch.start()
self.addCleanup(client_patch.stop)
def tearDown(self):
self._teardown_fake_home()
+52 -1
View File
@@ -631,9 +631,15 @@ class TestHttpEndToEnd(unittest.TestCase):
class _FakeResolver:
def __init__(self, bottle_id: str | None = None, raises: bool = False) -> None:
def __init__(
self,
bottle_id: str | None = None,
raises: bool = False,
policy: str = "",
) -> None:
self._bottle_id = bottle_id
self._raises = raises
self._policy = policy
self.calls: list[str] = []
def resolve_bottle_id(self, source_ip: str, identity_token: str = "") -> str | None:
@@ -645,6 +651,15 @@ class _FakeResolver:
raise supervise_server.PolicyResolveError("orchestrator down")
return self._bottle_id
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = "",
) -> "tuple[str, str | None, dict[str, str]]":
del identity_token
self.calls.append(source_ip)
if self._raises:
raise supervise_server.PolicyResolveError("orchestrator down")
return self._policy, self._bottle_id, {}
def _handler(resolver: object) -> MCPHandler:
"""A bare MCPHandler wired with a server (carrying the resolver) and a
@@ -682,5 +697,41 @@ class TestAttributedConfig(unittest.TestCase):
)
class TestResolvedRoutesPayload(unittest.TestCase):
"""`list-egress-routes` answers from the calling bottle's resolved policy in
consolidated mode not the gateway's empty static table. Regression: an
empty list led agents to propose replace-all route files that dropped base
hosts like api.anthropic.com on approval."""
def test_returns_resolved_bottle_routes(self) -> None:
policy = (
"routes:\n"
" - host: api.anthropic.com\n"
" - host: www.google.com\n"
)
payload = _handler(
_FakeResolver(bottle_id="b1", policy=policy)
)._resolved_routes_payload()
assert payload is not None
self.assertFalse(payload["isError"]) # type: ignore[index]
data = json.loads(payload["content"][0]["text"]) # type: ignore[index]
hosts = {r["host"] for r in data["routes"]}
self.assertEqual({"api.anthropic.com", "www.google.com"}, hosts)
def test_orchestrator_error_fails_closed_to_empty(self) -> None:
# resolve_client_context swallows resolver errors → deny-all (empty),
# never another bottle's routes.
payload = _handler(
_FakeResolver(raises=True)
)._resolved_routes_payload()
assert payload is not None
data = json.loads(payload["content"][0]["text"]) # type: ignore[index]
self.assertEqual([], data["routes"])
def test_single_tenant_returns_none(self) -> None:
# No resolver → caller falls back to the static introspection endpoint.
self.assertIsNone(_handler(None)._resolved_routes_payload())
if __name__ == "__main__":
unittest.main()