Compare commits

..

105 Commits

Author SHA1 Message Date
didericis 4a607ad098 refactor(macos): one infra container (control plane + gateway), fixes shared-DB races
lint / lint (push) Successful in 2m15s
test / unit (pull_request) Successful in 1m16s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m17s
Adopts the firecracker infra-VM pattern for macOS: the orchestrator control
plane and the gateway data plane now run in a SINGLE Apple container instead of
two. Apple Containers are lightweight VMs with separate kernels, so the prior
two-container design had both guests writing one bot-bottle.db over virtiofs,
where fcntl locks are not coherent across kernels — concurrent writes (the
orchestrator's registry vs the gateway supervise daemon's queue) could corrupt
it. One container = one kernel = coherent locking.

The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never
bind-mounted from the host, so no host process opens the live file either. The
host CLI already reaches registry + supervise state over the control-plane HTTP
surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's
VM-only DB requires.

Two simplifications fall out of the single container:
- No DNS dance: the control plane and gateway daemons reach each other over
  127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for
  Apple having no container DNS) is gone, along with the moved-IP recreate
  logic it needed.
- Net -243 lines.

Mechanics: the infra container runs from the gateway image with the
control-plane source bind-mounted read-only (like the docker orchestrator, so a
code change needs no rebuild) and a small sh -c init that starts both processes
(mirrors firecracker's _infra_init). Also implements the macOS backend's
ensure_orchestrator() and adds it to discover_orchestrator_url, so operator
tools (supervise) can bring up / find the control plane on demand — previously
the macOS backend died with "no orchestrator control plane".

Verified end-to-end on real Apple Container 1.0.0: the single infra container
comes up healthy (one address for control plane + gateway), both processes run,
the DB is written on the container-only volume, host-side supervise works over
HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied
one. 1824 unit tests pass with `container` absent (CI parity), pyright clean,
pylint 9.89.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 04:14:14 -04:00
didericis e24b62b6b9 fix(macos): review fixes — token on plan, self-heal, symmetric digest, DHCP poll
lint / lint (push) Successful in 2m18s
test / unit (pull_request) Successful in 1m6s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m18s
Addresses findings from a high-effort review of the PRD 0070 macOS backend.

Correctness:
- Stamp identity_token onto MacosContainerBottlePlan after registration. git's
  gitconfig extraHeader and the supervise MCP --header read
  getattr(plan,"identity_token","") at provision time, and both reach the
  gateway on NO_PROXY (bypassing the egress proxy that carries the token). The
  plan never carried it, so /resolve fail-closed and every git fetch/push and
  supervise call from a macOS bottle would have been denied. Registration
  precedes provision(), so — unlike the run-time env — the plan can carry it.
- Self-heal the orchestrator: recreate when it is not (source-current AND
  answering /health), not on the source-hash label alone. A container running
  current code but with a wedged HTTP server was left alone and polled to
  death, failing every launch until manual deletion.
- image_digest and container_image_digest now read the same descriptor.digest
  field; dropped image_digest's id/tag fallback that could yield a value the
  container side can't produce — a permanent mismatch would have recreated the
  shared gateway on every launch (severing every live bottle's egress, since
  the replacement gets a new DHCP address).
- Poll for the agent's and gateway's DHCP address instead of a fatal read
  right after `container run` (there is no --ip; the address can lag start).

Cleanup:
- One _inspect_first + _descriptor_digest behind the four inspect readers.
- Shared bind_mount_spec (util) and host_db_dir (paths) replace per-module
  copies; _GIT_HTTP_PORT now imports git_http_backend.DEFAULT_PORT.
- Drop the dead _url cache / url property and the write-only agent_proxy_url.

Deferred (noted on the PR, not fixed here): the gateway image rebuilding on
every launch (needs source-hash-labeled build), SQLite shared across VM
guests, and the sh -lc profile-override edge — each is design-level or
behavior-risk beyond a review fix.

Verified: real Apple Container bring-up is green and idempotent; 1826 unit
tests pass with `container` absent (CI parity), pyright clean, pylint 9.86.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-17 03:26:33 -04:00
didericis a5910696a5 fix(test): stop the macOS unit tests shelling out to the container CLI
lint / lint (push) Successful in 2m26s
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 21s
test / coverage (pull_request) Successful in 1m20s
CI's unit + coverage jobs failed with `FileNotFoundError: 'container'`: three
tests reached the real Apple CLI, which exists on a macOS dev host but not on
the Linux runner. They passed locally for that reason alone — and two of them
were quietly creating real Apple networks on the dev host as a side effect.

- `test_enumerate_active_is_empty_while_disabled` asserted the disabled-era
  stub and called `enumerate_active()` unmocked. The backend launches bottles
  again, so it now covers the real enumeration: slug parsing, exclusion of the
  shared gateway/orchestrator singletons, and the CLI-failure path.
- The two orchestrator tests patched `orchestrator_service.container_mod`, but
  `_run_orchestrator_container` reaches the CLI through `ensure_networks`,
  which is imported from the gateway module and resolves `container_mod` in
  *its* namespace. Patch the imported name instead.

Adds a test that the networks exist before the orchestrator runs — the
ordering the escaped call was hiding.

Verified by reproducing the CI environment locally (`PATH` without the
`container` binary): 3 failures before, 1818 passing after.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 01:27:40 -04:00
didericis c69642e568 feat(macos): consolidated per-host gateway for the Apple backend (PRD 0070)
Re-enables the macos-container backend on the shared per-host orchestrator +
gateway, replacing the per-bottle companion container removed in #385. This is
the last backend in PRD 0070's roadmap.

Apple Container 1.0.0 forced three departures from the docker shape, each
verified against the live CLI (findings recorded in the networking spike):

- No `--ip`. The address is DHCP-assigned and knowable only once the container
  runs, so the order inverts: gateway up -> run agent -> read its address ->
  register. The identity token is minted by registration and therefore cannot
  be in the agent's run-time env; it rides the proxy URL applied at
  `container exec` time (bare `--env` names keep it off argv).
- No container DNS. The gateway can only be handed the control plane's IP, so
  the orchestrator starts first and the gateway is pointed at its address.
- No `network connect`. Networks are fixed at run time, so the shared host-only
  network is created up front; per-bottle networks would restart the gateway
  on every launch and defeat the consolidation.

The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default,
which would let an agent forge a neighbour's source address on the shared
segment. NET_ADMIN is already absent, so this closes the source-address half of
PRD 0070's attribution invariant.

Verified end-to-end on real Apple Container 1.0.0: both images build, the
control plane comes up healthy, the gateway reaches it by IP, and a registered
agent gets 200 for a host in its routes and 403 for one outside them. Bring-up
is idempotent — a second launch does not churn the singletons.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-17 01:27:40 -04:00
didericis dfc693e0b6 fix(firecracker): keep the snapshot partial private even if one was left behind
test / unit (pull_request) Successful in 1m9s
test / integration (pull_request) Successful in 21s
test / coverage (pull_request) Successful in 1m14s
lint / lint (push) Successful in 2m21s
test / unit (push) Successful in 1m18s
test / integration (push) Successful in 30s
test / coverage (push) Successful in 1m27s
Update Quality Badges / update-badges (push) Successful in 1m17s
Follow-up to the codex review on #398. os.open's mode arg only applies on
creation, so a committed-rootfs.tar.partial left 0644 by an interrupted run
would be opened/truncated (not re-moded) and stay world-readable for the
whole SSH stream. Unlink any leftover and exclusively recreate it
(O_EXCL|O_NOFOLLOW), then fchmod 0600 immediately so umask can't loosen it.

Test pre-creates a 0644 partial and asserts the fd is 0600 mid-stream.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 01:02:19 -04:00
didericis 39d47b8108 fix(firecracker): harden committed-snapshot resume against guest-controlled data
lint / lint (push) Successful in 2m14s
test / unit (pull_request) Successful in 1m7s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m17s
Address the codex review on #398:

- P1: inject_guest_boot no longer follows a symlink at bb-init/bb-dropbear.
  A committed snapshot is guest-controlled and could plant those paths as
  symlinks aimed at a host file (e.g. bb-init -> ~/.bashrc); write_text /
  copy2 would then overwrite the target as the host user during resume.
  Replace any pre-existing entry and create the files with
  O_EXCL|O_NOFOLLOW so the write stays inside the staging tree.

- P2: write the snapshot tar owner-only (0600). It can contain the bottle's
  private workspace; it was being created world-readable (0644).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:44:06 -04:00
didericis d0a0ce8d60 test(firecracker): satisfy pyright strict + line length in committed-rootfs tests
lint / lint (push) Successful in 2m22s
test / unit (pull_request) Successful in 1m20s
test / integration (pull_request) Successful in 28s
test / coverage (pull_request) Successful in 1m20s
Annotate the counting_run subprocess.run wrapper (reportMissingParameterType)
and wrap an over-long patch target line.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:36:43 -04:00
didericis 5c08701983 feat(firecracker): port freeze/migrate off host Docker (PRD 0069 / #397)
lint / lint (push) Failing after 2m9s
test / unit (pull_request) Successful in 1m7s
test / integration (pull_request) Successful in 24s
test / coverage (pull_request) Successful in 1m22s
The last host-Docker dependency in the Firecracker launch path. Freeze
and resume no longer touch the docker daemon, so the backend needs
firecracker + KVM only — completing #348.

Freeze: stream the guest rootfs over SSH straight into a persistent
committed-rootfs.tar (the resumable/migratable artifact) instead of
round-tripping through `docker build` from a scratch image. Written to
a .partial sibling and atomically renamed so a failed freeze leaves no
truncated artifact.

Resume: extract the snapshot tar into a cached base dir and feed it to
the existing rootless `mke2fs -d` pipeline, replacing the
`docker create` + `docker export | tar` path. Recreate the
proc/sys/dev/run mount points the freezer excludes so the guest init
can mount them.

`util.build_base_rootfs_dir` / `docker_image_id` stay — they still back
the opt-in BOT_BOTTLE_INFRA_BUILD=local dev path and off-host
publish_infra, which are out of scope.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:31:00 -04:00
didericis e3e195f866 chore(firecracker): name the artifact package bot-bottle-firecracker-infra
test / unit (pull_request) Successful in 1m19s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m18s
lint / lint (push) Successful in 2m17s
test / unit (push) Successful in 1m18s
test / integration (push) Successful in 32s
test / coverage (push) Successful in 1m21s
Update Quality Badges / update-badges (push) Successful in 1m17s
Rename the Gitea generic package from bot-bottle-infra to
bot-bottle-firecracker-infra so it's self-evident in the package list which
backend it serves (and leaves room for other artifacts, e.g. a shipped
kernel). The version slot stays the content hash — "firecracker" belongs in
the package name, not the version. Docker image / VM names are unchanged.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-17 00:04:48 -04:00
didericis e3d24b7e41 chore(firecracker): ship an about.txt description with the infra artifact
lint / lint (push) Successful in 2m55s
test / unit (pull_request) Successful in 1m47s
test / integration (pull_request) Successful in 38s
test / coverage (pull_request) Successful in 1m28s
Generic packages have no description field, so publish_infra now uploads a
short about.txt alongside the rootfs on every publish — it's what identifies
the package as the Firecracker backend's infra rootfs on the package page.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:59:38 -04:00
didericis f2891a1634 fix(firecracker): hash all baked-in files + stream the artifact upload
lint / lint (push) Successful in 2m26s
test / unit (pull_request) Successful in 1m44s
test / integration (pull_request) Successful in 40s
test / coverage (pull_request) Successful in 1m32s
Address PR #395 review (two P1s):

- Version hash covered only `bot_bottle/**.py`, but the image `COPY`s the
  whole package — non-Python inputs baked in (egress_entrypoint.sh,
  netpool.defaults.env) didn't change the version, so a launch host could
  boot a stale rootfs whose code differs from its checkout. Hash every
  regular file under bot_bottle/ (excluding __pycache__/.pyc). Regression
  tests: a shell-script change bumps the version; .pyc/__pycache__ don't.

- publish_infra `_put` read the whole (hundreds-of-MB) gz into memory via
  read_bytes(). Stream it from disk with an explicit Content-Length; the
  tiny .sha256 stays in-memory. Test asserts the body is the file object,
  not bytes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:46:46 -04:00
didericis 8d8a88aeeb fix(firecracker): read only BOT_BOTTLE_INFRA_ARTIFACT_TOKEN for the artifact
lint / lint (push) Successful in 2m37s
test / unit (pull_request) Successful in 1m25s
test / integration (pull_request) Successful in 31s
test / coverage (pull_request) Successful in 1m31s
Drop the fallback to the general-purpose BOT_BOTTLE_CLAUDE_GITEA_TOKEN so
the artifact pull/publish uses a dedicated, package-scoped token that can
be granted (or revoked) independently of the general Gitea token.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:41:53 -04:00
didericis 18f190b7e3 feat(firecracker): pull the infra rootfs as a prebuilt artifact (PRD 0069 Stage 2)
Stage 2 of the docker-free Firecracker backend (#348): stop building the
fixed infra image on the launch host. The infra VM's rootfs is host- and
bottle-agnostic (authorized_keys + guest IP ride the kernel cmdline, not the
rootfs), so it's built once off-host and published as a versioned, ready-to-
boot ext4; the launch host downloads + verifies + boots it — no Docker, no
image tooling, just HTTP + gunzip.

- infra_artifact.py: version = content hash of the rootfs inputs (the shipped
  bot_bottle package + the three Dockerfiles + the init), so a launch host
  pulls the artifact matching its code and a content change can't silently
  boot a stale rootfs. Pull + sha256-verify (fail-closed) + gunzip from a
  Gitea generic package; base/owner/token configurable, default this Gitea.
- infra_vm.ensure_built/boot default to the pull path; BOT_BOTTLE_INFRA_BUILD=
  local keeps the docker build-from-source path for iterating on Dockerfiles.
- publish_infra.py: the off-host half — builds the images with Docker, mke2fs
  the rootfs (with buildah slack), gzips, and PUTs it to the generic package.

Rollout note: default=pull means a launch 404s until an artifact is published;
until the Gitea packages endpoint is enabled + an artifact published, use
BOT_BOTTLE_INFRA_BUILD=local. Freeze/migrate's remaining docker use is a
separate PR.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 23:41:53 -04:00
didericis-claude bbb8913382 refactor(git-gate): centralize hostname qualification in globalize_slug
test / unit (pull_request) Successful in 1m19s
test / integration (pull_request) Successful in 24s
test / coverage (pull_request) Successful in 1m24s
lint / lint (push) Successful in 2m37s
test / unit (push) Successful in 1m27s
test / integration (push) Successful in 35s
test / coverage (push) Successful in 1m36s
Update Quality Badges / update-badges (push) Successful in 1m29s
Adds globalize_slug(slug) to bottle_state alongside bottle_identity.
git_gate_provision now calls globalize_slug(slug) instead of inlining
socket.gethostname(), so the hostname-qualification logic has a single,
named home. Assumes slug is a mint_slug output.

Title format changes from bot-bottle:{host}:{slug}:{name}
to bot-bottle:{host}-{slug}:{name} to match the globalize_slug contract.
2026-07-16 23:07:12 -04:00
didericis-claude 59be808ab1 feat(git-gate): include hostname in deploy key title
Closes #388 (part 1 of 3). Deploy key titles now carry the machine
hostname so keys provisioned on different hosts don't collide with
each other on the forge when a prior bottle was never torn down.

Title format: bot-bottle:<hostname>:<slug>:<repo-name>
2026-07-16 23:07:12 -04:00
Quality Badge Bot eb63bd417d chore: update quality badges
- Coverage: 82%
- Core coverage: 95%

[skip ci]
2026-07-17 03:00:40 +00:00
didericis 943049733e test(supervise): stub the client in cmd_supervise crash-logging tests
test / unit (pull_request) Successful in 1m23s
test / integration (pull_request) Successful in 38s
test / coverage (pull_request) Successful in 1m42s
lint / lint (push) Successful in 2m15s
test / unit (push) Successful in 1m17s
test / integration (push) Successful in 31s
test / coverage (push) Successful in 1m31s
Update Quality Badges / update-badges (push) Successful in 1m35s
`cmd_supervise` establishes the orchestrator client up front (since the
HTTP-bridge move in 27fe03b), so these tests — which mock `curses.wrapper`
to exercise the KeyboardInterrupt / Die / crash-log paths — only reached
those paths when a live orchestrator happened to be reachable. On CI
(none reachable) the up-front connect errored and `cmd_supervise`
returned 1 before curses, failing all four. They passed locally only
because a dev orchestrator was up.

Stub `supervise_cli._client` in the class setUp so the tests isolate the
post-connect behavior they actually cover, independent of environment.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:44:47 -04:00
didericis c15eed4f2e fix(supervise): list-egress-routes returns the bottle's real routes
In consolidated mode the gateway's static route table is empty (routes
are resolved per request by source IP), but `list-egress-routes` was
reading that static table via the `_egress.local/allowlist` introspection
endpoint — so it always returned an empty allowlist. Agents, told to call
it before composing an egress proposal, then sent a routes.yaml with only
the newly-needed host; approving it replaced the whole policy and silently
dropped base routes like api.anthropic.com, breaking the bottle's egress.

Answer `list-egress-routes` from the calling bottle's resolved policy
(same (source_ip, identity-token) attribution the proposal path uses,
same JSON shape the single-tenant introspection endpoint returns).
Fail-closed to an empty list on an unreachable orchestrator; falls back
to the introspection endpoint in single-tenant mode (no resolver).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:18:14 -04:00
didericis b1df380ae1 feat(supervise): show the bottle's human slug, not its opaque id
Consolidated proposals are keyed by the orchestrator-assigned bottle_id,
so the supervise TUI was rendering a hex id (e.g. 3601cbe883c2786d) as
the bottle name — and the resume hint printed `./cli.py resume <id>`,
which resume can't take (it wants the human identity/slug).

`supervise_pending` now tags each dict with `bottle_label` — the human
slug resolved from registry metadata, falling back to the id when the
bottle is gone. `QueuedProposal` carries the label; every display site
(list rows, detail view, status lines, resume hint) shows it, while
approve/reject still key on `proposal.bottle_slug` (the id).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:05:21 -04:00
didericis 5f59df9e10 fix(supervise): resolve approval target by bottle_id, not human slug
In consolidated mode the supervise server attributes each proposal to
the orchestrator-assigned bottle_id and stores that as the proposal's
`bottle_slug` (supervise_server `_attributed_config`). But
`_record_for_slug` only scanned registry metadata for a matching human
slug, so approving an egress proposal 409'd with "bottle <id> is no
longer registered; cannot apply the route change" — the id never
matched a human slug.

Resolve the record by bottle_id first (the consolidated reality),
keeping the metadata-slug scan as a fallback for legacy single-tenant
proposals. The prior tests keyed proposals by the human slug matching
the metadata, exercising only the fallback path — added a test that
mirrors production (proposal keyed by bottle_id, distinct human slug in
metadata).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 22:01:14 -04:00
didericis 2641ab70fd feat(supervise): start orchestrator on demand via backend-agnostic bring-up
Add BottleBackend.ensure_orchestrator() -> str: the backend-agnostic
entry point that brings up the per-host orchestrator + shared gateway
(idempotent) and returns its host-reachable control-plane URL. Docker
starts the orchestrator + gateway containers (OrchestratorService);
firecracker boots the infra VM; macos-container dies with a pointer
(no orchestrator). Previously bring-up was reachable only through each
backend's consolidated_launch, with no shared handle.

Wire it into `bot-bottle supervise`: supervise is often the first thing
an operator runs, before any bottle has booted the control plane, so
`_resolve_orchestrator_url` now starts the selected backend's
orchestrator on demand when discovery finds nothing, instead of failing
with "launch a bottle first".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-16 21:52:11 -04:00
didericis 265119d601 fix(supervise): docker gateway shares the one host DB (Step 2d)
lint / lint (push) Successful in 2m24s
test / unit (pull_request) Failing after 1m27s
test / integration (pull_request) Successful in 36s
test / coverage (pull_request) Failing after 1m16s
The docker gateway container ran the supervise daemon but bind-mounted no
DB and set no SUPERVISE_DB_PATH — so the daemon wrote proposals to a
container-local, ephemeral SQLite file, disconnected from the host DB the
orchestrator (and now the operator, over HTTP) uses. Same split-DB bug
firecracker had; Dockerfile.gateway even documents the mount
(`/run/supervise/bot-bottle.db bind-mounted at run time`) that
ensure_running never provided.

Bind-mount the host DB dir into the gateway at /run/supervise and set
SUPERVISE_DB_PATH, so the daemon queues into the same file the
orchestrator container opens (its BOT_BOTTLE_ROOT bind-mount) and the
operator reaches over the control plane. One DB per host, shared by
bind-mounts — docker now on the exact same supervise path as firecracker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 19:01:45 -04:00
didericis 27fe03b612 feat(supervise): host TUI drives approvals over HTTP, not the DB (Step 2b/2c)
The `bot-bottle supervise` operator TUI read and wrote the queue DB
directly and tried a backend-specific live "apply" (which was unwired —
it raised). It now talks only to the orchestrator control plane:

- OrchestratorClient gains supervise_pending() + supervise_respond().
- discover_orchestrator_url() finds the one running per-host control
  plane by health-probing the backends' well-known :8099 addresses
  (docker publishes on loopback; the firecracker infra VM serves it on
  the orchestrator TAP) — no backend branching in the TUI.
- discover_pending/approve/reject call the client; the server does the
  apply + response + audit atomically. The dead direct-DB apply/audit
  helpers and the docker/macos applicator imports are gone.
- A missing control plane is now a clean one-line error up front, not a
  mid-curses crash.

CLI tests move to mocking the client (the DB-write behaviour they used to
assert is now server-side, covered by test_orchestrator_service). Docker's
orchestrator-container DB wiring lands next so its /supervise endpoints hit
the same shared DB.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:57:47 -04:00
didericis 9085d6f713 feat(supervise): orchestrator-side operator-approval API (Step 2a)
The orchestrator owns the single DB *and* the live policy, so operator
decisions belong there — applied server-side, reached over HTTP. Adds:

  GET  /supervise/proposals   -> pending proposals across bottles
  POST /supervise/respond     -> apply + record an operator decision

`supervise_respond` is one atomic server-side op on the one DB: approve/
modify on an egress tool rewrites the bottle's policy (so the gateway
serves the new routes on its next /resolve — the live "apply" that was a
documented TODO), then writes the queued Response (unblocking the agent's
MCP call) and an audit entry. reject records the response + audit only.
Fails closed (409) when the proposal is unknown or the bottle was torn
down before the operator acted (an egress apply would have no target).

This is the server half of unifying every backend onto one HTTP path for
supervise; the host TUI (direct-DB today) moves onto this client next.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:51:55 -04:00
didericis 38bc555dbf fix(supervise): single migrated DB for firecracker — orchestrator owns supervise tables
lint / lint (push) Successful in 2m24s
test / unit (pull_request) Successful in 1m21s
test / integration (pull_request) Successful in 29s
test / coverage (pull_request) Successful in 1m29s
The firecracker supervise MCP daemon 500'd (-32603) on egress-allow/block:
it ran with no BOT_BOTTLE_ROOT/SUPERVISE_DB_PATH, so it targeted a stray,
unmigrated SQLite file and `write_proposal` hit "no such table:
supervise_proposals". Meanwhile the in-VM control plane migrated only the
registry table (orchestrator_bottles) into its own DB, and the host
operator reads a third, disconnected DB — three files, none shared.

Consolidate to one DB per host, owned by the control plane on the
persisted registry volume (/var/lib/bot-bottle/db/bot-bottle.db):

- orchestrator startup now migrates the supervise queue + audit tables
  into the same file it migrates the registry into (StoreManager), so the
  control-plane DB carries every table.
- the in-VM supervise daemon is pointed at that same file via
  SUPERVISE_DB_PATH, so daemon and control plane share one queue.

`list-egress-routes` already worked (no DB); egress-allow now queues +
waits on the single persisted DB instead of erroring. Validated against a
live infra VM: migrate + write_proposal succeeds and the proposal is
queued. The host-operator HTTP bridge (so approvals complete from the
host, unifying docker onto the same path) is the follow-up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:24:45 -04:00
didericis a208bcde08 feat(firecracker): stream buildah build output live during agent-image build
lint / lint (push) Successful in 2m20s
test / unit (pull_request) Successful in 1m19s
test / integration (pull_request) Successful in 25s
test / coverage (pull_request) Successful in 1m19s
The agent-image build ran over ssh with output fully captured, shown
only as a 20-line tail on failure — so a successful (or in-progress)
first build was a long silent wait through the base pull + apt/npm
installs. Stream the `buildah build` step's stdout/stderr straight to
our own (like the docker backend's `docker build`) via a non-capturing
_ssh_streamed helper, so the operator sees `STEP i/n` progress live.
Failure now points at the streamed output above instead of a captured
tail. The smoke test and rootfs export stay captured (short / piped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:01:59 -04:00
didericis c0066d2cd2 fix(firecracker): quote guest argv tokens so codex's multi-word prompt survives ssh
The interactive agent command is sent to the guest by spreading the
remote argv as separate ssh arguments; ssh space-joins everything after
the host into one line that the guest login shell re-parses. That only
works while every token is a "simple word" — which held for claude
(`--append-system-prompt-file <path>`) but not for codex's
`read_prompt_file` mode, whose positional is a whole sentence:
"Read and follow the instructions in <path>.". The guest shell re-split
it on spaces, so codex received `Read` as the prompt and `and`, `follow`,
… as extra args — failing with `unrecognized subcommand 'and'` the moment
an interactive codex session attached.

Pre-quote each remote token with shlex.quote before ssh joins them (the
same ssh→guest-shell discipline infra_vm/cp_in already use). Simple words
are unchanged, so existing behaviour and the parity/structure tests are
untouched; an arg with spaces now survives as a single argument.

Regression test round-trips the joined remote command back through
shlex.split and asserts codex's prompt comes out as exactly one arg.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 18:01:59 -04:00
didericis 7118480d0a fix(codex): register supervise MCP via config.toml http_headers, not mcp add --header
lint / lint (push) Successful in 2m39s
test / unit (pull_request) Successful in 1m37s
test / integration (pull_request) Successful in 29s
test / coverage (pull_request) Successful in 1m44s
The Codex CLI has no `codex mcp add --header` flag (verified against
0.144.5 and the codex-rs `AddMcpStreamableHttpArgs` surface: only
`--url`, `--bearer-token-env-var`, `--oauth-*`, `--env`). The old call
therefore exited nonzero on every codex bottle; provisioning only
warned and continued, so supervise was silently unregistered — and
under mandatory (source_ip, token) attribution the suggested manual
recovery (`codex mcp add supervise --url ...`, no token) could not
restore access either.

Write the `[mcp_servers.supervise]` streamable-HTTP entry directly into
`~/.codex/config.toml` instead, delivering the identity token via the
Codex-supported `http_headers` key (the only way to attach a static
request header to an HTTP MCP server). Registration failure is now
FATAL when supervise is enabled, rather than a warning.

Test validates the generated entry against the real Codex config
surface: it must parse as TOML into a streamable-HTTP server carrying
the token as `http_headers["x-bot-bottle-identity"]`, and must use only
keys accepted by `RawMcpServerConfig` (config.toml is
`deny_unknown_fields`). Also covers custom `CODEX_HOME`, the no-token
case, and the now-fatal failure path.

Refs: PR #354 review (codex P1).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 17:42:36 -04:00
didericis d4b27ebf1f feat(gateway): mandatory identity-token attribution on every data plane (PR #354 review)
lint / lint (push) Successful in 2m12s
test / unit (pull_request) Successful in 1m12s
test / integration (pull_request) Successful in 27s
test / coverage (pull_request) Successful in 1m21s
Codex review: the /31 TAP doesn't make source IP unspoofable, and the
app-layer token was returned by launch but never delivered or enforced, so
a spoofed source could select a victim bottle's policy/tokens. Make the
token mandatory and deliver it on each attributed plane (anti-spoof landed
separately as the network boundary).

Enforcement (control plane):
- `Orchestrator.resolve` now requires a matching (source_ip, identity_token)
  pair (constant-time) — no source-IP-only fallback. `/resolve` fail-closes
  (403) on a missing/empty/mismatched token.

Delivery, per plane (the token is `token_urlsafe`, safe in a URL):
- egress: proxy credentials (`HTTPS_PROXY=http://bottle:<token>@gw`). The
  addon reads `Proxy-Authorization` — from the request (HTTP) or captured at
  the CONNECT for HTTPS tunnels (keyed by client conn, cleared on disconnect)
  — validates, and strips it (+ the legacy header) before upstream.
- git-http: a URL-scoped `http.<gate>/.extraHeader: x-bot-bottle-identity`
  in the agent's git config (only over the http transport).
- supervise: `mcp add --header x-bot-bottle-identity: <token>` (claude +
  codex); the server reads the header and passes it to resolve.

Wiring: thread `ctx.identity_token` onto the firecracker + docker plans and
into the agent env/config at launch.

Verified on a KVM host: egress with the correct proxy-cred token returns
200 (HTTP and HTTPS/CONNECT), and no-token / wrong-token return 403; a real
`cli.py start --backend=firecracker` launch provisions git config + the
supervise MCP header and reaches the agent session, all under mandatory
enforcement. Fixed a `claude mcp add` arg-order bug (--header must follow
the positional name/url) found by that launch.

Transparent proxy for tools that ignore proxy env is deferred to a
follow-up (see thread); anti-spoof + host firewall remain the fail-closed
boundary.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 17:02:41 -04:00
didericis 914f01fa8f fix(firecracker): serialize infra-VM create + agent builds (PR #354 review)
lint / lint (push) Successful in 2m8s
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 27s
test / coverage (pull_request) Successful in 1m22s
Codex flagged that parallel `start`s race on shared state: two cold
launches could both stop/build/boot the singleton on the same rootfs/PID,
concurrent builds share the infra VM's buildah store, the cleanup did
`buildah rm -a` (nuking a peer build's container), and the rootfs cache
was populated non-atomically.

- `infra_vm.ensure_running`: a host flock (`singleton.lock`) around the
  cold stop/build/boot path, with a double-checked health re-test under
  the lock so a second launcher adopts rather than re-boots. The healthy
  fast-path stays lock-free.
- `image_builder.build_agent_rootfs_dir`: a host flock (`.build.lock`)
  around cache-lookup + build + publish; build into `.building-<digest>`
  and publish by atomic `os.rename`, so a partial build never appears as
  `agent-<digest>`.
- scoped cleanup: per-build named working containers
  (`<tag>-smoke`/`-export`) removed by name instead of `buildah rm -a`;
  also cleared before the build to recover from a crashed prior run.

Verified: agent image builds via the locked/atomic path, infra VM stays
healthy, agent boots, `claude --version` = 2.1.172. Full unit suite +
pyright green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 16:15:58 -04:00
didericis 43c3d4408e fix(firecracker-netpool): per-TAP anti-spoof so source-IP attribution is sound (PR #354 review)
The /31 point-to-point TAP does NOT make a guest source address
unspoofable: root in an agent VM can source another bottle's guest IP on
its own bbfc TAP. The isolation table only matched iifname class + port
(DNAT) and never bound iifname to its assigned ip saddr — so a spoofed
source was DNAT'd to the gateway and attributed to the *victim* bottle,
getting the victim's policy/tokens. Source-IP attribution was therefore
not actually sound.

Add one anti-spoof rule per slot in the isolation forward chain, before
the established/DNAT accepts: `iifname bbfcN ip saddr != <guestN> drop`.
Generated in the existing setup loop — no new dependency, ~pool_size
lines. Legit traffic (correct saddr) is unchanged; a spoofed saddr on any
bbfc TAP is dropped before it can be attributed.

Apply with a nixos-rebuild (the systemd unit re-runs this script). Codex
review blocker; the app-layer identity token (defense-in-depth) is wired
separately.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 16:09:45 -04:00
didericis 0a26b8795a refactor(gateway): install gitleaks by pinned+verified download, drop third-party base image (PR #354 review)
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 24s
test / coverage (pull_request) Successful in 1m27s
The gateway used `FROM zricethezav/gitleaks AS gitleaks-src` purely to
COPY the binary out — a supply-chain surface (a whole third-party image as
a build input, tying us to its cadence). Install gitleaks from its official
release instead, pinned by version + SHA256 and verified. python (already
in the image) does the download, so no curl/wget is added. trixie apt also
ships gitleaks but an older 8.16; the pinned download keeps the verified
8.30.1 (byte-identical to what the image provided).

Verified: gateway + infra images build, gitleaks 8.30.1 runs in both.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 15:53:40 -04:00
didericis c60e6b7e9f refactor(firecracker): single infra VM builds too — buildah in one image (PR #354 review)
lint / lint (push) Successful in 2m18s
test / unit (pull_request) Successful in 1m13s
test / integration (pull_request) Successful in 26s
test / coverage (pull_request) Successful in 1m22s
Addresses the review finding that buildah lived only in the orchestrator
image, so the persistent infra VM wasn't the builder — a separate throwaway
builder VM contended with it for the orchestrator TAP. Consolidate:

- **Rebase the gateway (and thus infra) on `python:3.12-slim` = Debian
  trixie**, pip-installing mitmproxy instead of `FROM mitmproxy/mitmproxy`
  (Debian bookworm). trixie ships buildah 1.39, which can build agent
  Dockerfiles that use heredocs; bookworm's 1.28 can't (`Unknown
  instruction: "{"`). CA path is unchanged (set via `--set confdir=`).
- **buildah lives only in `Dockerfile.infra`** now (removed from the
  orchestrator image, which is lean/stdlib-only again).
- **Shared orchestrator content**: `Dockerfile.orchestrator` is the single
  definition of the control-plane payload; the infra image `COPY --from`s
  it (same trixie base → clean copy, and future deps like iroh are added
  once). The docker backend runs the orchestrator image directly.
- **`image_builder` builds inside the infra VM** (which now has buildah)
  over SSH — no throwaway builder VM, so the `bborch0` contention is gone.
  `ensure_built` builds orchestrator + gateway before infra (FROM gateway,
  COPY --from orchestrator).

Verified on a KVM host: images build (buildah 1.39 in infra), the agent
image builds *inside* the infra VM (heredoc Dockerfile and all), the infra
VM stays healthy, the agent boots and `claude --version` = 2.1.172. The
rebased gateway still starts as a docker container and generates its CA
(docker backend unaffected).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 15:23:36 -04:00
didericis 2d37965249 fix(firecracker-netpool): make nft table installs delete-first idempotent
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 28s
test / coverage (pull_request) Successful in 1m22s
Each nft table (isolation, orchestrator-egress nat, agent->gateway route)
was re-applied as a plain `table {...}` block, which on a bare `up`
re-apply (not the systemd down->up path) would append duplicate rules or
error on the existing base chains. Use the standard delete-first pattern
(create empty, delete, recreate) so `up` lands identical state regardless
of history — the setup reproduces cleanly on a fresh install and on
re-apply, not just via a full down->up cycle.

No functional change to the resulting ruleset; only its idempotency.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 4873030550 feat(firecracker): git-gate over SSH into the gateway VM (Stage B, 7/n)
git-gate now works end-to-end for git-upstream bottles on the infra VM.
Three fixes surfaced by driving a real clone through git-http:

- `SshGatewayTransport.cp_into` preserves the source file mode (docker cp
  does). The access-hook is staged 0700 and git-http execs it directly; a
  plain `cat >` landed it 0644 -> EACCES. Keys stay 0600.
- the infra init runs `BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise`:
  the VM backend reaches git over git-http (9420), so the git:// daemon
  (git-gate, whose /git-gate-entrypoint.sh the consolidated model doesn't
  stage) is left out instead of crash-looping.
- `build_infra_rootfs_dir` folds the init's content-hash into the rootfs
  cache key, so an init change actually rebuilds the rootfs (the base image
  digest alone wouldn't catch it).

Verified on a KVM host: launch_consolidated provisions a git-upstream
bottle's repo + creds into the gateway VM over SSH; an agent VM clones via
git-http (source-IP attributed) and the access-hook resolves the
provisioned key + known_hosts and attempts the upstream SSH fetch —
failing closed only because the test used a dummy key + fake upstream
("refusing to serve stale data"). With real creds the fetch serves.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis b93b14f5c2 feat(firecracker): persistent registry volume for the infra VM (Stage B, 6/n)
The infra VM's rootfs is ephemeral (rebuilt each boot), so the bottle
registry DB needs durable storage across restarts. Give the infra VM a
firecracker analogue of a docker volume: a host-side ext4 file attached as
a second virtio-block device (guest /dev/vdb), mounted at the control
plane's DB dir (/var/lib/bot-bottle, where host_db_path lives at
db/bot-bottle.db).

- firecracker_vm.boot/_config take an optional `data_drive` (a non-root,
  RW second drive).
- infra_vm creates the volume on first use (`mke2fs` an empty ext4 at
  <fc-cache>/infra/registry.ext4) and mounts /dev/vdb in the PID-1 init
  before the control plane starts. It's a plain ext4 file, so
  `sudo mount -o loop <path>` (VM stopped) inspects bot-bottle.db directly.

Verified on a KVM host: register a bottle, restart the infra VM (fresh
rootfs, same volume) — /dev/vdb re-mounts and the registry `db/` dir + a
marker file survive the restart.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 957eb19368 feat(firecracker): launch through the infra VM, not Docker (Stage B, 5/n)
consolidated_launch now drives the persistent infra VM instead of the
`_FirecrackerOrchestratorService` Docker containers — the point Docker
leaves the Firecracker launch path.

- launch_consolidated: `infra_vm.ensure_running()` (singleton) for the
  control plane + gateway; register the bottle over HTTP at the infra VM's
  guest IP; provision git-gate via `SshGatewayTransport` (over SSH into the
  gateway VM); fetch the gateway CA via `InfraVm.gateway_ca_pem()`.
- teardown_consolidated deregisters + deprovisions but does NOT stop the
  infra VM (persistent per-host singleton shared by every bottle).
- new `infra_vm.SshGatewayTransport` + `gateway_transport()` (built from the
  stable key + orchestrator link IP, so teardown needs no live handle).
- drop the DockerGateway/OrchestratorService machinery from the firecracker
  consolidated path (still used by the docker backend).

launch.py already calls launch_consolidated, so the whole firecracker
launch path now uses the VM. pyright + unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 5dfb9b0d75 refactor(gateway): parameterize git-gate provisioning transport (Stage B prep)
git-gate provisioning into the running gateway was hard-wired to docker
exec/cp. Extract a backend-neutral `GatewayTransport` (exec + cp_into) so
the same provisioning logic serves both the docker gateway container and
the firecracker gateway VM (over SSH, added with the launch swap).

- `provision_git_gate` / `deprovision_git_gate` now take a transport
  instead of a gateway name; `DockerGatewayTransport` wraps the existing
  docker exec/cp behavior. deprovision is best-effort (catches the
  transport error) — matching the prior idempotent teardown.
- both consolidated_launch callers pass `DockerGatewayTransport(name)` —
  no behavior change; the firecracker swap flips only its own to SSH.

Pure refactor: docker path unchanged, gateway_provision tests updated to
construct the transport, all green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis bb434b14d7 feat(firecracker): persistent infra-VM singleton lifecycle (Stage B, 4/n)
The infra VM must outlive the short-lived `start` launcher and be reused
across launches. Add an idempotent singleton:

- `ensure_running()` adopts the infra VM when its control plane is already
  healthy (a prior launcher booted it), else clears any stale VM and boots
  a fresh one. Returns a handle usable for CA fetch / git-gate provisioning
  whether we booted it or adopted it.
- boot is `detached` (firecracker in its own session via start_new_session)
  so it survives the launcher exiting; its PID is recorded so a later
  process can `stop()` it. `_kill_pidfile` SIGTERM/SIGKILLs but only if the
  PID is still a firecracker process (guards a recycled PID).
- a STABLE SSH key (generated once under the infra cache dir, re-injected
  each boot via the cmdline) so any launcher can SSH in to fetch the CA /
  provision, not just the one that booted the VM. `InfraVm.vm` is None in
  the adopted case; teardown then goes through the PID file.

Verified on a KVM host: first ensure_running boots; a second adopts it
(same PID, no reboot) and can still reach /health and fetch the gateway CA
over SSH; stop() tears it down (control plane then unreachable).

Next: git-gate provisioning into the VM over SSH (today docker exec/cp),
then swap consolidated_launch.py onto the infra VM.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis d79d5b295a feat(firecracker): route agent VMs to the gateway VM (Stage B, 3/n)
Agent VMs must reach the shared gateway that now runs in the infra VM
(egress:9099 / supervise:9100 / git-http:9420 at the orchestrator link's
guest IP). Add a PREROUTING DNAT: agents keep addressing their own
host-side TAP IP on the gateway ports, and the rule redirects that to the
infra VM. The isolation table's existing `ct status dnat accept` forward
rule lets the DNAT'd traffic through; every other agent egress stays
dropped, so a bottle still reaches only the gateway and nothing else.

Source IP is deliberately NOT masqueraded: the gateway attributes each
request to the originating bottle by its guest IP, which the /31 TAP + the
bot_bottle_fc nft table make unspoofable. Keeping the agent addressed at
its own host TAP IP means no per-bottle config change vs the docker-DNAT
path it replaces.

- scripts/firecracker-netpool.sh: `_install_gateway_route` adds
  `table ip <table>_gw` (prerouting dstnat -> orch_guest on the gateway
  ports); wired into up/down/status. The nix module needs no change — it
  runs this script, and the ports are baked in.

Verified on a KVM host: an agent VM's `curl -x http://<its-host-tap>:9099`
reaches mitmproxy in the infra VM and gets a 403 (correct policy denial
for an unregistered bottle) — i.e. the route lands end-to-end. Persist
with a nixos-rebuild; the imperative rule holds until then.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis e1610121c0 feat(firecracker): run the gateway data plane in the infra VM too (Stage B, 2/n)
The single infra VM now runs BOTH the orchestrator control plane and the
gateway data plane (egress / supervise / git-http), multi-tenant against
the local control plane — the single-VM shape from the Stage B design.

- Dockerfile.infra: the firecracker infra image = the gateway image +
  the baked control-plane source (FROM bot-bottle-gateway, COPY
  bot_bottle). Reuses the gateway payload rather than copying mitmproxy/
  gitleaks into a third image; the docker backend keeps its two separate
  images.
- infra_vm: build from source (gateway then infra image), and the PID-1
  init now also launches `gateway_init` with
  BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:8099. Adds `gateway_ca_pem`
  (fetch the mitmproxy CA over SSH) and the agent-facing port constants.
- init exports PATH — a bare-init shell resolves its own execs via a
  built-in default path, but that isn't in the environment, so
  gateway_init's `python3 ...` daemons would otherwise fail to spawn.

Verified on a KVM host: infra VM boots, control plane /health -> 200, and
egress:9099 / supervise:9100 / git-http:9420 all listen and are reachable
from the host over the TAP link; the gateway CA is retrievable. (git-gate
stays down until a bottle provisions its per-bottle entrypoint/creds, same
as a fresh docker gateway — non-fatal, the supervisor keeps the rest up.)

Next: agent->gateway VM-to-VM routing so bbfc* VMs reach these ports at
the infra VM and nowhere else.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 1614172423 feat(firecracker): run the orchestrator control plane as a VM (Stage B, 1/n)
First slice of Stage B: the orchestrator control plane runs as a
persistent Firecracker infra VM instead of a Docker container. The host
CLI reaches it over HTTP at the orchestrator link's guest IP; agent VMs
will reach its gateway ports (added next) over VM-to-VM routing.

- Dockerfile.orchestrator bakes the stdlib-only control-plane source
  (COPY bot_bottle) so the image is self-contained and runs from a built
  image with no runtime bind-mount — a guest VM can't bind-mount host
  source. (Build-from-source stays the default; a pull-from-registry mode
  lands later. The docker backend's dev bind-mount still overlays this.)
- util.build_base_rootfs_dir / inject_guest_boot take a `variant` +
  `init_script`, so the same orchestrator image is prepared two ways
  without a cache collision: the builder VM keeps the SSH-only agent init;
  the infra VM gets a control-plane PID-1 init.
- new firecracker/infra_vm.py: boot the infra VM on the orchestrator link,
  run `python -m bot_bottle.orchestrator` as PID 1, and poll /health.

Verified on a KVM host: infra VM boots, control plane answers
`GET /health -> 200 {"status":"ok"}` from the host over the TAP link.
Next: fold gateway_init (egress/git-gate/supervise) into the same VM,
then agent->gateway routing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 4edd7803e8 feat(firecracker): build agent images in a builder VM, not host docker (Stage 3)
Replace the host `docker build` + `docker export` behind the Firecracker
agent rootfs with an in-VM buildah build. `image_builder.build_agent_rootfs_dir`
boots a throwaway builder VM (the orchestrator image, which carries
buildah) on the NAT'd orchestrator link, sends the Dockerfile over SSH,
`buildah build`s it, smoke-tests the result with `buildah run` (the
image's own PATH, so it catches an npm silent-failure stub), and streams
the rootfs tar back into the content-addressed cache dir — the same base
dir `util.build_rootfs_ext4` already turns into a bootable ext4 with
`mke2fs -d`. No host Docker daemon, no root-equivalent `docker` group;
an untrusted Dockerfile runs in a confined microVM, not on the host.

- new firecracker/image_builder.py (boot → build → smoke → stream).
- launch.py: `_build_agent_image` → `_build_agent_base`, returning the
  base dir from the builder VM. The committed-snapshot (freeze/migrate)
  path still exports via host docker until it too is ported.
- util: `_inject_guest_boot` → public `inject_guest_boot` (shared with
  the builder). unit tests for the cache decision + smoke-test paths.

Verified on a KVM host end-to-end: builds the real claude Dockerfile
(node:22-slim + npm claude-code) in-VM in ~60s, the produced agent VM
boots and `claude --version` returns 2.1.172; the content cache skips
the rebuild on a repeat launch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 81a2f15046 feat(firecracker): NAT'd egress link for the orchestrator/builder VM
The orchestrator/gateway VM is trusted infra, not an isolated agent: it
builds agent images in-VM (buildah must FROM-pull + apt/npm) and, in the
Stage B cutover, forwards agent egress upstream. Give it a dedicated TAP
(`bborch0`) on a /31 at the top of the IP_BASE /16 (clear of the bbfc*
agent pool at the bottom), NAT'd out the host uplink — while agent VMs
keep their fail-closed, gateway-only isolation table.

- netpool.defaults.env / netpool.py: new BOT_BOTTLE_FC_ORCH_IFACE +
  `orch_slot()` (index -1 sentinel; host x.y.255.0 / guest x.y.255.1).
- scripts/firecracker-netpool.sh: create + address the orchestrator TAP;
  `bot_bottle_fc_nat` table masquerades its /31 out the uplink and
  accepts its forward path. Because bootstrap still runs Docker (whose
  FORWARD policy is DROP), a best-effort, guarded, idempotent DOCKER-USER
  ACCEPT is added too (skipped once Docker is gone). down/status updated.
- nix/firecracker-netpool.nix: mirror the option, pass it via the unit
  Environment= (the store-copied script can't read the defaults file),
  and add iptables to the unit path for the DOCKER-USER step.

Agent isolation is unchanged: the new rules only ever accept/masquerade
the orchestrator link and never drop, so they can't weaken the bbfc*
drops. Applied by re-running `sudo ./scripts/firecracker-netpool.sh up`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis 75b122398d feat(orchestrator): add buildah for in-VM agent-image builds (PRD 0069 Stage 3)
The Firecracker backend's remaining host-Docker dependency is building
users' agent Dockerfiles. Move that build *into the orchestrator VM*:
give the orchestrator image buildah (rootless, daemonless) so it builds
agent images itself, and the host needs no Docker daemon and no
root-equivalent `docker` group. An untrusted Dockerfile then builds
inside the confined orchestrator VM rather than on the host — strictly
more isolated than host `docker build`.

- buildah + crun (OCI runtime) + netavark/aardvark-dns (network backend
  for FROM pulls and RUN egress), installed explicitly since
  --no-install-recommends strips buildah's helper deps.
- vfs storage + chroot isolation (STORAGE_DRIVER/BUILDAH_ISOLATION) so
  buildah needs neither fuse-overlayfs / an overlay kernel module nor
  configured subuid maps — it works unconditionally as root in a minimal
  microVM rootfs. (Slower than overlay; a build-cache pass is deferred.)

Bootstrap: the orchestrator rootfs is still produced from this image via
host docker export; a later step pulls a pre-built image instead.

Verified: the orchestrator image boots as a Firecracker VM and buildah
runs inside it as real VM root (reaches the base-image pull; build-time
egress wiring is the next step).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-16 14:36:54 -04:00
didericis-claude 2e738c3338 fix(docker): use run_docker in docker_exec, docker_cp, verify_agent_image
The rebase onto lazy-backend-imports converts existing helpers (image_exists,
container_exists, etc.) to run_docker; the three new functions added in this
branch still called subprocess.run directly. Switch them over for consistency.
2026-07-16 14:36:54 -04:00
didericis-claude 9369fb7de5 feat(firecracker): implement consolidated orchestrator launch (PRD 0070)
Replace the per-bottle Docker sidecar bundle with the shared per-host
orchestrator + gateway, mirroring what the Docker backend already has.

- Add `bot_bottle/backend/firecracker/consolidated_launch.py`:
  `_FirecrackerOrchestratorService` (subclasses `OrchestratorService`,
  overrides `_gateway()` to return a `DockerGateway` with host port
  bindings so Firecracker VMs can reach it via their TAP link);
  `launch_consolidated()` registers the bottle by guest IP (attribution
  key), provisions git-gate into the shared gateway, and returns the
  shared CA + orchestrator URL for teardown; `teardown_consolidated()`
  deregisters and cleans up.

- Rewrite `bot_bottle/backend/firecracker/launch.py`: removes the
  per-bottle sidecar bundle (`_start_sidecar_bundle`, `_stage_git_gate`,
  etc.) and `_mint_certs`; wires `launch_consolidated()` instead. The VM
  still sends to `host_tap_ip:PORT` — Docker's PREROUTING DNAT + the nft
  `ct status dnat accept` rule in the forward chain route the traffic to
  the shared gateway container.

- Extend `DockerGateway` with `host_port_bindings` so the Firecracker
  gateway publishes its ports on the host (`0.0.0.0:PORT`).

- Parameterise `OrchestratorService` with `orchestrator_name` /
  `orchestrator_label` so Docker and Firecracker orchestrators can
  coexist on the same host (`bot-bottle-orchestrator` vs
  `bot-bottle-fc-orchestrator`).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-16 14:36:54 -04:00
didericis 9afdeff619 refactor(firecracker): use docker_mod instead of hand-rolled docker helpers
firecracker/launch.py reimplemented docker build/image-exists/rm/exec/cp
as private functions instead of the shared docker_mod used by the
docker and macos-container backends. Switching to docker_mod dedupes
the logic and gets --no-cache support for free (docker_mod.build_image
already reads BOT_BOTTLE_NO_CACHE); docker_mod gains docker_exec/
docker_cp general-purpose helpers to cover what the private versions did.
2026-07-16 14:36:54 -04:00
didericis e45df03bd9 fix: smoke-test agent images after build, add start --no-cache
npm treats optionalDependencies failures as non-fatal, so a transient
network blip fetching claude-code's platform-native binary during
`npm install -g` left a stub CLI in an image that still "built"
successfully — then got baked into the Docker/Container layer cache
until forced to rebuild. Post-build smoke test (provider-declared
argv, run in a throwaway container of the freshly built image) fails
the launch loudly instead of shipping a broken image; --no-cache
gives an escape hatch to force a from-scratch rebuild.

Closes #353.
2026-07-16 14:36:54 -04:00
Quality Badge Bot ec55dfde0c chore: update quality badges
- Coverage: 84%
- Core coverage: 95%

[skip ci]
2026-07-14 21:26:52 +00:00
didericis 56d879f0b3 fix(tests): pyright strict — export _plan fixture, drop unused imports
test / unit (pull_request) Successful in 1m10s
test / integration (pull_request) Successful in 25s
test / coverage (pull_request) Successful in 1m20s
lint / lint (push) Successful in 2m16s
test / unit (push) Successful in 1m12s
test / integration (push) Successful in 26s
test / coverage (push) Successful in 1m19s
Update Quality Badges / update-badges (push) Successful in 1m20s
- `tests/unit/_docker_bottle_plan.py`: add `__all__ = ["_plan"]` so the
  underscore-prefixed fixture isn't flagged reportUnusedFunction.
- `tests/unit/test_egress_apply.py`: drop `SimpleNamespace` / `patch`
  imports left unused after the reload test became fail-closed.

`pyright .` → 0 errors.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 17:12:12 -04:00
didericis 09393b354b refactor(de-sidecar): purge the "sidecar" name from live code, tests, and current docs
lint / lint (push) Failing after 2m3s
test / unit (pull_request) Successful in 1m11s
test / integration (pull_request) Successful in 26s
test / coverage (pull_request) Successful in 1m22s
Completes the de-sidecar cleanup: no live code, test, current doc,
script, or nix file mentions or is named "sidecar" any more. Only the
dated PRD/research docs keep the term as historical record (agreed on
the #385 thread).

- Rename `sidecar_init.py`→`gateway_init.py` was done earlier; this pass
  sweeps the remaining descriptive uses: the egress / git-gate / supervise
  components are the gateway's *daemons*, the shared container is the
  *gateway*, the old per-bottle container was the *companion container*.
- Rename `tests/integration/test_sidecar_bundle_image.py`→`test_gateway_image.py`
  and its class; update `docs/ci.md` + `tests/README.md` for the renamed/
  removed integration tests.
- `SIDECAR_PORTS` shell var in `scripts/firecracker-netpool.sh`→`GATEWAY_PORTS`.

Full unit suite green (bar the pre-existing `/bin/sleep`-missing env
errors in test_gateway_init); docker integration — gateway singleton,
broker, real two-bottle multitenant isolation, and the gateway-image
build — all pass.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 17:07:56 -04:00
didericis 77948ef56c refactor(de-sidecar): remove the per-bottle companion-container architecture
The per-agent companion container (the egress/git-gate/supervise data
plane run once per bottle) is the pre-consolidation architecture. Remove
it and disable the backends that still depend on it, per the #385 thread.

- Delete `backend/docker/sidecar_bundle.py`; docker's live path uses the
  consolidated shared gateway, not a per-bottle bundle.
- Disable the firecracker and macos-container backends: their `launch()`
  fails closed (they launched a per-bottle companion; firecracker's
  consolidated relaunch is #354, macos follows). Their `enumerate` return
  empty and `cleanup` drop the companion-container discovery (firecracker
  keeps VMM/run-dir cleanup).
- Fail-close both backends' `egress_apply` reload (it signalled the
  per-bottle container); consolidated egress policy resolves per-request
  against the orchestrator, so gateway-side apply is a follow-up.
- Rename `egress_sidecar_env_entries` → `egress_gateway_env_entries`,
  `SIDECAR_PORTS` → `GATEWAY_PORTS`.
- Move the shared DockerBottlePlan fixture to `tests/unit/_docker_bottle_plan.py`;
  delete tests for the removed launch paths; update cleanup/egress-apply tests.

Docker consolidated launch verified end-to-end (multitenant isolation
integration test passes). macos/firecracker are intentionally disabled.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 17:02:08 -04:00
didericis c10d1cb6e0 refactor(de-sidecar): rename sidecar_init→gateway_init, drop docker's dead per-bottle compose renderer
Part of the de-sidecar cleanup (#385 discussion): the per-bottle companion
container is the old architecture.

- Rename `bot_bottle/sidecar_init.py` → `gateway_init.py` (it's the gateway
  image's PID-1 supervisor); env var `BOT_BOTTLE_SIDECAR_DAEMONS` →
  `BOT_BOTTLE_GATEWAY_DAEMONS`; log prefix `sidecar-init:` → `gateway-init:`.
  Update Dockerfile.gateway COPY/ENTRYPOINT and the test.
- Remove the dead per-bottle compose renderer from `backend/docker/compose.py`
  (`bottle_plan_to_compose`, `_sidecar_bundle_service`, `_agent_service`, and
  the network/bind/proxy helpers). Docker's live path uses
  `consolidated_agent_compose`; only the compose *lifecycle* helpers
  (up/down/ls/write) remain. Trim `test_compose.py` to the surviving helpers.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 16:46:46 -04:00
didericis 96b84eb84d refactor(orchestrator): split conflated sidecar image into orchestrator + gateway images
lint / lint (push) Successful in 2m19s
test / unit (pull_request) Successful in 1m14s
test / integration (pull_request) Successful in 32s
test / coverage (pull_request) Successful in 1m28s
One image — `bot-bottle-sidecars:latest`, built from `Dockerfile.sidecars` —
served two unrelated roles: the egress/git-gate/supervise *data plane* and
the orchestrator *control plane* (which ran the same image with the entrypoint
overridden to `python3 -m bot_bottle.orchestrator`). The control plane is
stdlib-only, so it needed none of the mitmproxy/git/gitleaks payload it was
riding on — while being the most secret-dense process on the host (PRD 0070's
"secret concentration").

Split into two purpose-built images:

- `Dockerfile.gateway` -> `bot-bottle-gateway:latest` — the data plane
  (renamed from Dockerfile.sidecars; identical contents).
- `Dockerfile.orchestrator` -> `bot-bottle-orchestrator:latest` — a lean
  `python:3.12-slim` runtime; the bind-mounted `bot_bottle` package supplies
  the code (so the #381 source-hash recreate semantics are unchanged).

`OrchestratorService` now takes distinct `image` (control plane, default
`ORCHESTRATOR_IMAGE`) and `gateway_image` (data plane, default `GATEWAY_IMAGE`)
instead of feeding one `self.image` to both, and builds the lean image
(build-if-missing) before starting the container. The per-bottle bundle
constants in `backend/docker/sidecar_bundle.py` now alias the gateway
constants so a bundle and the shared gateway can never drift onto different
images. The `bot-bottle-sidecars` *image* name and `Dockerfile.sidecars` are
gone; the per-bottle *container* name prefix (`bot-bottle-sidecars-<slug>`) is
intentionally left for a separate change.

Verified end-to-end: both images build; the lean image runs the control plane;
`ensure_running` brings up the orchestrator on `bot-bottle-orchestrator:latest`
and the gateway on `bot-bottle-gateway:latest` (distinct images) and reports
healthy.

Closes #384.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 16:24:16 -04:00
didericis 910267b8a8 fix(orchestrator): only recreate the orchestrator on a real source change
test / unit (pull_request) Successful in 1m7s
test / integration (pull_request) Successful in 20s
test / coverage (pull_request) Successful in 1m19s
lint / lint (push) Successful in 2m7s
test / unit (push) Successful in 1m11s
test / integration (push) Successful in 28s
test / coverage (push) Successful in 1m18s
Update Quality Badges / update-badges (push) Successful in 1m9s
ensure_running() unconditionally recreated the bot-bottle-orchestrator
container on every bottle launch, added so bind-mounted code changes take
effect (the process never reloads). But the orchestrator's per-bottle
egress auth tokens live only in its process memory (Orchestrator._tokens),
by design never persisted to disk. So starting a second bottle recreated
the container and silently dropped the FIRST bottle's already-injected
token -- its next authed egress call 403s with "env var EGRESS_TOKEN_0 is
unset", and it needs a full restart to recover.

Now the orchestrator container is labeled with a content hash of its
bind-mounted bot_bottle source, mirroring the gateway's existing
image-staleness check. ensure_running only recreates on a hash mismatch
(a real code change), so a bottle launch that isn't accompanied by a code
change leaves a healthy orchestrator -- and every other active bottle's
in-memory tokens -- alone.

Fixes #381.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-14 04:07:25 -04:00
Quality Badge Bot 03eacd9f57 chore: update quality badges
- Coverage: 83%
- Core coverage: 95%

[skip ci]
2026-07-14 07:42:30 +00:00
didericis c839cee319 test(orchestrator): skip multitenant isolation under act_runner
test / unit (pull_request) Successful in 1m3s
test / integration (pull_request) Successful in 20s
test / coverage (pull_request) Successful in 1m13s
lint / lint (push) Successful in 2m17s
test / unit (push) Successful in 1m5s
test / integration (push) Successful in 31s
test / coverage (push) Successful in 1m22s
Update Quality Badges / update-badges (push) Successful in 1m15s
The test brings up the orchestrator container, which bind-mounts the repo
path into a container on the host daemon. Under the Gitea act_runner the job
runs in a container sharing the host docker socket, so that path
(/workspace/...) doesn't exist on the host daemon and the mount fails
('mkdir /workspace: read-only file system'). Same host-bind-mount constraint
that already skips test_sandbox_escape and the other bottle-bringup tests, so
gate it the same way (GITEA_ACTIONS). Runs for real on a normal docker host.

This unblocks the integration + coverage CI jobs, which failed only because
this test errored (coverage.sh runs under set -e; no percentage gate tripped).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 03:15:26 -04:00
didericis ea75fab3b4 fix(orchestrator): supersede a prior active bottle at the same source IP
test / unit (pull_request) Successful in 1m4s
test / integration (pull_request) Failing after 30s
test / coverage (pull_request) Failing after 1m17s
register() used INSERT OR REPLACE keyed on bottle_id, so re-registering an
existing source_ip with a new bottle_id left TWO active rows for that IP.
by_source_ip fail-closes on ambiguity (>1 active -> None), so the reused IP
was then bricked to a 403 on egress resolve — even for hosts its own policy
allowed. Happy path (fresh IPs, teardown on stop) never hit it, but IP reuse,
crash recovery, or a persisted registry DB across an orchestrator restart
(now that ensure_running always recreates the container) all could.

register() now deletes any other active row at the same source_ip before
insert; the fail-closed ambiguity guard stays as defense in depth. Adds a
unit test for the supersede, and a docker integration test
(test_multitenant_isolation) that drives two bottles through one shared
gateway and asserts each gets only its own injected token and its own
allowlist — the core PRD 0070 multi-tenancy invariant, previously only
verified by hand.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 13a8bdd7ca fix(orchestrator): recreate the orchestrator container on ensure_running
The orchestrator runs the repo's code bind-mounted, but the Python process
loads it at startup and never reloads — and ensure_running reused a
healthy-but-stale container. So after a code change (e.g. the token fix), the
running orchestrator kept executing OLD control-plane code that dropped the
new /bottles 'tokens' field, and egress auth injection stayed broken no matter
how many times the gateway image was rebuilt.

ensure_running now always recreates the orchestrator container (cheap; the
registry DB persists and the current launch re-registers its in-memory
state). Combined with the gateway's image-staleness recreate, a fresh 'start'
now runs current code end to end.

Validated live: register an authed route + in-memory token -> a client at the
bottle IP curling through the gateway proxy receives the injected
'Authorization: Bearer <token>' header.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 3318bdce28 fix(orchestrator): recreate the gateway when its image is stale
Container-level counterpart to the ensure_built cache-aware fix. ensure_built
now rebuilds the gateway image on a source change, but ensure_running still
reused an already-running container built from the OLD image — so a rebuild
(even BOT_BOTTLE_NO_CACHE) never took effect on a warm gateway, and it kept
running the pre-fix flat daemons (this is why token injection stayed broken
after the rebuild). ensure_running now compares the running container's image
to the current image and recreates on mismatch. Validated on docker: a stale
gateway is detected and recreated with the current image + new egress addon.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis b2f61053ad fix(egress+orchestrator): inject per-bottle auth tokens in the shared gateway
The cut-over dropped the per-bottle token flow, so an authed egress route on
the shared gateway failed with 'env var EGRESS_TOKEN_0 is unset' — the gateway
reads the token from its env, but a shared gateway has no per-bottle env.

Now the bottle's egress auth tokens travel to the gateway over /resolve and
the addon injects from them, mirroring what the per-bottle sidecar's env did:
- launch resolves the token values from the host env and hands them to the
  orchestrator, which holds them IN MEMORY (keyed by bottle_id, never written
  to the registry DB) and serves them on /resolve;
- PolicyResolver.resolve_policy_and_bottle_id + resolve_client_context now
  return the token map alongside policy + bottle_id (one round-trip);
- the egress addon overlays the process env with the bottle's tokens per
  request and uses that env for auth injection AND DLP — the agent never sees
  the credential.

Secrets stay off disk (validated: /resolve returns the token, the registry DB
does not contain it). SecretProvider (#355) is the future hardening.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 485d101430 fix(orchestrator): poll for the gateway CA (fresh-start readiness race)
On a fresh gateway container, mitmproxy writes mitmproxy-ca-cert.pem a beat
after mitmdump boots, so ca_cert_pem() read it too early and launch died with
'gateway CA cert not available'. It now polls (up to 30s) until mitmproxy has
generated it. Surfaced running the docker backend locally on a cold gateway.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis e97366dad5 fix(orchestrator): gateway image builds cache-aware, not build-if-missing
The cut-over dropped compose (whose `build:` rebuilt the sidecar image on
every up), and DockerGateway.ensure_built was build-if-missing — so a change
to the gateway's flat sources (egress addon / git-http / policy_resolver /
supervise) left a STALE image silently running the old single-tenant daemons
(this bit e2e validation). ensure_built now always `docker build`s
(cache-aware: a cache check when nothing changed, a real rebuild when sources
moved), matching the old compose behavior. BOT_BOTTLE_NO_CACHE forces a full
rebuild (parity with #354's `start --no-cache`).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 8e7f5c9572 fix(tests): annotate on_network param — pyright strict (CI lint)
The cut-over's edit to test_consolidated_launch left the new on_network
kwarg untyped; pyright strict rejects it. No behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 69db629e16 feat(backend): slice 13d — cut docker launch over to the consolidated gateway (e2e green)
Rewire DockerBottleBackend.launch to the consolidated model, replacing the
per-bottle sidecar bundle. VALIDATED END-TO-END on real docker:
test_sandbox_escape passes all 5 attacks (egress DLP + git-gate gitleaks)
through the shared gateway.

launch now:
  - mints git-gate dynamic keys (if any), then launch_consolidated() to
    register the bottle + provision its git-gate state into the gateway and
    get the agent's attach context (pinned source IP, gateway address);
  - installs the SHARED gateway CA (gateway.ca_cert_pem) into the agent;
  - renders the agent-only consolidated compose on the gateway network at the
    pinned IP, proxied through the gateway;
  - points the agent's git-gate insteadOf (http://<gw>:9420) and supervise
    MCP (http://<gw>:9100) at the gateway (DockerBottlePlan.agent_git_gate_url
    / agent_supervise_url);
  - teardown = compose down + teardown_consolidated (dereg + deprovision).
Dropped the per-bottle networks, per-bottle egress CA, and bundle service.

Fixes surfaced by the real e2e (couldn't be caught by unit mocks):
  - provision_git_gate installs the bottle-agnostic pre-receive/access hooks
    into the gateway (were cp'd per-bundle before);
  - source-IP allocation reads the *actual* container IPs on the network
    (gateway + orchestrator + agents), not just gateway + registry — the
    orchestrator container sits on the network and was colliding.

Unit tests for the old bundle launch rewritten against the new collaborators.

NOTE for reviewers: the gateway/bundle image (bot-bottle-sidecars) must be
rebuilt when its flat sources change — `ensure_built` only builds-if-missing,
so a stale image silently runs the OLD single-tenant daemons. A content-hash
/ --rebuild path is a follow-up.

pyright 0 errors; pylint 9.83/10; unit suite green (1760 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise);
test_sandbox_escape green on real docker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 96f75599a1 feat(orchestrator): slice 13d(i) — containerize the orchestrator (validated on docker)
Real-on-host testing surfaced two issues the unit-mocked slices couldn't:

1. The host (NixOS) firewall DROPS container->host traffic, so a host-process
   orchestrator is unreachable from the gateway container. Fix: run the
   orchestrator AS a container on the shared gateway network (PRD 0070's
   "virtualize the orchestrator") — the gateway reaches it by container name
   over docker DNS (container<->container, no firewall), and the host CLI
   reaches it via a published loopback port.
2. The control plane crashed the connection on a dispatch error (e.g. a
   broker failure) instead of returning 500.

Changes:
- lifecycle: OrchestratorProcess (host process) -> OrchestratorService
  (containers). Runs the control plane in the bundle image with the repo
  bind-mounted (orchestrator is stdlib-only), register-only stub broker so it
  needs NO docker socket (the backend launches agents; the host manages both
  containers). Registry DB persists via a host-root mount. ensure_running is
  an idempotent singleton over both containers.
- gateway: BOT_BOTTLE_ORCHESTRATOR_URL is now the orchestrator's *by-name*
  URL on the shared network (dropped the host.docker.internal hack).
- control_plane: _serve wraps dispatch — a failure returns 500, never crashes
  the connection.
- OrchestratorProcess default broker -> stub (register-only) for docker.

Validated live end-to-end: both containers up, gateway->orchestrator by name
OK, register -> resolve-by-source-IP returns the bottle's policy from inside
the gateway.

pyright 0 errors; pylint 9.83/10; unit suite green (1760 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 3062fc9230 feat(orchestrator): slice 13c(viii) — agent-only consolidated compose render
The per-bottle model rendered a compose project with the agent + a sidecar
bundle on two per-bottle networks. Consolidated has no sidecars — one shared
gateway serves everyone — so this renders JUST the agent, attached to the
external shared gateway network at the orchestrator-allocated pinned IP, and
pointed at the gateway's address for egress.

- consolidated_agent_compose(plan, *, gateway_ip, source_ip, network):
  agent service only; networks {<gateway>: {ipv4_address: source_ip}} on the
  external gateway network; HTTPS_PROXY -> http://<gateway_ip>:9099; NO_PROXY
  includes the gateway address so git-http + supervise (also on the gateway)
  bypass the proxy and are reached directly. No depends_on/sidecars.
- Pure (takes the launch-time LaunchContext values), so it's unit-testable
  without docker.

pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init
/bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 404369a3e0 feat(orchestrator): slice 13c(vii) — stable shared gateway CA
The consolidated gateway TLS-intercepts every bottle, so all agents must
trust ONE CA. Rather than host-generate + mount a CA, let the gateway's own
mitmproxy generate its CA into a persistent named volume and extract the
cert for agents — no host openssl, keeps gateway.py lean.

- DockerGateway mounts GATEWAY_CA_VOLUME at /home/mitmproxy/.mitmproxy so the
  self-generated CA survives container recreation (it must not rotate, or
  already-launched agents would stop trusting the gateway).
- ca_cert_pem(): read the CA cert (PEM) out of the running gateway
  (docker exec cat mitmproxy-ca-cert.pem) — the agent installs this into its
  trust store to accept the gateway's bumped certs.

pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init
/bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 327e756eef feat(orchestrator): slice 13c(vi) — consolidated launch sequence (composition)
Composes the orchestrator primitives into the register/teardown sequence
that replaces the per-bottle bundle:

  launch_consolidated(egress_plan, git_gate_plan): ensure orchestrator +
    gateway up -> read the gateway network's subnet + the gateway's own
    address -> allocate the bottle a pinned source IP (skip the gateway +
    live bottles) -> register (egress policy blob + slug metadata) ->
    provision its git-gate repos/creds into the gateway. Returns a
    LaunchContext (bottle_id, identity_token, source_ip, network, gateway_ip,
    orchestrator_url) — everything the agent container needs to attach.
    Rolls the registration back if provisioning fails, so no orphan.

  teardown_consolidated(bottle_id): deregister + deprovision (idempotent).

The agent `docker run` itself stays the backend's job (provider
provisioning); this owns the orchestrator-facing wiring so the sequence is
unit-testable — collaborators mocked, next_free_ip + registration_inputs run
for real (IP allocation + policy blob asserted end to end).

pyright 0 errors; pylint 9.83/10; unit suite green (1755 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 444924d95d feat(git-gate+orchestrator): slice 13c(v) — provision git-gate into the running gateway
Places a bottle's git-gate credentials + bare repos into the live shared
gateway container when it registers — the execution side of the 13c(i)
provisioning render.

- provision_git_gate(gateway, bottle_id, plan): mkdir the per-bottle creds
  dir, docker cp each upstream's identity key (+ known_hosts when present)
  into /git-gate/creds/<bottle_id>/, then docker exec the namespaced,
  init-only script from git_gate_render_provision. No-op with no upstreams.
- deprovision_git_gate(gateway, bottle_id): rm the bottle's /git/<id> +
  creds on teardown (idempotent).
- bottle_id is validated (shell/path-safe alphabet) BEFORE any docker cp/rm,
  so a traversal id can never reach a path arg — the creds/repo dirs land in
  docker cp/rm arguments. Registry ids are token_hex; defense in depth.

pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init
/bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 118d81533b feat(orchestrator): slice 13c(iv) — gateway joins a shared network
The consolidated gateway ran on the default bridge; the model needs it on a
dedicated user-defined network that every agent bottle also joins, reaching
the gateway's egress / git-http / supervise ports by its address (no host
port publishing) — and the source IP the gateway attributes by is the
bottle's address on this network.

- GATEWAY_NETWORK = "bot-bottle-gateway"; DockerGateway gains a `network`
  param.
- ensure_running now ensures the network exists (idempotent create,
  tolerating a concurrent 'already exists') then runs with `--network`.
- Docker picks the subnet; the launcher reads it back (13c source-IP
  allocator) to hand each bottle a pinned address.

pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init
/bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 353b1ad984 feat(orchestrator): slice 13c(iii) — host-side control-plane client
The launch path's counterpart to the gateway-side PolicyResolver: a trusted
host-side HTTP client for the orchestrator control plane, so the CLI can
register/teardown/re-policy bottles. Stdlib-only.

- OrchestratorClient.register_bottle(source_ip, *, image_ref, metadata,
  policy) -> RegisteredBottle(bottle_id, identity_token)  (POST /bottles)
- teardown_bottle(id) -> bool (DELETE; 404 -> False, idempotent for cleanup)
- set_policy(id, policy) -> bool (PUT; live reload)
- list_bottles() / health()
- Unlike the fail-closed data-plane resolver, this is the trusted caller: a
  non-2xx (other than the meaningful 404s) raises OrchestratorClientError so
  the launch path surfaces failures rather than swallowing them.

pyright 0 errors; pylint 9.82/10; unit suite green (1742 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 40ad2364ab feat(orchestrator): slice 13c(ii) — shared-gateway source-IP allocator
Deterministic per-bottle address assignment on the consolidated docker
gateway's shared network — the address the gateway uses as its attribution
key. Pure ipaddress logic; the docker-specific inputs (subnet CIDR, the
gateway container's own address) are gathered by the caller and passed as
`taken`, keeping it testable without docker.

- next_free_ip(cidr, taken): lowest host address not reserved (network +
  broadcast excluded by hosts(); docker's router .1 excluded here) and not
  already assigned (gateway container + live bottles from the registry).
  NoFreeAddressError when the subnet is exhausted.

pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init
/bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 9e5d00c6fa feat(git-gate+orchestrator): slice 13c(i) — per-bottle git-gate provisioning render
The consolidated gateway serves every bottle's repos under /git/<bottle_id>/
(slice 10), so each bottle's bare repos + per-repo credential config must be
provisioned into that namespace. This adds the renderer for that; placing
the creds + running it inside the gateway is the launch-wiring step.

- Extract the credential-wiring `init_repo` shell into a shared
  `_git_gate_init_repo_fn(repo_root, creds_dir)` — one source for the
  security-sensitive logic. The single-tenant daemon entrypoint is
  byte-unchanged (still /git + /git-gate/creds; existing tests green).
- git_gate_render_provision(bottle_id, upstreams): posix-sh that inits ONE
  bottle's bare repos under /git/<bottle_id>/ reading creds from
  /git-gate/creds/<bottle_id>/. Init-only (no `git daemon` — the shared
  gateway already serves). Isolating each bottle's repo root + creds dir by
  id is what keeps one bottle's push credentials out of another's repos.
- bottle_id is embedded unquoted in the script, so it's restricted to a
  shell/path-safe alphabet (registry ids are token_hex; defense in depth).

pyright 0 errors; pylint 9.83/10; unit suite green (1724 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 0c158c5718 feat(orchestrator): slice 13b — consolidated registration inputs (egress policy)
Bridges the per-bottle `prepare` output to the consolidated registry:
`registration_inputs(plan)` turns a prepared egress plan into the
backend-neutral inputs `Orchestrator.launch_bottle` takes.

- egress_policy(plan): the policy blob = the exact routes YAML the
  per-bottle sidecar read from a file; the multi-tenant gateway's
  PolicyResolver now fetches it from the registry per request instead.
  Same egress_render_routes, so a bottle's allow-list is byte-identical
  moving onto the shared gateway.
- RegistrationInputs bundles policy + metadata; metadata carries the human
  slug so the shared registry can map a minted bottle id back to a name
  (console / supervise display).

Host-side glue (imports bot_bottle.egress) used by the launch path — not
the lean orchestrator control-plane process. Not yet wired into launch
(that's 13c/13d), consistent with the slice-6/10 pattern of landing the
primitive before the cut-over.

Key test: the policy round-trips through load_config back to the same
routes + log level — the resolver-served policy reconstructs the per-bottle
egress config exactly.

pyright 0 errors; pylint 9.83/10; unit suite green (1718 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis fe25ad7623 feat(orchestrator): slice 13a — orchestrator process lifecycle (idempotent singleton)
First sub-slice of docker launch integration: the foundation the CLI needs
to ensure exactly one orchestrator control plane + shared gateway is up
before registering/launching bottles. Does NOT touch the real launch path
yet — it's the lifecycle primitive the cut-over slices build on.

- OrchestratorProcess.ensure_running(): idempotent singleton — returns the
  control-plane URL if a healthy one already answers /health, else spawns
  `python -m bot_bottle.orchestrator --broker docker --gateway` detached
  (start_new_session, output tee'd to <root>/orchestrator.log) and polls
  /health until healthy or timeout (OrchestratorStartError).
- The control-plane port is the singleton key: a second orchestrator can't
  bind it, so a stray double-start fails fast rather than forking a rival.
- Host-process (not container) per the PRD's dev-harness sequencing — it
  already has the host user's docker access to broker launches; the
  data-plane gateway it manages is the container.

pyright 0 errors; pylint 9.83/10; unit suite green (1714 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 6570e9f044 feat(supervise+orchestrator): slice 12 — multi-tenant git-gate hook + supervise server
Finishes the supervise data-plane writers for the shared gateway. Both
still keyed off a single SUPERVISE_BOTTLE_SLUG env; now each proposal is
attributed to the calling bottle, keeping slices 10/11's per-bottle model.

- git_http_backend: in consolidated mode stamp SUPERVISE_BOTTLE_SLUG=
  <bottle_id> into the CGI env. The gitleaks-allow pre-receive hook runs as
  a child of the `git http-backend` we spawn, so it inherits the stamp and
  queues under the right bottle — no change to the (fragile) embedded hook.
  The bottle id is the namespaced root's final component (slice 10), the
  same key egress uses. Single-tenant leaves the container-stamped slug.
- supervise_server: resolve the proposal slug per request by source IP when
  BOT_BOTTLE_ORCHESTRATOR_URL is set (`_attributed_config`), fail-closed —
  an unattributed / unreachable source raises rather than queue under the
  wrong or empty slug. `serve`/`main` build + attach the resolver and make
  the env slug optional in consolidated mode. Single-tenant unchanged.

Tests: a real git push proving the slug reaches the hook's env; the
supervise attribution matrix (single-tenant slug kept, source-IP bottle
bound, unattributed + resolver-error fail closed).

Remaining supervise gap (noted): websocket DLP still self.config-only.

pyright 0 errors; pylint 9.83/10; unit suite green (1705 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 39c823d3c0 feat(supervise+orchestrator): slice 11 — per-bottle supervise queue + DLP safelist
Consolidated egress ran every bottle through one process but keyed the
supervise proposal queue off a single SUPERVISE_BOTTLE_SLUG env and kept
one *global* DLP safelist — so in the shared gateway an operator's token
approval for bottle A would (a) be attributed to the wrong bottle and
(b) leak into bottle B's DLP scan (A's approved secret passes B's egress).
This slice keys both per bottle, resolved by source IP.

- policy_resolver: add `resolve_policy_and_bottle_id` — policy + bottle id
  in one `/resolve`, so egress keys routing *and* the supervise
  queue/safelist from a single round-trip. Fail-closed (403 -> (None,None)).
- egress_addon_core: add `resolve_client_context` (+ `ContextResolverLike`)
  returning `(Config, bottle_id)`, sharing the fail-closed parse with
  `resolve_client_config` via `_config_from_policy`.
- egress_addon: `_active_config` -> `_resolve_flow` returns `(Config, slug)`;
  `safe_tokens` set -> per-bottle `_safe_tokens_for(slug)`; the token-allow
  write/await/archive + the approved-token add all use the resolved slug.
  Single-tenant (no resolver) unchanged — slug = the env SUPERVISE_BOTTLE_SLUG.

New tests cover the resolver, the fail-closed context matrix, and the
cross-tenant isolation (an approval lands only in the calling bottle's
safelist; the proposal is keyed by the source-IP-attributed bottle;
unattributed IPs can't supervise).

Out of scope (noted): the git-gate gitleaks-allow hook + supervise_server
agent-proposal paths, and websocket DLP (still self.config-only, inert in
consolidated mode) — follow-up slices.

pyright 0 errors; pylint 9.83/10; unit suite green (1700 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 8a44537f7e fix(git-gate): review — sandbox naming, ResolverLike Protocol, token-required note
Addresses PR #365 review:
- Type `resolve_sandbox_root`'s resolver param as a `ResolverLike` Protocol
  (structural, `resolve_bottle_id` only) — fixes the pyright errors from
  passing a duck-typed fake resolver in tests; mirrors
  egress_addon_core.PolicyResolverLike.
- Rename `_project_root`/`project_root`/`resolve_repo_root`/
  `DEFAULT_PROJECT_ROOT` -> `_sandbox_root`/`sandbox_root`/
  `resolve_sandbox_root`/`DEFAULT_REPO_ROOT`: "sandbox" names the per-tenant
  scope; "project" was too amorphous. `GIT_PROJECT_ROOT` keeps git's own
  env-var name.
- Note the single-tenant path is transitional (stripped once every backend
  runs the consolidated gateway).
- policy_resolver: document that the optional identity token is
  transitional — the consolidated end state requires it (flips at the
  /resolve boundary once token *delivery* lands, a PRD 0070 open question).
- Split the long line in main() (was 105 cols).

pyright 0 errors; pylint 9.95/10; unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 72ee1f77da feat(git-gate): source-IP-keyed multi-tenant repo namespace (PRD 0070)
First slice of git-gate consolidation: make the smart-HTTP git backend
serve every bottle from one process, selecting each request's repo root by
the calling bottle's source IP — the same attribution invariant + resolver
the multi-tenant egress addon uses. `git daemon` can't source-IP-route per
connection, so the consolidated gateway serves git-gate over this HTTP
backend (the transport firecracker/macOS already use); wiring the docker
path onto it lands with the launch-integration slice.

- policy_resolver: factor out `_post_resolve`; add `resolve_bottle_id`
  (source IP -> bottle id, same fail-closed 403->None contract as
  `resolve`) — the git-gate has no policy blob to parse, the bottle *is*
  the namespace.
- git_http_backend: `resolve_repo_root(resolver, base, source_ip, token)`
  — single-tenant passthrough when no resolver; else `<base>/<bottle_id>`,
  fail-closed on unattributed / resolver error / namespace escape.
  `BOT_BOTTLE_ORCHESTRATOR_URL` (same env as egress) flips the shared
  gateway multi-tenant; the identity header is read for attribution and
  never forwarded to the CGI. Per-repo creds + hooks scope by repo dir, so
  isolating the root per bottle isolates its creds too.

Single-tenant path unchanged (existing real-git-push tests green). New unit
coverage for the resolver + repo-root selection matrix. Full unit suite
green (1690 tests; the 13 test_sidecar_init /bin/sleep errors are the
pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 00418a2834 refactor(orchestrator): rename Sidecar -> Gateway for the consolidated data plane
Retire "sidecar" for the consolidated per-host path (PRD 0070 naming
decision): the orchestrator is the umbrella/control plane, and the
egress/git/supervise data-plane unit it runs is the "gateway".

- git mv sidecar.py -> gateway.py and the two integration + one unit test
  files; DockerSidecar->DockerGateway, Sidecar->Gateway,
  SidecarError->GatewayError, SIDECAR_*->GATEWAY_*, ensure_sidecar->
  ensure_gateway, sidecar_status->gateway_status, container name
  bot-bottle-orch-sidecar->bot-bottle-orch-gateway.
- Prose rename across broker/registry/egress/policy_resolver + PRD 0070.
- Preserved: the image name bot-bottle-sidecars, the
  BOT_BOTTLE_SIDECAR_IMAGE env var, Dockerfile.sidecars, and PRD 0069's
  own stage-name cross-references (that doc still uses "sidecar").

No behavior change. Full unit suite green (1679 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 2cbb178f88 feat(orchestrator+egress): slice 8 — multi-tenant egress via the resolver (#352)
The egress addon now selects each request's Config by the calling bottle's
source IP, so one shared sidecar serves every bottle. Opt-in and fail-closed;
single-tenant behaviour is unchanged.

Orchestrator side (source-IP-primary attribution, per the PRD invariant):
  * registry: `by_source_ip` (the single active bottle at a source IP —
    network-layer attribution); `attribute` now composes it + the token.
  * service: `resolve(source_ip, token="")` — with a token, strict
    attribution; without, source IP alone.
  * control_plane: `POST /resolve`'s identity_token is now OPTIONAL (absent
    → source-IP-only); split cleanly from the token-required `/attribute`.
  * policy_resolver: `resolve` token now optional.

Egress side:
  * egress_addon_core: `resolve_client_config(resolver, client_ip, token)` —
    fetches + parses the client's Config, **fail-closed**: unattributed, a
    resolver error, or an unparseable policy all yield deny-all (no routes).
    Host-testable; `PolicyResolverLike` Protocol keeps it import-free.
  * egress_addon: consolidated mode when `BOT_BOTTLE_ORCHESTRATOR_URL` is
    set → `_active_config(flow)` resolves per client IP (reads + strips the
    `x-bot-bottle-identity` header); `request()` uses it. Unset → the static
    routes file, exactly as before. `PolicyResolver` added to the bundle.

Security note: source-IP-only resolution is safe where the IP is unspoofable
(Firecracker /31 + nft) AND the control plane is reachable only by the
trusted sidecar; the identity token, when the agent injects it, strengthens
it on weaker backends.

Scope note: the egress data plane is now multi-tenant. Remaining to be fully
live: the network topology routing every bottle's proxy to the one shared
sidecar, git-gate multitenancy, and agent-side identity-token injection.

Tests: registry by_source_ip; orchestrator resolve (with/without token);
control-plane /resolve token-optional; resolver token-optional;
resolve_client_config fail-closed matrix. All 182 egress tests still pass
(single-tenant unchanged). Full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis ed9a19bb38 refactor(orchestrator): drop the PolicyResolver cache (#352)
Review: the resolver is called rarely enough that a round-trip doesn't
matter, and correctness beats speed — always fetching means a revocation,
policy change, or teardown the orchestrator knows about is honored
immediately instead of lingering for a cache TTL. Remove the TTL cache
(and `invalidate`); `resolve` now hits the orchestrator every call. Noted
in the docstring that any future caching should use orchestrator-driven
invalidation, not a blind TTL.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis c5ae93c8ba feat(orchestrator): slice 7 — sidecar-side PolicyResolver (#352)
The data-plane bridge that lets the consolidated sidecar apply each
bottle's policy per request. `PolicyResolver` resolves a client's policy
from the orchestrator's `POST /resolve` keyed on (source_ip, identity
token) and caches it briefly (short TTL) so it isn't a round-trip per
request; `invalidate()` drops an entry on teardown / live reload.

Fail-closed: an unattributed client (orchestrator answers 403) resolves to
None so the caller denies; unreachable / unexpected status raises so the
caller can fail closed too rather than serve stale/empty policy. Stdlib
only and free of bot-bottle imports, so it can be COPYed flat into the
sidecar bundle.

Scope note: this is the sidecar-side *client*. Wiring it into the live
egress mitmproxy addon (select `Config` per client IP in the request path)
and git-gate, plus routing all bottles' egress to the one shared sidecar,
are the remaining data-plane pieces — a heavier change to the sidecar
bundle's adversarial-input code, taken next.

Tests: resolve returns/caches/expires/invalidates; 403 -> None (fail
closed); other HTTP status + unreachable raise; missing policy -> empty;
posts source_ip + identity_token. Full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 186b905dff feat(orchestrator): slice 6 — source-IP-keyed multi-tenant policy (#352)
The orchestrator side of the multi-tenant consolidated sidecar: hold each
bottle's sidecar policy and serve it by verified source IP, with live
reload. One shared sidecar can now get per-bottle config keyed on who's
calling.

  * registry.py — a `policy` column (migration v3, opaque JSON the sidecar
    interprets) on BottleRecord; `register(..., policy=)` stores it,
    `set_policy(bottle_id, policy)` updates it live, and `attribute` returns
    it (the source-IP-keyed resolution).
  * service.py — `launch_bottle(..., policy=)` and `set_policy`.
  * control_plane.py — `POST /bottles` accepts `policy`; `PUT
    /bottles/<id>/policy` live-reloads it; `POST /resolve` returns
    {bottle_id, policy} for a verified (source_ip, token) — the per-request
    call the multi-tenant sidecar makes; `/attribute` stays identity-only.

Scope note: this is the control-plane / state half. The data-plane half —
the egress mitmproxy addon (and git-gate) selecting allowlist / DLP /
token-injection per client IP by calling `/resolve` — is the next slice
(route agent bottles through the shared sidecar). The orchestrator stays
policy-agnostic: it stores and serves the blob verbatim.

Tests: registry policy store/update/persist; Orchestrator launch-with-policy
+ live set_policy; control-plane resolve returns policy (403 on bad token),
PUT policy updates / 404 / 400. Verified live over HTTP (launch -> resolve
-> PUT reload -> resolve reflects). Full suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 3519b924cb feat(orchestrator): slice 5 — build the consolidated sidecar bundle image (#352)
Answers "where do we build the consolidated sidecar": nowhere, until now.

  * sidecar.py — `Sidecar.ensure_built()` (default no-op) + `DockerSidecar`
    now defaults its image to the real bundle (`bot-bottle-sidecars`) and
    `ensure_built()` builds it from `Dockerfile.sidecars` when
    `docker image inspect` shows it's missing (no-op when present or when no
    dockerfile is configured, e.g. a pre-pulled image). `image_exists()`
    added.
  * service.py — `ensure_sidecar()` now builds then runs.
  * __main__.py — `--sidecar` runs the consolidated bundle (build-if-missing).

Scope note: this builds + launches the bundle *container*; making the
running instance functional across bottles needs the per-bottle,
source-IP-keyed multi-tenant config + registration/reload, and routing
agent bottles to it — the next slices (added to PRD 0070's roadmap).

Tests: unit (docker mocked) — image_exists, ensure_built builds when
missing / no-op when present / no-op without a dockerfile / raises on build
failure; ensure_sidecar builds-then-runs; integration (gated, no heavy
build) — image_exists reflects real docker state. Full suite green (only
pre-existing /bin/sleep errors).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis c77f578fb4 refactor(orchestrator): move run_docker to a lean top-level docker_cmd (#352)
Review feedback: don't bury a parallel docker util in the orchestrator.
But reusing backend.docker.util (docker_mod) isn't right either — importing
it runs backend/__init__.py, which eagerly loads all three backends
(docker + firecracker + macos) plus the manifest/egress/git-gate/supervise
framework (~76 modules), so every orchestrator import would drag the whole
backend layer in.

Compromise: promote the helper to a top-level, framework-free
bot_bottle/docker_cmd.py (single stdlib import), a proper shared home the
orchestrator's docker components use now and backend.docker.util can adopt
later. Verified `import bot_bottle.orchestrator` stays lean (12 modules, no
firecracker/macos backends).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis bc52c3525c feat(orchestrator): slice 4 — consolidated per-host sidecar (#352)
The core consolidation win: one persistent sidecar per host, shared by
every bottle, instead of a sidecar bundle per bottle. Safe to share
because the attribution invariant (source IP + identity token) lets the
sidecar map each request to the right bottle.

  * orchestrator/sidecar.py — a backend-neutral `Sidecar` lifecycle
    contract (mirrors LaunchBroker) + a `DockerSidecar` impl. The defining
    behaviour is idempotent singleton: `ensure_running` starts the instance
    if absent and is a no-op if it's already up, so N launches never spawn
    N sidecars; `stop` is idempotent.
  * orchestrator/dockerutil.py — a shared `run_docker` helper; DockerBroker
    now uses it too (DRY with slice 3).
  * service.py — the Orchestrator holds an optional `Sidecar`, exposes
    `ensure_sidecar()` + `sidecar_status()`.
  * control_plane.py — `GET /sidecar` reports it; __main__ gains
    `--sidecar-image` and ensures the single sidecar on startup.

Tests: unit (docker mocked) — is_running, ensure idempotent (no-op when up,
starts when absent), failure raises, stop idempotent; Orchestrator sidecar
wiring/status; control-plane /sidecar; integration (gated) — ensure is a
real idempotent singleton (one container after two ensures), stop removes.
Full suite green (only pre-existing /bin/sleep errors); integration
verified locally against real docker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis b70f3228ca fix(orchestrator): pyright — pass explicit LaunchRequest in the docker integration test (#352)
The **kw unpacking put str into slot: int|None. Pass explicit
LaunchRequests instead.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis f7b3d6aaca feat(orchestrator): slice 3 — real Docker launch broker (#352)
The first concrete LaunchBroker, proving the orchestrator -> backend seam
on the cheapest backend (the sidecar bundle is already containers):

  * orchestrator/docker_broker.py — DockerBroker runs a container on a
    verified launch (`docker run --detach --name <bottle> --label ...
    <image_ref>`) and removes it on teardown (`docker rm --force`,
    idempotent on an already-absent container). The argv is built only from
    the request's static ids/flags, so nothing free-form reaches docker;
    provenance/schema verification is inherited from LaunchBroker.submit.
  * __main__.py gains `--broker {stub,docker}` so the harness can drive real
    containers.

Slice 3 launches a single container from image_ref (the seam); the full
agent + sidecar bundle is a later slice.

Tests: unit (docker mocked) — argv from static fields, launch/teardown call
the right commands, missing-image and docker-failure raise, teardown
idempotent on missing, forged token never touches docker; integration
(gated on a reachable daemon) — launch creates a real container, teardown
removes it. Full suite green (only pre-existing /bin/sleep errors);
integration verified locally against real docker.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis b05f245299 feat(orchestrator): slice 2 — launch lifecycle + signed launch-broker (#352)
Second slice of PRD 0070, still a backend-neutral dev-harness:

  * orchestrator/broker.py — the launch-broker contract. A LaunchRequest is
    structured (static ids/flags only — bottle id, pool slot, a
    content-addressed image_ref; never a path/argv) and signed as a compact
    HS256 JWT so the broker verifies PROVENANCE before acting: a compromised
    co-located component can't forge a launch without the shared secret.
    verify_request is fail-closed (bad sig / malformed / off-schema -> raise).
    Stdlib only (no runtime deps). Ships a StubBroker that records verified
    requests for the harness/tests.
  * orchestrator/service.py — the Orchestrator: owns the registry and brokers
    the lifecycle. launch_bottle mints the bottle + sends a signed launch,
    rolling the registry entry back if the launch fails (no orphans);
    teardown_bottle brokers teardown then deregisters; attribute delegates.
  * control_plane.py — POST /bottles now launches, DELETE tears down (both go
    through the Orchestrator + broker). dispatch/server take an Orchestrator.
  * __main__.py wires an ephemeral secret + StubBroker for the harness.

Tests: broker sign/verify round-trip, tamper/wrong-secret/malformed/off-schema
rejection, StubBroker fail-closed; Orchestrator launch->registry->attribute,
teardown, rollback-on-broker-failure; control-plane updated for launch/teardown.
Full suite green (only the pre-existing /bin/sleep errors); harness does
launch -> attribute -> teardown over HTTP.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 8d4e7f2582 refactor: drop the vestigial bot_bottle_root/host_db_path re-exports (#352)
paths is the single home now, so stop re-exporting the path helpers from
supervise: remove host_db_path from supervise's imports + __all__ (it was
re-export-only) and drop bot_bottle_root from __all__ (kept as an import,
still used by audit_dir). supervise_types was already clean. Repoint the
last readers (test_supervise imports host_db_path from paths;
test_supervise_edge calls paths.bot_bottle_root) and refresh the doc
mentions. No supervise.bot_bottle_root / supervise.host_db_path references
remain.

Behavior-preserving: full unit suite unchanged (only the pre-existing
/bin/sleep sidecar-init errors).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis a0e17d56de refactor: move bot_bottle_root/host_db_path to paths; kill the monkeypatch (#352)
Create bot_bottle/paths.py as the canonical home for the app-root path
helpers (bot_bottle_root, host_db_path, HOST_DB_FILENAME) — foundational,
not supervise- or db-specific. `bot_bottle_root()` now honours a
BOT_BOTTLE_ROOT env override.

Repoint every consumer (supervise, supervise_types, db_store, queue_store,
audit_store, store_manager, bottle_state, cli/supervise, docker/cleanup,
orchestrator/registry) at paths; remove the definitions (and supervise's
duplicate host_db_path) and the now-dead `import sys`. Add paths.py to the
sidecar bundle (Dockerfile.sidecars) for the flat-import copies.

Tests: replace ~12 files' monkeypatching of supervise.bot_bottle_root (and
the flat/pkg/supervise_types triple-patch dance) with a single
`use_bottle_root()` helper that sets BOT_BOTTLE_ROOT — every module and
flat/package copy reads the same env var, so one override covers them all.
Net -97 lines. Behaviour-preserving: full unit suite unchanged (only the
pre-existing /bin/sleep sidecar-init errors remain).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 934c1617b9 refactor: move host_db_path/HOST_DB_FILENAME to db_store (#352)
host_db_path is shared DB infrastructure, not supervise-specific, so its
canonical home is db_store (alongside DbStore). It resolves bot_bottle_root
from supervise_types lazily inside the function — no load-time cycle, and a
monkey-patch of supervise_types.bot_bottle_root still propagates.
supervise_types re-exports both names for the historical import path
(queue_store/audit_store unchanged); the orchestrator registry now imports
from db_store. Drops the now-unused `import sys` from supervise_types.

Behavior-preserving: full unit suite unchanged (only the pre-existing
/bin/sleep sidecar-init errors remain); monkeypatch propagation verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis bd964c0278 feat(orchestrator): registry co-tenants the shared bot-bottle.db (#352)
Per review: use one shared bot-bottle.db for all runtime state including
the registry (DbStore namespaces by schema_key), so it's one queryable
file for backup/console. default_db_path() -> host_db_path(). Drop the
unilateral WAL flip — WAL on the shared DB affects supervise/audit and is
finicky over guest shares, so it's a deliberate future change; keep a
busy_timeout for lock contention.

PRD State section updated: integrity now by SOLE ownership (only the
orchestrator opens bot-bottle.db; data plane + console reach state via the
control-plane RPC, never a file handle) rather than ro/rw mount-splitting,
which one shared file can't do. Notes the transitional caveat that the
supervise sidecar currently rw-mounts bot-bottle.db.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 93756f2a8b docs(prd): 0070 registry DB is host-resident with rw/ro access split (#352)
Per review: the SQLite runtime-state DB lives on the host, not owned
inside the orchestrator unit. Durability (re-adoption must survive an
orchestrator restart) + integrity via access-scoping (control-plane rw,
data-plane ro) rather than location. Note the WAL-over-guest-share wrinkle
for the VM slices (may want a host-side DB owner reached over the RPC).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis d9843fa41e feat(orchestrator): slice 1 — registry + attribution + HTTP control plane (#352)
First implementation slice of PRD 0070, the backend-neutral consolidation
core as a plain-process dev-harness (no VM packaging yet):

  * orchestrator/registry.py — SQLite (WAL) runtime-state store on the
    existing DbStore/TableMigrations base. Live bottle registry keyed by
    source IP + per-bottle identity token, with fail-closed attribution:
    a request resolves to a bottle only when its source IP AND identity
    token both match exactly one active record (unknown/ambiguous IP,
    empty token, or token mismatch all deny). Tokens are 256-bit urandom.
  * orchestrator/control_plane.py — the HTTP control plane (the universal
    transport chosen in 0070): register / deregister / list / attribute /
    health. Routing is a pure dispatch() so it is socket-free testable;
    Handler/ControlPlaneServer/make_server are a thin stdlib adapter.
    register/deregister are the live-reload path; listing redacts tokens.
  * orchestrator/__main__.py — `python -m bot_bottle.orchestrator` harness.

Launch/teardown, the launch broker, and the egress/git/supervise data
plane come in later slices. 24 unit tests (attribution matrix, persistence,
dispatch, one real-socket round-trip).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis ae6c8b13d9 docs(prd): 0070 VM-to-VM routing is not a blocker (#352)
Per review: resolvable at implementation time (host tweak acceptable, as
the pool setup already needs on NixOS); the earlier sidecars-in-VMs spike
showed feasibility. No need to hunt the spike now.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 13a8d66c9b docs(prd): 0070 fold in review decisions (#352)
Resolve open questions from review: consolidate egress (hardened minimal
surface + identity token + vault mitigations); HTTP control-plane
transport; broker schema = signed-JWT JSON of static flags+ids (provenance
+ un-coercible); state re-adoption procedure (singleton orchestrator →
wait-healthy → adopt via SQLite + process inspection before serving, with
write-ahead intent to close the in-flight-launch race). Add a per-bottle
identity-token defense-in-depth layer on the attribution invariant.
Remaining open: VM-to-VM routing (per-backend wire(), pending spike link),
live-reload protocol, identity-token delivery.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis c46d392b44 docs(prd): 0070 secret-handling as a future pattern (SecretProvider, #355)
Add a "Secret handling — FUTURE pattern (not v1)" subsection: vault as a
separate trust domain holding long-lived roots, deriving short-lived
scoped creds where the upstream allows (with the honest limit that a
compromised proxy can still abuse currently-authorized access). The
mechanism is a generic, user-extensible SecretProvider that generalizes
PRD 0048's DeployKeyProvisioner and drops into the manifest wherever a raw
token is accepted — discovered from ~/.bot-bottle/contrib like user
AgentProviders. Marked explicitly as not required for the initial
orchestrator; tracked as #355.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 2a473eb404 docs(prd): 0070 per-host orchestrator service
Fold the per-bottle sidecar bundle into a single persistent per-host
orchestrator: runs the sidecar functions (egress/git-gate/supervise),
coordinates with the console, and brokers agent launches. Virtualized
from the start with backend-native isolation (fc VM / apple ctr / docker
ctr), fronted by a single backend-agnostic contract; per-backend
variation lives on BottleBackend, not a parallel Orchestrator hierarchy.

Leads with the security review (secret concentration, shared fate, the
launch-broker-as-new-privileged-core, and the source-IP attribution
invariant each backend must enforce). Proposes one SQLite DB owned by the
orchestrator for runtime state (slot leases, approvals, registry) —
distinct from build-time constants (flat .env) and user config
(declarative ~/.bot-bottle). Sequences docker -> firecracker -> macos,
developing the service as a plain-process dev-harness before the VM.

Supersedes 0069's Stage-1/4 sidecar framing; depends on 0069's nix-built
fixed images. Tracking issue #351.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-14 02:38:20 -04:00
didericis 4d89ffce8b docs(research): add local/internal-reach edge to the OneCLI comparison
Self-hosting means the agent runs inside your own network boundary (homelab,
corporate LAN, Tailnet), so scoping it to internal resources is just an
egress-route line — a reach advantage distinct from the isolation ones, which a
cloud-first credential broker / agent product can't match.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LhiafsABCr46bu3oHUm7wa
2026-07-14 01:48:48 -04:00
didericis 822cff48ad docs(research): mark credential-proxy note as implemented (agents get placeholders)
The note still described the pre-gateway state as "today" — raw provider
tokens env-injected into the agent, readable via printenv — which is no longer
true and was actively misleading. Add a dated Status banner and flag the stale
"today's wiring" / "recommended path forward" passages: the agent now holds only
a placeholder (e.g. CLAUDE_CODE_OAUTH_TOKEN=egress-placeholder) and the real
token is injected as the upstream Authorization header by the existing
pipelock/mitmproxy egress sidecar (EgressRoute auth_scheme + token_ref, one MITM
not two), generalized across Claude/Codex/Pi and git-host tokens. Reasoning
preserved per the research-note convention.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LhiafsABCr46bu3oHUm7wa
2026-07-14 01:42:05 -04:00
didericis ce440fabc4 docs(research): refresh OneCLI competitor entry + product verdict
OneCLI (onecli.sh) was already tracked in the credential-proxy landscape but
the entry was stale (May 2026). Correct it: it uses the phantom-token pattern
this note recommends (not "Bitwarden integration"), and it's now GA, Rust,
YC-backed (~2.5k stars, 300k+ downloads). Add build-vs-adopt + competitor
commentary, and a product-side entry in the containerized-claude landscape with
a verdict on how close a competitor it is and where bot-bottle's edge lies
(isolation as the product, fleet/manifest layer, self-hosted trust posture).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01LhiafsABCr46bu3oHUm7wa
2026-07-14 01:23:32 -04:00
148 changed files with 7545 additions and 3450 deletions
+3 -3
View File
@@ -8,10 +8,10 @@ broad permissions inside a sandbox, so a misbehaving agent cannot reach the
host. A Python CLI (entry point `cli.py`, package `bot_bottle/`) orchestrates
the runtime lifecycle and the copying of skills and env vars into it.
The default backend on compatible macOS hosts is macos-container:
agents and sidecar bundles run through Apple's `container` CLI without
agents and gateways run through Apple's `container` CLI without
requiring Docker. On KVM-capable Linux hosts the default is firecracker:
agents run in a Firecracker microVM reached over SSH on a point-to-point
TAP, while the sidecar bundle still uses Docker. The legacy Docker
TAP, while the gateway still uses Docker. The legacy Docker
backend remains available with `BOT_BOTTLE_BACKEND=docker` or
`--backend=docker`.
@@ -59,7 +59,7 @@ backend remains available with `BOT_BOTTLE_BACKEND=docker` or
in a PRD, research note, or decision record.
- Low dependencies by default. The project is Python, stdlib-first (no
runtime pip dependencies in the package itself; the only language
runtime is the Python 3.13 used by the CLI + sidecars). Ask before
runtime is the Python 3.13 used by the CLI + companion containers). Ask before
adding new tools, runtimes, or package managers.
- Commit messages follow [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/):
`<type>[(scope)][!]: <description>`, where `<type>` is one of `feat`, `fix`,
+41 -24
View File
@@ -1,8 +1,15 @@
# Per-bottle sidecar bundle image (PRD 0024).
# Gateway data-plane image (PRD 0024 bundle shape; PRD 0070 gateway).
#
# Collapses the prior per-sidecar images (egress, git-gate,
# The egress / git-gate / supervise *data plane* — one image, run by
# the consolidated per-host gateway (PRD 0070). It is NOT the
# orchestrator control plane: that is the separate, lean
# `bot-bottle-orchestrator` image (Dockerfile.orchestrator, #384), which
# ships only python + the stdlib-only `bot_bottle` package and none of
# this image's mitmproxy / git / gitleaks payload.
#
# Collapses the prior per-daemon images (egress, git-gate,
# supervise) into one. A small stdlib-Python init supervisor at
# /app/sidecar_init.py spawns all daemons, forwards SIGTERM, and
# /app/gateway_init.py spawns all daemons, forwards SIGTERM, and
# propagates per-daemon stdout/stderr to the container log with a
# `[name]` prefix. See PRD 0024 for the rationale.
#
@@ -12,7 +19,7 @@
# /app/egress_addon.py + siblings mitmproxy addon (egress)
# /app/egress-entrypoint.sh mitmdump launcher
# /app/supervise_server.py + .py supervise MCP server
# /app/sidecar_init.py PID 1 supervisor
# /app/gateway_init.py PID 1 supervisor
# /etc/egress/routes.yaml bind-mounted at run time
# /etc/git-gate/pre-receive docker-cp'd at start time
# /git-gate-entrypoint.sh docker-cp'd at start time
@@ -27,35 +34,45 @@
# 9420 git-gate smart HTTP (VM-backend agent-facing transport)
# 9100 supervise (MCP HTTP)
# Stage 1: gitleaks binary. The upstream gitleaks image is alpine
# with the binary at /usr/bin/gitleaks. Pinned by digest in lockstep
# with Dockerfile.git-gate's prior base (now deleted at chunk 3).
FROM zricethezav/gitleaks@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f AS gitleaks-src
# Stage 2: assembly. mitmproxy/mitmproxy is debian-slim-based with
# Python + mitmdump pre-installed — heavier than the others, so
# this stage starts there and pulls the standalone binaries in.
FROM mitmproxy/mitmproxy:11.1.3
# Run as root inside the bundle. The bundle is the isolation
# boundary; per-daemon user separation inside it is not load-bearing
# and complicates the supervisor's spawn path.
USER root
# Based on `python:3.12-slim` (Debian trixie) rather than the
# `mitmproxy/mitmproxy` image (Debian bookworm) so the whole stack —
# gateway here, and the firecracker infra image that builds FROM this —
# lands on trixie, whose buildah (1.39) can build agent Dockerfiles that
# use heredocs. mitmproxy is pip-installed to the same effect as the
# upstream image. (bookworm's buildah is 1.28, which can't parse
# `RUN ... <<EOF`; see the infra image + PR discussion.)
FROM python:3.12-slim
# Runtime system deps:
# git supplies the `git daemon` subcommand (no separate package)
# plus the core `git` binary the pre-receive hook invokes.
# openssh-client supplies the upstream SSH transport the
# pre-receive hook uses to forward accepted refs.
# ca-certificates is needed for mitmdump upstream TLS (the
# base image already has it; listed for explicitness).
# ca-certificates is needed for mitmdump upstream TLS.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
git openssh-client ca-certificates \
&& rm -rf /var/lib/apt/lists/*
# Pull the standalone binaries into the final image.
COPY --from=gitleaks-src /usr/bin/gitleaks /usr/bin/gitleaks
# mitmdump (the egress data plane). The upstream mitmproxy image baked
# this in; on the plain python base we pip-install the same pinned
# version. Its CA dir is set explicitly via `--set confdir=` in
# egress-entrypoint.sh, so it doesn't depend on a `mitmproxy` home user.
RUN pip install --no-cache-dir mitmproxy==11.1.3
# gitleaks (the pre-receive hook's secret scanner). Installed from its
# official release, pinned by version + SHA256 and verified — rather than
# using a third-party image as a build stage (supply-chain surface, and it
# would pin us to that image's cadence). python (already present) does the
# download so we add no curl/wget. trixie apt also ships gitleaks, but an
# older 8.16; the pinned download keeps the verified 8.30.1.
ARG GITLEAKS_VERSION=8.30.1
ARG GITLEAKS_SHA256=551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb
RUN url="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
&& python3 -c "import sys,urllib.request; urllib.request.urlretrieve(sys.argv[1], '/tmp/gitleaks.tar.gz')" "$url" \
&& echo "${GITLEAKS_SHA256} /tmp/gitleaks.tar.gz" | sha256sum -c - \
&& tar -xzf /tmp/gitleaks.tar.gz -C /usr/bin gitleaks \
&& rm /tmp/gitleaks.tar.gz
# Project Python: addon + server modules + the init supervisor.
# Kept flat under /app/ so mitmdump's loader resolves them as
@@ -76,7 +93,7 @@ COPY bot_bottle/audit_store.py /app/audit_store.py
COPY bot_bottle/store_manager.py /app/store_manager.py
COPY bot_bottle/supervise.py /app/supervise.py
COPY bot_bottle/supervise_server.py /app/supervise_server.py
COPY bot_bottle/sidecar_init.py /app/sidecar_init.py
COPY bot_bottle/gateway_init.py /app/gateway_init.py
COPY bot_bottle/git_http_backend.py /app/git_http_backend.py
COPY bot_bottle/egress_entrypoint.sh /app/egress-entrypoint.sh
RUN chmod +x /app/egress-entrypoint.sh
@@ -102,4 +119,4 @@ WORKDIR /app
# PID 1 is the supervisor. It owns signal handling and exit-code
# propagation; no `exec` chain in the entrypoint itself.
ENTRYPOINT ["python3", "/app/sidecar_init.py"]
ENTRYPOINT ["python3", "/app/gateway_init.py"]
+45
View File
@@ -0,0 +1,45 @@
# Firecracker single infra-VM image (PRD 0070 Stage B).
#
# The per-host infra VM runs the orchestrator control plane, the gateway
# data plane, AND builds agent images (buildah) — all in one microVM (see
# backend/firecracker/infra_vm.py). It composes:
# * FROM the gateway image (mitmproxy / git / gitleaks / supervise + the
# flat daemon modules) — now trixie-based, so buildah 1.39 is available;
# * `COPY --from` the orchestrator image's content (the single definition
# of the control-plane payload — see Dockerfile.orchestrator), so this
# VM and the docker backend share one orchestrator definition; and
# * buildah, installed HERE only (the docker orchestrator/gateway images
# never carry it).
#
# multi-`FROM` can't union two bases (that's multi-stage, not multiple
# inheritance), so the orchestrator content is pulled in via `COPY --from`
# rather than a second base. Both images share the trixie `python:3.12-slim`
# base, so the copy is clean (same python; future installed deps copy too).
#
# The docker backend keeps orchestrator + gateway as separate images; this
# combined image exists only for the Firecracker single-VM cut. Splitting a
# service back into its own VM later is a routing change, not a repackaging
# (PRD 0070's "secret concentration"; a disposable builder can boot from
# this same image on its own TAP).
FROM bot-bottle-gateway:latest
# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
# The Firecracker backend builds users' agent Dockerfiles *inside this VM*
# with buildah (rootless, daemonless) instead of on the host — no host
# Docker daemon, no root-equivalent `docker` group. `crun` is the OCI
# runtime; `netavark` + `aardvark-dns` are the network backend for `FROM`
# pulls + `RUN` egress. Requires the trixie base (buildah 1.39: bookworm's
# 1.28 can't parse Dockerfile heredocs that agent images use).
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
buildah crun netavark aardvark-dns \
&& rm -rf /var/lib/apt/lists/*
# vfs + chroot: buildah works as root in the bare microVM (no
# fuse-overlayfs / overlay module / subuid maps). Matches image_builder.
ENV STORAGE_DRIVER=vfs \
BUILDAH_ISOLATION=chroot
# The orchestrator content, pulled from its single definition. The gateway
# image already has the flat daemon modules under /app; this adds the full
# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves.
COPY --from=bot-bottle-orchestrator:latest /app/bot_bottle /app/bot_bottle
+36
View File
@@ -0,0 +1,36 @@
# Orchestrator control-plane image (PRD 0070, #384).
#
# This is the **single definition of the orchestrator's content** — the
# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
# * the docker backend, which runs this image directly as the lean
# control-plane container; and
# * the firecracker infra image (Dockerfile.infra), which `COPY --from`s
# this image's `/app/bot_bottle` so the single infra VM runs the same
# control plane. Keeping it in one place means future orchestrator deps
# (e.g. iroh) are added here once, not duplicated per backend.
#
# It stays deliberately lean: the control plane is **stdlib-only** today, so
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
# builder, and lives only in Dockerfile.infra). Keeping the secret-dense
# control plane on a minimal dependency surface is the point (PRD 0070's
# "secret concentration").
#
# Shares the trixie `python:3.12-slim` base with the gateway image, so when
# the orchestrator grows real deps they can be `COPY --from`'d into the
# infra image cleanly (same base/python — installed packages copy safely).
FROM python:3.12-slim
WORKDIR /app
# The orchestrator content. Baked so the image is self-contained (runs from
# a built image, no runtime bind-mount); the docker backend may still
# bind-mount /app for dev live-reload, which simply overlays this copy.
# `.dockerignore` keeps .git/docs/*.md out of the context. (Future deps like
# iroh go here too — a shared requirements installed on this same base.)
COPY bot_bottle /app/bot_bottle
# Documentation only; lifecycle.py overrides the entrypoint to
# `python3 -m bot_bottle.orchestrator` with the runtime flags.
ENTRYPOINT ["python3", "-m", "bot_bottle.orchestrator"]
+9 -9
View File
@@ -16,7 +16,7 @@
- **Per-bottle egress allowlist** — TLS-bumped HTTP/HTTPS chokepoint with a per-manifest host allowlist; per-route path/method/header `matches` filtering; outbound DLP scanning for known tokens and secrets, inbound DLP scanning for prompt-injection attempts; DoH and arbitrary hosts blocked by default.
- **Per-route token-match policy** — each egress route picks what happens when the outbound DLP catches a token via `dlp.outbound_on_match`: `supervise` (default) holds the request and surfaces it in `./cli.py supervise` for approval (an approved value is remembered for the life of the proxy); `redact` scrubs the value and forwards; `block` is a hard `403`. Cuts false-positive friction without weakening default-deny.
- **Tokens the agent never sees** — host secrets live in a sidecar; the agent dials `http://sidecar:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
- **Tokens the agent never sees** — host secrets live in a gateway; the agent dials `http://gateway:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
- **Gitleaks-scanned push (git-gate)** — `bottle.git` remotes route through a per-bottle `git daemon` that gitleaks-scans incoming refs pre-receive and forwards clean refs upstream over SSH. The agent never holds the upstream credential.
- **Manifest-scoped skills + secrets** — each bottle declares its skills, env, git identity, remotes, and egress routes; unknown keys die at load.
- **Trust boundary at `$HOME`** — bottles (credentials, egress, remotes) live only under `~/.bot-bottle/bottles/`. Repos may ship agents but not bottles, so a cloned repo can't redirect an env var to an attacker host.
@@ -24,17 +24,17 @@
- **Parallel, isolated bottles** — each bottle runs in its own backend-owned isolation boundary; bottles don't share state or talk to each other.
- **Provider templates (Claude, Codex)** — `Dockerfile.claude` / `Dockerfile.codex`, or a bottle-supplied Dockerfile. Claude auth via long-lived OAuth token; Codex via opt-in host device-auth forwarding.
- **gVisor auto-detect** — on Linux hosts where `runsc` is registered with Docker, every bottle launches under it for a userspace syscall barrier; no manifest config required.
- **Apple Container backend (macOS default when available)** — runs the agent and sidecar bundle with Apple's `container` CLI, using a host-only agent network plus a separate sidecar egress network.
- **Firecracker backend (Linux default when available)** — runs the agent in a KVM Firecracker microVM reached over SSH on a point-to-point TAP, with the sidecar bundle in Docker. A dedicated, fail-closed `nftables` table isolates the guest, closing the raw DNS/IP exfiltration gap that exists in the legacy Docker backend. Requires KVM (`/dev/kvm`) and a one-time privileged network-pool setup.
- **Apple Container backend (macOS default when available)** — runs the agent and gateway with Apple's `container` CLI, using a host-only agent network plus a separate gateway egress network.
- **Firecracker backend (Linux default when available)** — runs the agent in a KVM Firecracker microVM reached over SSH on a point-to-point TAP, with the gateway in Docker. A dedicated, fail-closed `nftables` table isolates the guest, closing the raw DNS/IP exfiltration gap that exists in the legacy Docker backend. Requires KVM (`/dev/kvm`) and a one-time privileged network-pool setup.
- **Legacy Docker backend** — still available for examples, CI, and hosts without Apple Container or KVM via `BOT_BOTTLE_BACKEND=docker` or `--backend=docker`.
## Architecture
On the default macOS Apple Container backend, a bottle is an agent container on a host-only internal network plus a sidecar bundle attached to both that internal network and a NAT egress network. The agent gets HTTP(S)_PROXY and CA bundle env vars pointing at the sidecar's internal-network IP, so HTTP/HTTPS traffic flows through the sidecar instead of direct egress. `bottle.git` / git-gate is intentionally deferred on this backend until a safe Apple Container key-delivery path exists.
On the default macOS Apple Container backend, a bottle is an agent container on a host-only internal network plus a gateway attached to both that internal network and a NAT egress network. The agent gets HTTP(S)_PROXY and CA bundle env vars pointing at the gateway's internal-network IP, so HTTP/HTTPS traffic flows through the gateway instead of direct egress. `bottle.git` / git-gate is intentionally deferred on this backend until a safe Apple Container key-delivery path exists.
On the Firecracker backend, a bottle is an agent microVM plus a Docker sidecar bundle for egress, git-gate, and supervise. The VM reaches the sidecars over a per-bottle point-to-point TAP link; a dedicated fail-closed `nftables` table (`inet bot_bottle_fc`) confines the guest to that link, so nothing leaves the box except through the sidecars. The TAP pool and nft table are provisioned once (root); per-launch needs no privilege.
On the Firecracker backend, a bottle is an agent microVM plus a Docker gateway for egress, git-gate, and supervise. The VM reaches the gateway over a per-bottle point-to-point TAP link; a dedicated fail-closed `nftables` table (`inet bot_bottle_fc`) confines the guest to that link, so nothing leaves the box except through the gateway. The TAP pool and nft table are provisioned once (root); per-launch needs no privilege.
On the legacy Docker backend, the same logical bottle is two containers per agent: an `agent` container and a `sidecars` container. They share a per-agent Docker `--internal` network; the agent has no default route off-box.
On the legacy Docker backend, the same logical bottle is two containers per agent: an `agent` container and a `companion containers` container. They share a per-agent Docker `--internal` network; the agent has no default route off-box.
The Docker topology looks like this:
@@ -67,11 +67,11 @@ The Docker topology looks like this:
└─────────────────────────────────────────────────────────────────────┘
```
When the agent exits, `cli.py` tears down every sidecar and both networks; nothing about a bottle persists between runs.
When the agent exits, `cli.py` tears down every gateway and both networks; nothing about a bottle persists between runs.
## Quickstart
On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The Firecracker backend (Linux) requires Docker on the host for the sidecar bundle plus the `firecracker` binary and KVM. The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The Firecracker backend (Linux) requires Docker on the host for the gateway plus the `firecracker` binary and KVM. The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
@@ -81,7 +81,7 @@ On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
- **`/dev/kvm`** present and accessible. Load `kvm-intel` or `kvm-amd` (and enable virtualization in BIOS/firmware). The invoking user must be in the `kvm` group: `sudo usermod -aG kvm "$USER"` then re-login. bot-bottle preflights this and reports exactly what's missing.
- **`firecracker`** on `PATH`: grab a release from <https://github.com/firecracker-microvm/firecracker/releases>. Start flows print this pointer when the binary is missing.
- **Docker** for the sidecar bundle and image build.
- **Docker** for the gateway and image build.
- **A one-time privileged network setup** — the per-bottle TAP pool plus the fail-closed `nftables` isolation table. Run `./cli.py backend setup --backend=firecracker` for the host-appropriate config (a NixOS module, a `sudo` script elsewhere); `./cli.py backend status --backend=firecracker` reports what's present, including whether the pool range collides with an existing route. The pool defaults to `10.243.0.0/16` (an obscure RFC-1918 block that dodges docker/libvirt/LAN and, deliberately, Tailscale's `100.64.0.0/10` CGNAT range); override with `BOT_BOTTLE_FC_IP_BASE` if it clashes on your host.
```sh
+10 -2
View File
@@ -61,6 +61,13 @@ class AgentProviderRuntime:
prompt_mode: PromptMode
bypass_args: tuple[str, ...]
resume_args: tuple[str, ...]
# argv run inside a throwaway container of a freshly built agent
# image, right after `build_image()`, to catch a build that
# exited 0 but produced a broken CLI (e.g. an npm
# optionalDependencies fetch for a platform-native binary that
# silently no-ops on a transient failure). Empty tuple skips the
# check — not every provider has opted in yet.
smoke_test: tuple[str, ...] = ()
@dataclass(frozen=True)
@@ -204,9 +211,9 @@ class AgentProvider(ABC):
bottle: "Bottle",
supervise_url: str,
) -> None:
"""Register the per-bottle supervise sidecar as an MCP server
"""Register the per-bottle supervise daemon as an MCP server
in the provider's in-guest config. Called by the backend after
the supervise sidecar is reachable. No-op when
the supervise daemon is reachable. No-op when
`plan.supervise_plan is None`."""
@abstractmethod
@@ -259,6 +266,7 @@ class AgentProvider(ABC):
gate_scheme = getattr(plan, "git_gate_insteadof_scheme", "git")
content = git_gate_render_gitconfig(
manifest_bottle.git, gate_host, scheme=gate_scheme,
identity_token=getattr(plan, "identity_token", ""),
)
guest_gitconfig = f"{plan.guest_home}/.gitconfig"
with tempfile.NamedTemporaryFile(
+17 -5
View File
@@ -199,7 +199,7 @@ class ActiveAgent:
bottle is the container, the agent is what runs in it.)
Fields are deliberately backend-neutral. `services` is the set
of sidecar daemons currently up for this bottle (`egress`,
of gateway daemons currently up for this bottle (`egress`,
`git-gate`, `supervise`); the dashboard uses it to
gate edit verbs. `backend_name` is the matching key in
`_BACKENDS` (`docker` / `firecracker` / `macos-container`) — used by the active-
@@ -251,7 +251,7 @@ class Bottle(ABC):
`user` (default `node`, matching the agent image's USER
directive) and return the captured stdout/stderr/returncode.
The bottle's environment (including HTTPS_PROXY pointing at
the egress sidecar) is inherited by the child. Non-zero
the egress daemon) is inherited by the child. Non-zero
exit does not raise — callers inspect `returncode`
themselves.
@@ -454,7 +454,7 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
declarative provision-plan apply, supervise MCP registration)
live on the `AgentProvider` plugin. The backend only owns the
steps that are about backend infrastructure (CA, workspace,
git) and surfaces the supervise sidecar URL its launch step
git) and surfaces the supervise daemon URL its launch step
knows about via `supervise_mcp_url`.
PRD 0017: cred-proxy's agent-side dotfile rewrites (~/.npmrc,
@@ -502,7 +502,7 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
def supervise_mcp_url(self, plan: PlanT) -> str:
"""Return the agent-side URL of the per-bottle supervise
sidecar, or "" when this bottle has no sidecar. The provider
gateway, or "" when this bottle has no gateway. The provider
plugin's `provision_supervise_mcp` uses it to register the
MCP entry inside the guest.
@@ -511,6 +511,18 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
del plan
return ""
def ensure_orchestrator(self) -> str:
"""Bring up this backend's per-host orchestrator + shared gateway
(idempotent) and return the host-reachable control-plane URL.
This is the backend-agnostic bring-up entry point: `launch` calls
it as part of starting a bottle, and operator tools (`supervise`)
call it to start the control plane on demand when none is running
yet. Docker starts the orchestrator + gateway containers;
firecracker boots the infra VM. Backends with no orchestrator
(macos-container) die with a pointer — the default here."""
die(f"backend {self.name!r} has no orchestrator control plane")
@abstractmethod
def prepare_cleanup(self) -> CleanupT:
"""Enumerate orphaned resources from previous bottles. No side
@@ -524,7 +536,7 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
def enumerate_active(self) -> Sequence[ActiveAgent]:
"""Return every currently-running agent on this backend.
Empty when none. Backend-specific: docker queries `docker
compose ls`; firecracker cross-references its running sidecar
compose ls`; firecracker cross-references its running gateway
containers against per-bottle metadata."""
@classmethod
+10 -1
View File
@@ -105,12 +105,21 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def ensure_orchestrator(self) -> str:
from ...orchestrator.lifecycle import OrchestratorService
return OrchestratorService().ensure_running()
def supervise_mcp_url(self, plan: DockerBottlePlan) -> str:
"""Docker bottles reach the supervise sidecar via the
"""Docker bottles reach the supervise daemon via the
compose-network alias `supervise:9100`. No per-bottle URL
plumbing needed; the alias resolves inside the bridge."""
if plan.supervise_plan is None:
return ""
# Consolidated: the supervise daemon lives on the shared gateway, so
# the agent registers the gateway address (NO_PROXY bypasses egress),
# not the per-bottle `supervise` alias.
if plan.agent_supervise_url:
return plan.agent_supervise_url
return f"http://{SUPERVISE_HOSTNAME}:{SUPERVISE_PORT}/"
def prepare_cleanup(self) -> DockerBottleCleanupPlan:
+23
View File
@@ -28,11 +28,34 @@ class DockerBottlePlan(BottlePlan):
# accidental log of the plan dataclass.
forwarded_env: dict[str, str] = field(repr=False)
use_runsc: bool
# Consolidated mode (PRD 0070): the agent reaches git-gate over HTTP at the
# shared gateway's address (`http://<gateway_ip>:9420`), set at launch.
# Empty → single-tenant defaults (the `git-gate` alias over git://).
agent_git_gate_url: str = ""
# Likewise the supervise MCP endpoint at the gateway (`http://<gw>:9100/`);
# empty → the single-tenant `supervise` alias.
agent_supervise_url: str = ""
# Per-bottle identity token the agent presents on every attributed request
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
@property
def container_name(self) -> str:
return self.agent_provision.instance_name
@property
def git_gate_insteadof_host(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return self.agent_git_gate_url.removeprefix("http://").rstrip("/")
return super().git_gate_insteadof_host
@property
def git_gate_insteadof_scheme(self) -> str:
if self.agent_git_gate_url.startswith("http://"):
return "http"
return super().git_gate_insteadof_scheme
@property
def image(self) -> str:
return self.agent_provision.image
+6 -243
View File
@@ -1,20 +1,10 @@
"""Compose-spec rendering for a Docker bottle (PRD 0018, chunk 1).
"""Docker compose lifecycle helpers (PRD 0018).
`bottle_plan_to_compose(plan)` returns a Compose v2 spec dict
describing the per-bottle container topology — one project per
bottle instance, services for the agent + every applicable sidecar,
two networks, no named volumes.
Pure function. No I/O, no subprocess. Expects every launch-time
field (network names, CA host paths, etc.) on the plan's inner
plans to be populated; chunks 2+3 own that ordering.
Conditional services follow the plan content:
- agent + sidecars bundle: always.
- git-gate: iff plan.git_gate_plan.upstreams.
- egress: iff plan.egress_plan.routes.
- supervise: iff plan.supervise_plan is not None.
Serialize a compose spec to disk, drive `docker compose up/down`,
dump the merged log on teardown, and enumerate `bot-bottle-*`
projects. The spec itself is built by `consolidated_compose.py`
(the consolidated per-host gateway topology); this module owns the
I/O side that persists and runs it.
"""
from __future__ import annotations
@@ -25,233 +15,7 @@ import sys
from pathlib import Path
from typing import Any
from ...egress import (
EGRESS_HOSTNAME,
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_sidecar_env_entries,
)
from ...git_gate import GIT_GATE_HOSTNAME
from ...log import die, warn
from ...supervise import (
DB_PATH_IN_CONTAINER,
SUPERVISE_HOSTNAME,
SUPERVISE_PORT,
)
from ...util import expand_tilde
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from .bottle_plan import DockerBottlePlan
from .egress import (
EGRESS_CA_IN_CONTAINER,
EGRESS_PORT,
)
from .git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from . import network as network_mod
from .sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
sidecar_bundle_container_name,
)
# Repo root, used as the build context for the bundle Dockerfile.
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
def bottle_plan_to_compose(plan: DockerBottlePlan) -> dict[str, Any]:
"""Render a Compose v2 spec dict from a fully-resolved
DockerBottlePlan.
The plan must have its inner plans (`git_gate_plan`,
`egress_plan`, `supervise_plan`) populated with launch-time
fields — network names, CA host paths. The renderer doesn't
validate; callers feed it a fully-resolved plan or get an
incomplete compose spec back.
"""
project = f"bot-bottle-{plan.slug}"
services: dict[str, Any] = {
"sidecars": _sidecar_bundle_service(plan),
"agent": _agent_service(plan),
}
return {
"name": project,
"services": services,
"networks": _networks(plan),
}
def _networks(plan: DockerBottlePlan) -> dict[str, Any]:
"""Compose-managed networks with explicit `name:` matching the
existing slug-suffixed convention. Compose creates them on `up`
and destroys them on `down`. The internal one is `--internal`
(no default gateway); the egress one is a normal user-defined
bridge."""
return {
"internal": {
"name": network_mod.network_name_for_slug(plan.slug),
"internal": True,
},
"egress": {
"name": network_mod.network_egress_name_for_slug(plan.slug),
},
}
def _bind(host: str | Path, target: str, *, read_only: bool = True) -> dict[str, Any]:
"""One bind-mount entry in the long-form `volumes:` shape.
Long form is preferred over `host:target:ro` strings because
it's easier to inspect in tests and survives whitespace in
host paths."""
return {
"type": "bind",
"source": str(host),
"target": target,
"read_only": read_only,
}
def _sidecar_bundle_service(plan: DockerBottlePlan) -> dict[str, Any]:
"""The `sidecars` service: one container per bottle, bundle
image, all daemons under a Python init supervisor.
Daemon subset narrows via `BOT_BOTTLE_SIDECAR_DAEMONS` env.
egress is always present; git-gate / supervise are conditional.
"""
daemons: list[str] = ["egress"]
if plan.git_gate_plan.upstreams:
daemons.append("git-gate")
if plan.supervise_plan is not None:
daemons.append("supervise")
env: list[str] = [f"BOT_BOTTLE_SIDECAR_DAEMONS={','.join(daemons)}"]
volumes: list[dict[str, Any]] = []
# --- egress -------------------------------------------------------
ep = plan.egress_plan
volumes.append(_bind(ep.mitmproxy_ca_host_path, EGRESS_CA_IN_CONTAINER))
if ep.routes:
volumes.append(_bind(ep.routes_path.parent, str(Path(EGRESS_ROUTES_IN_CONTAINER).parent)))
env.extend(egress_sidecar_env_entries(ep))
# --- git-gate -----------------------------------------------------
gp = plan.git_gate_plan
if gp.upstreams:
volumes += [
_bind(gp.entrypoint_script, GIT_GATE_ENTRYPOINT_IN_CONTAINER),
_bind(gp.hook_script, GIT_GATE_HOOK_IN_CONTAINER),
_bind(gp.access_hook_script, GIT_GATE_ACCESS_HOOK_IN_CONTAINER),
]
for u in gp.upstreams:
keypath = expand_tilde(u.identity_file)
volumes.append(_bind(
keypath,
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-key",
))
if u.known_hosts_file:
volumes.append(_bind(
u.known_hosts_file,
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{u.name}-known_hosts",
))
# --- supervise ----------------------------------------------------
sp = plan.supervise_plan
if sp is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
volumes.append({
"type": "bind",
"source": str(sp.db_path),
"target": DB_PATH_IN_CONTAINER,
"read_only": False,
})
internal_aliases = [EGRESS_HOSTNAME]
if gp.upstreams:
internal_aliases.append(GIT_GATE_HOSTNAME)
if sp is not None:
internal_aliases.append(SUPERVISE_HOSTNAME)
service: dict[str, Any] = {
"image": SIDECAR_BUNDLE_IMAGE,
"build": {
"context": _REPO_DIR,
"dockerfile": SIDECAR_BUNDLE_DOCKERFILE,
},
"container_name": sidecar_bundle_container_name(plan.slug),
"networks": {
"internal": {"aliases": internal_aliases},
"egress": None,
},
"environment": env,
"volumes": volumes,
}
return service
def _agent_service(plan: DockerBottlePlan) -> dict[str, Any]:
"""Agent container. Runs `sleep infinity`; claude is `docker
exec -it`'d into it later. HTTP_PROXY/HTTPS_PROXY point at the
egress sidecar."""
proxy_url = _agent_proxy_url(plan)
no_proxy = _agent_no_proxy(plan)
env: list[str] = [
f"HTTPS_PROXY={proxy_url}",
f"HTTP_PROXY={proxy_url}",
f"https_proxy={proxy_url}",
f"http_proxy={proxy_url}",
f"NO_PROXY={no_proxy}",
f"no_proxy={no_proxy}",
f"NODE_EXTRA_CA_CERTS={AGENT_CA_PATH}",
f"SSL_CERT_FILE={AGENT_CA_BUNDLE}",
f"REQUESTS_CA_BUNDLE={AGENT_CA_BUNDLE}",
]
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars (OAuth token, manifest host-interpolations):
# bare name → inherits from compose-up process env, value
# never lands on argv or in the compose file.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
service: dict[str, Any] = {
"image": plan.image,
"container_name": plan.container_name,
"command": ["sleep", "infinity"],
"networks": {"internal": None},
"environment": env,
}
if plan.use_runsc:
service["runtime"] = "runsc"
# The init supervisor inside the bundle owns intra-bundle
# daemon ordering, so the agent only waits for the bundle
# container itself.
service["depends_on"] = ["sidecars"]
return service
def _agent_proxy_url(plan: DockerBottlePlan) -> str:
"""Agent's HTTP_PROXY — always points at egress."""
return f"http://{EGRESS_HOSTNAME}:{EGRESS_PORT}"
def _agent_no_proxy(plan: DockerBottlePlan) -> str:
"""NO_PROXY for the agent: loopback always; supervise hostname
when the supervise sidecar is up (MCP long-poll must bypass
the egress proxy)."""
hosts = ["localhost", "127.0.0.1"]
if plan.supervise_plan is not None:
hosts.append(SUPERVISE_HOSTNAME)
return ",".join(hosts)
# --- Lifecycle helpers (PRD 0018 chunk 3) ----------------------------------
@@ -442,7 +206,6 @@ __all__ = [
"COMPOSE_FILE_NAME",
"COMPOSE_LOG_NAME",
"COMPOSE_PROJECT_PREFIX",
"bottle_plan_to_compose",
"compose_down",
"compose_dump_logs",
"compose_file_path",
@@ -0,0 +1,84 @@
"""Agent-only compose for the consolidated docker backend (PRD 0070).
The per-bottle model rendered a compose project with the agent *and* a
gateway on two per-bottle networks. In the consolidated model the
per-bottle companion containers are gone — one shared gateway serves every bottle — so this renders
just the agent, attached to the **external shared gateway network** with the
pinned source IP the orchestrator allocated, and pointed at the gateway's
address for egress (and, around the proxy, for git-http / supervise).
Pure: it takes the launch-time `LaunchContext` values (gateway address,
source IP, network) and the prepared plan, and returns a compose dict — no
docker, so it's testable in isolation.
"""
from __future__ import annotations
from typing import Any
from ...egress import egress_agent_env_entries
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from .bottle_plan import DockerBottlePlan
from .egress import EGRESS_PORT
def consolidated_agent_compose(
plan: DockerBottlePlan,
*,
gateway_ip: str,
source_ip: str,
network: str,
) -> dict[str, Any]:
"""A compose spec with only the agent service, on the external gateway
network at `source_ip`, proxying egress through `gateway_ip`."""
# Deliver the identity token as egress proxy credentials — the gateway
# reads Proxy-Authorization, validates the (source_ip, token) pair, and
# strips it before upstream. git-http/supervise get it via their own
# headers (git config extraHeader / MCP header).
token = getattr(plan, "identity_token", "")
cred = f"bottle:{token}@" if token else ""
proxy_url = f"http://{cred}{gateway_ip}:{EGRESS_PORT}"
# git-http + supervise live on the gateway too and must NOT go through the
# egress proxy — the agent reaches them directly by the gateway address.
no_proxy = f"localhost,127.0.0.1,{gateway_ip}"
env: list[str] = [
f"HTTPS_PROXY={proxy_url}",
f"HTTP_PROXY={proxy_url}",
f"https_proxy={proxy_url}",
f"http_proxy={proxy_url}",
f"NO_PROXY={no_proxy}",
f"no_proxy={no_proxy}",
f"NODE_EXTRA_CA_CERTS={AGENT_CA_PATH}",
f"SSL_CERT_FILE={AGENT_CA_BUNDLE}",
f"REQUESTS_CA_BUNDLE={AGENT_CA_BUNDLE}",
]
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars: bare name → inherits from the compose-up process env so
# the secret value never lands on argv or in the compose file.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
service: dict[str, Any] = {
"image": plan.image,
"container_name": plan.container_name,
"command": ["sleep", "infinity"],
# Pinned address on the shared gateway network — the orchestrator
# registered this IP, and the gateway attributes the bottle by it.
"networks": {network: {"ipv4_address": source_ip}},
"environment": env,
}
if plan.use_runsc:
service["runtime"] = "runsc"
return {
"name": f"bot-bottle-{plan.slug}",
"services": {"agent": service},
# The gateway network is created + owned by the orchestrator; compose
# attaches to it (external) and must not create or destroy it.
"networks": {network: {"external": True}},
}
__all__ = ["consolidated_agent_compose"]
@@ -0,0 +1,158 @@
"""Consolidated bottle launch sequence for the docker backend (PRD 0070).
Composes the orchestrator primitives into the register/teardown sequence that
replaces the per-bottle gateway:
1. ensure the orchestrator control plane + shared gateway are up;
2. allocate the bottle a pinned source IP on the gateway network (the
attribution key), skipping the gateway's own address + live bottles;
3. register it (egress policy blob + slug metadata) → bottle id + identity
token;
4. provision its git-gate repos/creds into the running gateway.
It returns a `LaunchContext` with everything the agent container needs to
attach — network, pinned IP, the gateway's address (its proxy target), the
orchestrator URL, and the identity token. The agent `docker run` itself is
the backend's job (it owns provider provisioning); this owns the
orchestrator-facing wiring so that sequence stays testable in isolation.
"""
from __future__ import annotations
from dataclasses import dataclass
from ...docker_cmd import run_docker
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.gateway import GATEWAY_NAME, GATEWAY_NETWORK
from ...orchestrator.lifecycle import OrchestratorService
from ...orchestrator.registration import registration_inputs
from .gateway_net import next_free_ip
from .gateway_provision import (
DockerGatewayTransport,
deprovision_git_gate,
provision_git_gate,
)
class ConsolidatedLaunchError(RuntimeError):
"""The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
class LaunchContext:
"""What the agent container needs to join the shared gateway."""
bottle_id: str
identity_token: str
source_ip: str # the agent's pinned address (attribution key)
network: str # the shared gateway network to attach to
gateway_ip: str # the gateway's address — the agent's proxy target
orchestrator_url: str
def _network_cidr(network: str) -> str:
"""The gateway network's IPv4 subnet, or raise."""
proc = run_docker([
"docker", "network", "inspect",
"--format", "{{range .IPAM.Config}}{{.Subnet}}{{end}}", network,
])
cidr = proc.stdout.strip()
if proc.returncode != 0 or not cidr:
raise ConsolidatedLaunchError(
f"gateway network {network} has no subnet: {proc.stderr.strip()}"
)
return cidr
def _container_ip(name: str, network: str) -> str:
"""A container's IPv4 address on `network`, or raise."""
proc = run_docker([
"docker", "inspect", "--format",
f'{{{{(index .NetworkSettings.Networks "{network}").IPAddress}}}}', name,
])
ip = proc.stdout.strip()
if proc.returncode != 0 or not ip:
raise ConsolidatedLaunchError(
f"gateway {name} has no address on {network}: {proc.stderr.strip()}"
)
return ip
def _network_container_ips(network: str) -> list[str]:
"""Every address currently assigned on the gateway network — the ground
truth for "in use": the gateway + orchestrator infrastructure containers
and every live agent. Read from the network so a new bottle can't collide
with anything actually attached (a registry-only view would miss the
orchestrator/gateway containers)."""
proc = run_docker([
"docker", "network", "inspect", "--format",
"{{range .Containers}}{{.IPv4Address}} {{end}}", network,
])
ips: list[str] = []
for entry in proc.stdout.split():
# entries look like "172.20.0.2/16" — keep the address.
ips.append(entry.split("/", 1)[0])
return ips
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
image_ref: str = "",
tokens: dict[str, str] | None = None,
service: OrchestratorService | None = None,
gateway_name: str = GATEWAY_NAME,
network: str = GATEWAY_NETWORK,
) -> LaunchContext:
"""Ensure the orchestrator + gateway are up, allocate + register the
bottle, and provision its git-gate state. Returns the agent's attach
context. Raises `ConsolidatedLaunchError` (or the primitives' own errors)
if any step fails — the caller tears down on failure."""
service = service or OrchestratorService()
url = service.ensure_running()
client = OrchestratorClient(url)
cidr = _network_cidr(network)
gateway_ip = _container_ip(gateway_name, network)
source_ip = next_free_ip(cidr, _network_container_ips(network))
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
raise
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=source_ip,
network=network,
gateway_ip=gateway_ip,
orchestrator_url=url,
)
def teardown_consolidated(
bottle_id: str, *, orchestrator_url: str, gateway_name: str = GATEWAY_NAME,
) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id)
__all__ = [
"LaunchContext",
"launch_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
]
+1 -1
View File
@@ -4,7 +4,7 @@ prepare-time routes-yaml rendering itself lives on the
platform-neutral `Egress` ABC — backends instantiate it directly.
The per-container `.start()` / `.stop()` lifecycle was removed in
PRD 0024 chunk 3; the sidecar bundle (PRD 0024) runs egress
PRD 0024 chunk 3; the gateway (PRD 0024) runs egress
under its python init supervisor."""
from __future__ import annotations
+13 -44
View File
@@ -1,60 +1,29 @@
"""Host-side helper for egress sidecar inspection and live updates.
"""Host-side egress route-apply for the docker backend.
The approve path uses this module to validate a proposed routes file,
write it to the bottle's live egress state dir, and signal the sidecar
bundle so the mitmproxy addon reloads it.
The per-bottle companion container this used to signal (`docker kill
--signal HUP <container>`) was removed in the companion-container removal (#385).
In the consolidated model the shared gateway resolves egress policy
per-request against the orchestrator rather than reloading a per-bottle
routes file, so the live per-bottle reload is not supported here and
fails closed until the gateway-side apply lands.
"""
from __future__ import annotations
import os
import subprocess
from ...egress import EGRESS_ROUTES_IN_CONTAINER
from ...log import warn
from ..egress_apply import EgressApplicator, EgressApplyError
from .sidecar_bundle import sidecar_bundle_container_name
def fetch_current_routes(slug: str) -> str:
container = sidecar_bundle_container_name(slug)
r = subprocess.run(
["docker", "exec", container, "cat", EGRESS_ROUTES_IN_CONTAINER],
capture_output=True, text=True, check=False,
)
if r.returncode != 0:
raise EgressApplyError(
f"could not read routes.yaml from {container}: "
f"{(r.stderr or '').strip() or 'container not running?'}"
)
return r.stdout
class DockerEgressApplicator(EgressApplicator):
def _signal_bundle_reload(self, slug: str) -> None:
container = sidecar_bundle_container_name(slug)
result = subprocess.run(
["docker", "kill", "--signal", "HUP", container],
capture_output=True, text=True, check=False, env=os.environ,
del slug
raise EgressApplyError(
"live egress route-apply was removed with the per-bottle "
"companion container (#385); route changes will flow through "
"the consolidated gateway in a follow-up."
)
if result.returncode != 0:
last_error = (result.stderr or "").strip() or (result.stdout or "").strip()
warn(
f"egress: routes updated on disk for {slug}, but bundle reload failed: "
f"{last_error or 'docker kill failed'}"
)
raise EgressApplyError(
f"could not reload egress bundle {container}: "
f"{last_error or 'docker kill failed'}"
)
applicator = DockerEgressApplicator()
__all__ = [
"DockerEgressApplicator",
"EgressApplyError",
"applicator",
"fetch_current_routes",
]
__all__ = ["DockerEgressApplicator", "EgressApplyError", "applicator"]
+66 -34
View File
@@ -15,14 +15,15 @@ bottle's push credentials out of another's repos on the shared gateway.
from __future__ import annotations
import re
from typing import Protocol
from ...docker_cmd import run_docker
from ...git_gate import GitGatePlan, git_gate_render_provision
# bottle ids index the gateway's per-bottle repo + creds dirs; they land in
# `docker cp`/`rm` path arguments, so validate before any path is built (a
# traversal id like "../etc" must never reach the container). Registry ids are
# token_hex — this is defense in depth at the docker boundary.
# exec/cp path arguments, so validate before any path is built (a traversal
# id like "../etc" must never reach the gateway). Registry ids are token_hex —
# this is defense in depth at the transport boundary.
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
@@ -30,6 +31,42 @@ class GatewayProvisionError(RuntimeError):
"""A git-gate provisioning step against the running gateway failed."""
class GatewayTransport(Protocol):
"""How the launcher stages files + runs commands in the running gateway.
Backend-neutral so the same provisioning logic serves the docker gateway
(exec/cp over the docker socket) and the firecracker gateway VM (over
SSH)."""
def exec(self, argv: list[str]) -> None:
"""Run `argv` in the gateway, raising `GatewayProvisionError` on
failure."""
def cp_into(self, src: str, dest: str) -> None:
"""Copy host file `src` to `dest` in the gateway, raising on
failure."""
class DockerGatewayTransport:
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
def __init__(self, gateway: str) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
proc = run_docker(["docker", "exec", self.gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def cp_into(self, src: str, dest: str) -> None:
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
def _require_safe(bottle_id: str) -> None:
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise GatewayProvisionError(f"unsafe bottle id {bottle_id!r}")
@@ -39,27 +76,11 @@ def _creds_dir(bottle_id: str) -> str:
return f"/git-gate/creds/{bottle_id}"
def _exec(gateway: str, argv: list[str]) -> None:
"""`docker exec` a command in the gateway, raising on non-zero exit."""
proc = run_docker(["docker", "exec", gateway, *argv])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
)
def _cp_into(gateway: str, src: str, dest: str) -> None:
"""`docker cp` a host file into the gateway, raising on non-zero exit."""
proc = run_docker(["docker", "cp", src, f"{gateway}:{dest}"])
if proc.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
)
def provision_git_gate(gateway: str, bottle_id: str, plan: GitGatePlan) -> None:
"""Place `bottle_id`'s git-gate credentials into the running `gateway`
container and init its bare repos under `/git/<bottle_id>/`.
def provision_git_gate(
transport: GatewayTransport, bottle_id: str, plan: GitGatePlan,
) -> None:
"""Place `bottle_id`'s git-gate credentials into the running gateway and
init its bare repos under `/git/<bottle_id>/`.
Copies each upstream's identity key (and known_hosts, when present) into
`/git-gate/creds/<bottle_id>/`, then runs the namespaced provisioning
@@ -67,28 +88,39 @@ def provision_git_gate(gateway: str, bottle_id: str, plan: GitGatePlan) -> None:
_require_safe(bottle_id)
if not plan.upstreams:
return
# The pre-receive + access hooks are bottle-agnostic and shared by every
# bottle's repos; install them into the gateway (idempotent — same content
# each time). The per-bottle model cp'd these into each bundle at start.
transport.exec(["mkdir", "-p", "/etc/git-gate"])
transport.cp_into(str(plan.hook_script), "/etc/git-gate/pre-receive")
transport.cp_into(str(plan.access_hook_script), "/etc/git-gate/access-hook")
creds = _creds_dir(bottle_id)
_exec(gateway, ["mkdir", "-p", creds])
transport.exec(["mkdir", "-p", creds])
for u in plan.upstreams:
if u.identity_file:
_cp_into(gateway, u.identity_file, f"{creds}/{u.name}-key")
transport.cp_into(u.identity_file, f"{creds}/{u.name}-key")
known_hosts = str(u.known_hosts_file)
if known_hosts and known_hosts != ".":
_cp_into(gateway, known_hosts, f"{creds}/{u.name}-known_hosts")
transport.cp_into(known_hosts, f"{creds}/{u.name}-known_hosts")
# Init the bare repos + per-repo credential config for this namespace.
script = git_gate_render_provision(bottle_id, plan.upstreams)
_exec(gateway, ["sh", "-c", script])
transport.exec(["sh", "-c", script])
def deprovision_git_gate(gateway: str, bottle_id: str) -> None:
def deprovision_git_gate(transport: GatewayTransport, bottle_id: str) -> None:
"""Remove a bottle's repos + creds from the gateway on teardown. Idempotent
— an already-absent namespace is a clean no-op (best effort; a stray dir
can't leak, since attribution is by source IP and the bottle is gone)."""
_require_safe(bottle_id)
run_docker([
"docker", "exec", gateway, "rm", "-rf",
f"/git/{bottle_id}", _creds_dir(bottle_id),
])
try:
transport.exec([
"rm", "-rf", f"/git/{bottle_id}", _creds_dir(bottle_id),
])
except GatewayProvisionError:
pass # best-effort teardown; absent namespace is success
__all__ = ["provision_git_gate", "deprovision_git_gate", "GatewayProvisionError"]
__all__ = [
"provision_git_gate", "deprovision_git_gate",
"GatewayProvisionError", "GatewayTransport", "DockerGatewayTransport",
]
+1 -1
View File
@@ -2,7 +2,7 @@
bind-mounts target + the listening port. The prepare-time entrypoint
/ hook render lives on the platform-neutral `GitGate` ABC — backends
instantiate it directly. The git-gate daemon's container lifecycle
is owned by the sidecar bundle (PRD 0024)."""
is owned by the gateway (PRD 0024)."""
from __future__ import annotations
+62 -62
View File
@@ -5,7 +5,7 @@ PRD 0018 chunk 3: each instance is one `docker compose` project.
The flow is:
1. Build the agent image from the provider Dockerfile (compose
builds the sidecar images via the `build:` directive on first up).
builds the gateway image on first up).
2. Mint the per-bottle egress CA (chunk 2 writes it under
state/<slug>/egress/).
3. Populate the inner plans with launch-time fields so the
@@ -36,13 +36,13 @@ from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...egress import egress_resolve_token_values
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import info, warn
from . import network as network_mod
from . import util as docker_mod
from .bottle import DockerBottle
from .bottle_plan import DockerBottlePlan
@@ -53,7 +53,6 @@ from ...bottle_state import (
read_committed_image,
)
from .compose import (
bottle_plan_to_compose,
compose_down,
compose_dump_logs,
compose_file_path,
@@ -62,7 +61,9 @@ from .compose import (
compose_up,
write_compose_file,
)
from .egress import egress_tls_init
from .consolidated_compose import consolidated_agent_compose
from .consolidated_launch import launch_consolidated, teardown_consolidated
from ...orchestrator.gateway import DockerGateway
# Where the repo root lives, for `docker build` context. Computed once.
@@ -97,8 +98,7 @@ def launch(
try:
# Step 1: agent image. Use a committed snapshot when one exists
# and is present in the local daemon; otherwise build from the
# Dockerfile. Sidecar images get built lazily by `docker compose
# up` via the renderer's `build:` directives.
# Dockerfile. (The gateway image is built by the orchestrator.)
committed = read_committed_image(plan.slug)
if committed and docker_mod.image_exists(committed):
info(f"using committed image {committed!r}")
@@ -111,83 +111,87 @@ def launch(
plan.image, _REPO_DIR,
dockerfile=plan.dockerfile_path,
)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
internal_network = network_mod.network_name_for_slug(plan.slug)
egress_network = network_mod.network_egress_name_for_slug(plan.slug)
egress_ca_host, egress_ca_cert_only = egress_tls_init(
egress_state_dir(plan.slug),
)
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any, before
# provisioning the bottle's repos into the shared gateway.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle,
git_gate_plan,
git_gate_state_dir(plan.slug),
)
git_gate_plan = dataclasses.replace(
git_gate_plan,
internal_network=internal_network,
egress_network=egress_network,
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
# Step 3: register on the orchestrator + provision this bottle's
# git-gate state into the shared gateway; get the agent's attach
# context (pinned source IP, gateway address, shared network). The
# per-bottle egress auth tokens are resolved from the host env now and
# handed to the orchestrator (in memory) for the gateway to inject —
# the agent never sees them.
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan, image_ref=plan.image, tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id, orchestrator_url=ctx.orchestrator_url,
)
# Step 4: install the SHARED gateway CA into the agent (replaces the
# per-bottle CA) — read it out of the running gateway.
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(DockerGateway(network=ctx.network).ca_cert_pem())
egress_plan = dataclasses.replace(
plan.egress_plan,
internal_network=internal_network,
egress_network=egress_network,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
mitmproxy_ca_host_path=ca_file,
mitmproxy_ca_cert_only_host_path=ca_file,
)
# Point the agent's git-gate insteadOf rewrites at the shared gateway's
# HTTP git endpoint (9420) instead of the dead per-bottle `git-gate`
# alias. git-http + supervise on the gateway bypass the egress proxy
# (NO_PROXY includes the gateway address).
git_gate_url = (
f"http://{ctx.gateway_ip}:9420" if git_gate_plan.upstreams else ""
)
supervise_url = (
f"http://{ctx.gateway_ip}:9100/" if plan.supervise_plan is not None else ""
)
supervise_plan = plan.supervise_plan
if supervise_plan is not None:
supervise_plan = dataclasses.replace(
supervise_plan,
internal_network=internal_network,
)
plan = dataclasses.replace(
plan,
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
supervise_plan=supervise_plan,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
identity_token=ctx.identity_token,
)
# Step 6: render + write the compose file. metadata.json
# was written at prepare time and already carries
# compose_project; nothing to update here.
# Step 5: render + up the agent-only compose, pinned on the shared
# gateway network and proxied through the gateway's address.
state_dir = bottle_state_dir(plan.slug)
spec = bottle_plan_to_compose(plan)
spec = consolidated_agent_compose(
plan, gateway_ip=ctx.gateway_ip, source_ip=ctx.source_ip, network=ctx.network,
)
compose_file = write_compose_file(spec, compose_file_path(state_dir))
project = compose_project_name(plan.slug)
# Step 7: compose up. Token values + the OAuth placeholder
# flow through subprocess env; the compose file holds only
# bare names for the secret-carrying entries.
effective_env = {**dict(os.environ), **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
compose_env: dict[str, str] = {
**os.environ,
**plan.forwarded_env,
**token_values,
}
# Forwarded vars (OAuth token, host interpolations) flow through the
# subprocess env as bare names so values never land in the file.
compose_env: dict[str, str] = {**os.environ, **plan.forwarded_env}
info(
f"docker compose up -d (project {project}, "
f"{len(spec['services'])} services)"
f"docker compose up -d (project {project}, agent on shared "
f"gateway {ctx.gateway_ip}, ip {ctx.source_ip})"
)
compose_up(project, compose_file, env=compose_env)
# Register teardown in reverse order: log dump first, then
# `compose down`. Networks come down last via callbacks
# registered in step 2.
stack.callback(compose_down, project, compose_file)
stack.callback(
compose_dump_logs, project, compose_file, compose_log_path(state_dir),
)
# Step 8: provision. Create the bottle first so provisioners
# can use bottle.exec / bottle.cp_in; set the prompt path
# returned by provision_prompt after the fact.
# Step 6: provision (CA install now uses the gateway CA) + yield.
bottle = DockerBottle(
plan.container_name,
teardown,
@@ -200,10 +204,6 @@ def launch(
agent_workdir=plan.workspace_plan.workdir,
)
bottle.prompt_path = provision(plan, bottle)
# Step 9: yield. exec_agent continues to use `docker exec -it`
# — the agent runs `sleep infinity` per the renderer's
# service spec.
yield bottle
finally:
teardown()
+1 -1
View File
@@ -76,7 +76,7 @@ def network_create_internal(slug: str) -> str:
def network_create_egress(slug: str) -> str:
"""Create a per-agent user-defined bridge (NOT the legacy `bridge`)
so the egress sidecar has working DNS for upstream hostnames."""
so the egress daemon has working DNS for upstream hostnames."""
return _network_create_with_prefix(network_egress_name_for_slug(slug), internal=False)
+3 -3
View File
@@ -1,7 +1,7 @@
"""Host setup + status for the Docker backend.
Unlike Firecracker, the Docker backend needs no privileged one-time
host provisioning (no TAP pool / nft table) — networks and the sidecar
host provisioning (no TAP pool / nft table) — networks and the gateway
bundle are created per-launch. So `setup()` is mostly an install/daemon
pointer, and `status()` reports whether docker is usable.
@@ -45,7 +45,7 @@ def setup() -> int:
return 1
sys.stderr.write(
"Docker backend: no privileged host setup required — networks and "
"the sidecar bundle are created per-launch.\n"
"the gateway are created per-launch.\n"
)
if not _daemon_reachable():
sys.stderr.write(
@@ -64,7 +64,7 @@ def setup() -> int:
def teardown() -> int:
sys.stderr.write(
"Docker backend: nothing to undo — it provisions no privileged host "
"state (networks and the sidecar bundle are per-launch and are "
"state (networks and the gateway are per-launch and are "
"removed by `./cli.py cleanup`). Docker itself is left installed.\n"
)
return 0
@@ -1,30 +0,0 @@
"""Sidecar bundle constants + helpers for the Docker backend
(PRD 0024).
The bundle image (built by Dockerfile.sidecars, PRD 0024 chunk 1)
runs egress + git-gate + supervise as one container per bottle
under a small Python init supervisor. As of chunk 5 the bundle
is the only shape — the legacy four-sidecar topology and its
`BOT_BOTTLE_SIDECAR_BUNDLE` feature flag are gone."""
from __future__ import annotations
import os
# Bundle image. Defaults to a built-locally tag (built from the
# repo's Dockerfile.sidecars via compose `build:`). Operators
# pinning to a published digest can override via env.
SIDECAR_BUNDLE_IMAGE = os.environ.get(
"BOT_BOTTLE_SIDECAR_IMAGE",
"bot-bottle-sidecars:latest",
)
SIDECAR_BUNDLE_DOCKERFILE = "Dockerfile.sidecars"
def sidecar_bundle_container_name(slug: str) -> str:
"""`bot-bottle-sidecars-<slug>`. Same prefix scheme as the
per-sidecar containers it replaces, so the dashboard's
discovery-by-prefix logic keeps working."""
return f"bot-bottle-sidecars-{slug}"
+51 -1
View File
@@ -4,11 +4,13 @@ existence, and building images."""
from __future__ import annotations
import os
import re
import shutil
import subprocess
from typing import Iterable, Iterator
from ...docker_cmd import run_docker
from ...log import die, info
# from ...workspace import WorkspacePlan
@@ -88,6 +90,29 @@ def docker_exec_root(container: str, argv: list[str]) -> None:
)
def docker_exec(container: str, argv: list[str], *, user: str = "") -> None:
"""Run `docker exec` in the named container, dying with the
command's own stderr on failure. Pass `user=\"0\"` to run as root."""
cmd = ["docker", "exec"]
if user:
cmd += ["-u", user]
cmd += [container, *argv]
result = run_docker(cmd)
if result.returncode != 0:
die(
f"docker exec in {container} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def docker_cp(src: str, dest: str) -> None:
"""Run `docker cp`, dying with the command's own stderr on failure."""
result = run_docker(["docker", "cp", src, dest])
if result.returncode != 0:
die(f"docker cp {src} -> {dest} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
_SLUG_RE = re.compile(r"[^a-z0-9]+")
@@ -108,15 +133,40 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
`dockerfile` is an optional path (relative to `context`, or
absolute) for callers that need to build from a non-default
Dockerfile in the same context — e.g. `Dockerfile.git-gate`."""
Dockerfile in the same context — e.g. `Dockerfile.git-gate`.
Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
`--no-cache`. The npm/curl installers some provider Dockerfiles
shell out to can silently no-op on a transient network failure —
e.g. an `optionalDependencies` fetch for a platform-native binary —
and Docker will then cache that broken layer indefinitely."""
info(f"building image {ref} from {context} (layer cache keeps repeat builds fast)")
args = ["docker", "build", "-t", ref]
if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
args.append("--no-cache")
if dockerfile:
args.extend(["-f", dockerfile])
args.append(context)
subprocess.run(args, check=True)
def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
"""Run `argv` inside a throwaway container of a freshly built agent
image and die loudly if it fails, instead of shipping an image
whose CLI only breaks at first real use. No-op when the provider
hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
if not argv:
return
result = run_docker(["docker", "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]])
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(
f"agent image {image!r} failed its post-build smoke test "
f"({' '.join(argv)}): {detail}\n"
f"Try rebuilding from scratch: bot-bottle start --no-cache"
)
# def build_image_with_cwd(
# derived: str,
# base: str,
@@ -110,3 +110,7 @@ class FirecrackerBottleBackend(
def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
return plan.agent_supervise_url
def ensure_orchestrator(self) -> str:
from . import infra_vm
return infra_vm.ensure_running().control_plane_url
+13 -6
View File
@@ -7,7 +7,7 @@ session. `ssh -t` forwards the host terminal's SIGWINCH to the remote
PTY natively, so no separate resize bridge is needed.
Commands run as the image's `node` user via `runuser`, with HOME/USER/
PATH and the bottle env (HTTPS_PROXY at the sidecar, CA paths, …) set
PATH and the bottle env (HTTPS_PROXY at the gateway, CA paths, …) set
per-invocation through `env` (the VM itself just runs the init; it has
no baked-in process env like a `docker run` container would).
"""
@@ -105,10 +105,9 @@ class FirecrackerBottle(Bottle):
# root-owned and unreadable by node, which breaks Node's
# process.cwd(), the shell-snapshot machinery, and `/doctor`.
# Use `env --chdir` rather than a `sh -c 'cd … && exec "$@"'`
# wrapper: ssh space-joins everything after the host into one
# string for the guest shell, so a quoted script + $@ would be
# re-split and mangled (exec'ing the $0 placeholder). All-simple
# words survive that join.
# wrapper: it keeps the guest command a flat argv that `agent_argv`
# can quote token-by-token for the ssh→guest-shell round trip,
# avoiding a fragile nested-quoting `"$@"` script.
workdir = self.agent_workdir or _HOME_FOR["node"]
remote = ["runuser", "-u", "node", "--",
"env", f"--chdir={workdir}",
@@ -117,7 +116,15 @@ class FirecrackerBottle(Bottle):
return remote
def agent_argv(self, argv: list[str], *, tty: bool = True) -> list[str]:
return [*self._ssh(tty=tty), "--", *self._agent_remote_argv(argv)]
# ssh space-joins everything after the host into one line the guest
# shell re-parses, so pre-quote each remote token for that shell.
# Simple words are unchanged (existing behaviour); an arg containing
# spaces — e.g. codex's `read_prompt_file` positional "Read and follow
# the instructions in <path>." — is quoted so it survives as ONE
# argument instead of being re-split (which made codex parse "and" as
# a subcommand).
remote = self._agent_remote_argv(argv)
return [*self._ssh(tty=tty), "--", *(shlex.quote(t) for t in remote)]
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
agent_argv = self.agent_argv(argv, tty=tty)
@@ -10,10 +10,9 @@ from .. import BottleCleanupPlan
@dataclass(frozen=True)
class FirecrackerBottleCleanupPlan(BottleCleanupPlan):
# PIDs of orphaned firecracker VMM processes and the sidecar
# containers left behind by previous bottles.
# PIDs of orphaned firecracker VMM processes and the per-bottle run
# dirs left behind by previous bottles.
vm_pids: tuple[int, ...] = ()
containers: tuple[str, ...] = ()
run_dirs: tuple[str, ...] = ()
def print(self) -> None:
@@ -22,11 +21,9 @@ class FirecrackerBottleCleanupPlan(BottleCleanupPlan):
return
for pid in self.vm_pids:
info(f"firecracker VM process: pid {pid}")
for name in self.containers:
info(f"firecracker sidecar container: {name}")
for path in self.run_dirs:
info(f"firecracker run dir: {path}")
@property
def empty(self) -> bool:
return not (self.vm_pids or self.containers or self.run_dirs)
return not (self.vm_pids or self.run_dirs)
@@ -13,15 +13,19 @@ from .. import BottlePlan
class FirecrackerBottlePlan(BottlePlan):
slug: str
forwarded_env: dict[str, str] = field(repr=False)
# Stamped by launch once the sidecar is up and its ports are
# Stamped by launch once the gateway is up and its ports are
# published on the host-side TAP IP (empty at prepare time).
agent_proxy_url: str = ""
agent_git_gate_url: str = ""
agent_supervise_url: str = ""
# Per-bottle identity token the agent presents on every attributed request
# (egress proxy credentials, git-gate/supervise headers); set by launch
# from the orchestrator registration. Empty pre-registration.
identity_token: str = ""
@property
def container_name(self) -> str:
"""Instance name, reused for the sidecar container + VM run dir.
"""Instance name, reused for the gateway container + VM run dir.
Matches the `bot-bottle-<slug>` convention the other backends
use so cleanup/enumerate discovery-by-prefix keeps working."""
return self.agent_provision.instance_name
+2 -23
View File
@@ -1,9 +1,8 @@
"""Cleanup for the Firecracker backend.
Orphans are: firecracker VMM processes whose config lives under our run
dir, the `bot-bottle-sidecars-*` containers, and the per-bottle run
dirs. TAP slots free themselves (the flock drops when the launcher
exits), so there is nothing to reclaim there.
dir, and the per-bottle run dirs. TAP slots free themselves (the flock
drops when the launcher exits), so there is nothing to reclaim there.
"""
from __future__ import annotations
@@ -18,8 +17,6 @@ from ...log import info
from . import util
from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan
_SIDECAR_PREFIX = "bot-bottle-sidecars-"
def _run_root() -> Path:
return util.cache_dir() / "run"
@@ -46,17 +43,6 @@ def _orphan_vm_pids() -> list[int]:
return pids
def _sidecar_containers() -> list[str]:
result = subprocess.run(
["docker", "ps", "-a", "--format", "{{.Names}}",
"--filter", f"name={_SIDECAR_PREFIX}"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
return sorted(n.strip() for n in result.stdout.splitlines() if n.strip())
def _run_dirs() -> list[str]:
run_root = _run_root()
if not run_root.is_dir():
@@ -67,7 +53,6 @@ def _run_dirs() -> list[str]:
def prepare_cleanup() -> FirecrackerBottleCleanupPlan:
return FirecrackerBottleCleanupPlan(
vm_pids=tuple(_orphan_vm_pids()),
containers=tuple(_sidecar_containers()),
run_dirs=tuple(_run_dirs()),
)
@@ -79,12 +64,6 @@ def cleanup(plan: FirecrackerBottleCleanupPlan) -> None:
os.kill(pid, signal.SIGTERM)
except ProcessLookupError:
pass
for name in plan.containers:
info(f"docker rm -f {name}")
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
for path in plan.run_dirs:
info(f"rm -rf {path}")
shutil.rmtree(path, ignore_errors=True)
@@ -0,0 +1,109 @@
"""Consolidated bottle launch sequence for the Firecracker backend
(PRD 0070, Stage B).
The shared gateway + orchestrator control plane run in a single persistent
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
the gateway's egress / supervise / git-http ports at the infra VM via a
PREROUTING DNAT on their own host-side TAP IP (see
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
over HTTP at the infra VM's guest IP.
Attribution is by the agent VM's guest IP, unspoofable by construction: the
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
expected VM can source-IP that address.
Sequence:
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
prior launcher may already have booted it);
2. register the bottle by its guest IP (attribution key) → bottle id +
identity token;
3. provision its git-gate repos/creds into the gateway VM (over SSH);
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
The TAP slot allocation, rootfs build, and VM boot are the caller's job.
"""
from __future__ import annotations
from dataclasses import dataclass
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.lifecycle import (
OrchestratorStartError, # re-exported so callers can catch it
)
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from . import infra_vm
class ConsolidatedLaunchError(RuntimeError):
"""The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
class LaunchContext:
"""What the Firecracker launch needs from the consolidated sequence."""
bottle_id: str
identity_token: str
source_ip: str # the VM's guest IP — the attribution key
gateway_ca_pem: str # the shared gateway CA the provisioner installs
orchestrator_url: str
def launch_consolidated(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
guest_ip: str,
image_ref: str = "",
tokens: dict[str, str] | None = None,
) -> LaunchContext:
"""Ensure the infra VM is up, register the bottle by its guest IP, and
provision its git-gate state into the gateway VM. Returns the context the
agent-VM launch needs. Raises on failure — the caller tears down."""
infra = infra_vm.ensure_running()
url = infra.control_plane_url
client = OrchestratorClient(url)
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
guest_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(
infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
except Exception:
client.teardown_bottle(reg.bottle_id)
raise
# The shared gateway CA every agent on this host trusts for TLS
# interception — fetched from the infra VM over SSH.
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=guest_ip,
gateway_ca_pem=infra.gateway_ca_pem(),
orchestrator_url=url,
)
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
NOT stop the infra VM — it's a persistent per-host singleton shared by
every bottle."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
__all__ = [
"LaunchContext",
"launch_consolidated",
"teardown_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
]
+4 -33
View File
@@ -1,43 +1,14 @@
"""Active-agent enumeration for the Firecracker backend.
The agent runs in a VM (no container to list), so a live bottle is
identified by its running sidecar container `bot-bottle-sidecars-<slug>`
— the same discovery-by-prefix the other backends use.
The backend is disabled during the companion-container removal (#385) — it can't
launch bottles, so there are none to enumerate. Real enumeration returns
with the backend's consolidated relaunch (#354).
"""
from __future__ import annotations
import subprocess
from ...bottle_state import read_metadata
from .. import ActiveAgent
_SIDECAR_PREFIX = "bot-bottle-sidecars-"
def enumerate_active() -> list[ActiveAgent]:
result = subprocess.run(
["docker", "ps", "--format", "{{.Names}}",
"--filter", f"name={_SIDECAR_PREFIX}"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
return []
out: list[ActiveAgent] = []
for name in sorted(n.strip() for n in result.stdout.splitlines() if n.strip()):
slug = name[len(_SIDECAR_PREFIX):]
metadata = read_metadata(slug)
if metadata is None or metadata.backend != "firecracker":
# Skip sidecars owned by another backend (docker shares the
# container-name prefix).
continue
out.append(ActiveAgent(
backend_name="firecracker",
slug=slug,
agent_name=metadata.agent_name,
started_at=metadata.started_at,
services=(),
label=metadata.label,
color=metadata.color,
))
return out
return []
@@ -78,20 +78,32 @@ def _config(
vcpus: int,
mem_mib: int,
guest_mac: str,
data_drive: Path | None = None,
) -> dict[str, object]:
drives: list[dict[str, object]] = [
{
"drive_id": "rootfs",
"path_on_host": str(rootfs),
"is_root_device": True,
"is_read_only": False,
}
]
# A second virtio-block device (guest /dev/vdb) — the infra VM's
# persistent registry "volume", a host-side ext4 file that outlives the
# ephemeral rootfs across VM restarts.
if data_drive is not None:
drives.append({
"drive_id": "data",
"path_on_host": str(data_drive),
"is_root_device": False,
"is_read_only": False,
})
return {
"boot-source": {
"kernel_image_path": str(util.kernel_path()),
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
},
"drives": [
{
"drive_id": "rootfs",
"path_on_host": str(rootfs),
"is_root_device": True,
"is_read_only": False,
}
],
"drives": drives,
"network-interfaces": [
{
"iface_id": "eth0",
@@ -118,9 +130,16 @@ def boot(
vcpus: int = 2,
mem_mib: int = 2048,
guest_mac: str = "06:00:AC:10:00:02",
detached: bool = False,
data_drive: Path | None = None,
) -> VmHandle:
"""Write the config and launch the VMM. Returns once the process is
spawned; callers wait for SSH readiness separately."""
spawned; callers wait for SSH readiness separately.
`detached` starts the VMM in its own session (`start_new_session`) so it
survives the launcher exiting — used for the persistent per-host infra
VM, which must outlive the short-lived `start` process (agent VMs stay
attached and are torn down with the launcher)."""
run_dir.mkdir(parents=True, exist_ok=True)
config_path = run_dir / "config.json"
console_log = run_dir / "console.log"
@@ -128,6 +147,7 @@ def boot(
_config(
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
data_drive=data_drive,
),
indent=2,
))
@@ -137,6 +157,7 @@ def boot(
process = subprocess.Popen(
["firecracker", "--no-api", "--config-file", str(config_path)],
stdout=log_fh, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
start_new_session=detached,
)
return VmHandle(process=process, guest_ip=guest_ip, console_log=console_log)
+44 -30
View File
@@ -1,9 +1,12 @@
"""FirecrackerFreezer — snapshot a running microVM to a Docker image.
"""FirecrackerFreezer — snapshot a running microVM to a rootfs tar.
The VM is live and can't be block-copied safely, so — like the macOS
backend — we stream the guest root filesystem out over the control
channel (SSH here) and rebuild an image from it. The bottle keeps
running after the snapshot.
channel (SSH here). Unlike the other backends this needs no Docker: the
tar *is* the resumable artifact. `resume` extracts it and rebuilds a
fresh per-bottle ext4 with `mke2fs -d` (see `util.build_committed_rootfs_dir`
and `launch._build_agent_base`). The bottle keeps running after the
snapshot.
"""
from __future__ import annotations
@@ -11,9 +14,9 @@ from __future__ import annotations
import json
import os
import subprocess
import tempfile
from pathlib import Path
from ...bottle_state import committed_rootfs_path
from ...log import die, info
from .. import ActiveAgent
from ..freeze import Freezer
@@ -30,14 +33,13 @@ class FirecrackerFreezer(Freezer):
if not private_key.is_file() or not guest_ip:
die(f"cannot freeze {agent.slug}: run dir {run_dir} is missing the "
f"SSH key or VM config (is the bottle still running?)")
image_tag = f"bot-bottle-committed-{agent.slug}:latest"
_commit_via_ssh(private_key, guest_ip, image_tag)
info(f"committed {agent.slug} -> {image_tag!r}")
return image_tag
tar_path = committed_rootfs_path(agent.slug)
_commit_rootfs_via_ssh(private_key, guest_ip, tar_path)
info(f"committed {agent.slug} -> {tar_path}")
return str(tar_path)
def _export_hint(self, slug: str, image_ref: str) -> None:
info(f"to export for migration: docker image save {image_ref} "
f"-o {slug}.tar")
info(f"to export for migration: cp {image_ref} {slug}.tar")
def _guest_ip_from_config(config_path: Path) -> str:
@@ -53,24 +55,36 @@ def _guest_ip_from_config(config_path: Path) -> str:
return ""
def _commit_via_ssh(private_key: Path, guest_ip: str, image_tag: str) -> None:
with tempfile.TemporaryDirectory(prefix="bot-bottle-fc-commit.") as tmp:
rootfs_tar = os.path.join(tmp, "rootfs.tar")
ssh = util.ssh_base_argv(private_key, guest_ip)
with open(rootfs_tar, "wb") as tar_out:
result = subprocess.run(
[*ssh, "--", "tar", "--create", "--one-file-system",
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
"--exclude=./run", "--file=-", "--directory=/", "."],
stdout=tar_out, stderr=subprocess.PIPE, check=False,
)
if result.returncode != 0:
die(f"ssh tar for {guest_ip} failed: "
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
with open(os.path.join(tmp, "Dockerfile"), "w", encoding="utf-8") as f:
f.write("FROM scratch\nADD rootfs.tar /\nUSER node\nWORKDIR /home/node\n")
build = subprocess.run(
["docker", "build", "-t", image_tag, tmp], check=False,
def _commit_rootfs_via_ssh(private_key: Path, guest_ip: str, tar_path: Path) -> None:
"""Stream the guest rootfs out over SSH into `tar_path`. Excludes the
virtual/live mounts (proc/sys/dev/run) — resume recreates those empty
mount points. Written to a `.partial` sibling and renamed on success so
a failed freeze never leaves a truncated artifact in its place."""
tar_path.parent.mkdir(parents=True, exist_ok=True)
partial = tar_path.with_name(tar_path.name + ".partial")
ssh = util.ssh_base_argv(private_key, guest_ip)
# The snapshot can contain the bottle's private workspace, so keep it
# owner-only (0600) for the whole stream. The `os.open` mode only applies
# on *creation*, so unlink any leftover partial (a prior interrupted run
# could have left it world-readable, or something could swap in a symlink
# at this predictable name) and exclusively recreate it — O_EXCL|O_NOFOLLOW
# — then fchmod immediately so umask can't loosen it. Re-assert after the
# rename too (os.replace carries the source mode, but be explicit).
partial.unlink(missing_ok=True)
fd = os.open(
partial, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600
)
os.fchmod(fd, 0o600)
with os.fdopen(fd, "wb") as tar_out:
result = subprocess.run(
[*ssh, "--", "tar", "--create", "--one-file-system",
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
"--exclude=./run", "--file=-", "--directory=/", "."],
stdout=tar_out, stderr=subprocess.PIPE, check=False,
)
if build.returncode != 0:
die(f"docker build for {image_tag!r} failed")
if result.returncode != 0:
partial.unlink(missing_ok=True)
die(f"ssh tar for {guest_ip} failed: "
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
os.replace(partial, tar_path)
os.chmod(tar_path, 0o600)
@@ -0,0 +1,224 @@
"""Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
Agent Dockerfiles build **inside the persistent per-host infra VM**
(`infra_vm.py`), which carries buildah (rootless, daemonless): no host Docker
daemon, no root-equivalent `docker` group. The build runs over SSH against the
infra VM and its rootfs streams back to the host, where the existing
`mke2fs -d` path (`util.build_rootfs_ext4`) turns it into a bootable ext4.
Building in the infra VM — rather than a throwaway builder VM — means there is
one buildah image (`bot-bottle-infra`) and no contention for the orchestrator
TAP. Tradeoff: an untrusted Dockerfile's `RUN` steps share the VM with the
control plane + gateway (buildah `--isolation chroot` isn't a hard boundary) —
the accepted single-VM blast-radius tradeoff, re-splittable into a disposable
builder (booted from this same image on its own TAP) later.
"""
from __future__ import annotations
import fcntl
import hashlib
import os
import shutil
import subprocess
from contextlib import contextmanager
from pathlib import Path
from typing import Generator
from ...log import die, info
from . import infra_vm, util
# vfs + chroot: buildah works as root in the microVM (no fuse-overlayfs /
# overlay module / subuid maps). `--isolation` is a build/run-only flag;
# `from`/`mount` take just the store.
_BUILD_FLAGS = "--isolation chroot --storage-driver vfs"
_STORE_FLAG = "--storage-driver vfs"
_BUILD_TIMEOUT_SECONDS = 900.0
def _dockerfile_hash(dockerfile: Path) -> str:
"""Cache key: the Dockerfile's content. The shipped agent Dockerfiles
COPY nothing from the build context (see .dockerignore), so their content
fully determines the image; a Dockerfile that adds COPY will want the
context folded in here too."""
return hashlib.sha256(dockerfile.read_bytes()).hexdigest()[:16]
def build_agent_rootfs_dir(
dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
) -> Path:
"""Build `dockerfile` in the infra VM (buildah, no host docker), export its
rootfs, inject the guest boot bits, and return the cached base dir — the
same shape `util.build_rootfs_ext4` consumes. Cached by Dockerfile content,
so a repeat launch skips the rebuild.
`smoke_test` (the provider's declared argv, e.g. `("claude","--version")`)
is run in the freshly built image before export, catching an npm
silent-failure image at build time rather than at first agent use."""
digest = _dockerfile_hash(dockerfile)
base = util.cache_dir() / "rootfs" / f"agent-{digest}"
if (base / ".bb-ready").is_file():
info(f"using cached agent rootfs {base.name}")
return base
# Serialize builds: the infra VM's buildah store + this cache dir are
# shared, so concurrent `start`s must not build into them at once. The
# lock covers the cache lookup + build + atomic publish; the ready
# fast-path above takes no lock.
with _build_lock():
if (base / ".bb-ready").is_file(): # another build finished while we waited
info(f"using cached agent rootfs {base.name}")
return base
# Build into a temp dir and publish by atomic rename, so a partial
# build is never visible as `agent-<digest>`.
staging = util.cache_dir() / "rootfs" / f".building-{digest}"
shutil.rmtree(staging, ignore_errors=True)
staging.mkdir(parents=True)
info(f"building agent image {image_tag!r} in the infra VM")
_build_in_infra(dockerfile, staging, smoke_test, digest)
util.inject_guest_boot(staging)
(staging / ".bb-ready").write_text("ok\n")
shutil.rmtree(base, ignore_errors=True)
os.rename(staging, base)
return base
@contextmanager
def _build_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing agent-image builds (shared infra
buildah store + cache dir). flock auto-releases on a crash."""
lock_path = util.cache_dir() / "rootfs" / ".build.lock"
lock_path.parent.mkdir(parents=True, exist_ok=True)
handle = open(lock_path, "w", encoding="utf-8")
try:
fcntl.flock(handle, fcntl.LOCK_EX)
yield
finally:
handle.close()
def _build_in_infra(
dockerfile: Path, base: Path, smoke_test: tuple[str, ...], digest: str,
) -> None:
"""Ensure the infra VM is up, `buildah build` the Dockerfile in it, smoke
test the image, and stream its rootfs into `base`. The infra VM persists;
only the per-build container/image/context are cleaned up."""
infra = infra_vm.ensure_running()
key, ip = infra.private_key, infra.guest_ip
tag = f"bot-bottle-agent-build-{digest}"
ctx = f"/tmp/agent-build-{digest}"
smoke_ctr, export_ctr = f"{tag}-smoke", f"{tag}-export"
def _cleanup() -> None:
# Remove only THIS build's working containers/image/context — never
# `buildah rm -a`, which would nuke a concurrent build's container.
_ssh(key, ip,
f"buildah rm {smoke_ctr} {export_ctr} >/dev/null 2>&1; "
f"buildah rmi {_STORE_FLAG} {tag} >/dev/null 2>&1; rm -rf {ctx}",
timeout=60)
_cleanup() # clear leftovers from a crashed prior build of this digest
try:
prep = _ssh(key, ip, f"mkdir -p {ctx}/ctx")
if prep.returncode != 0:
die(f"preparing build dir in the infra VM failed: {prep.stderr.strip()}")
_send_dockerfile(key, ip, dockerfile, ctx)
_buildah_build(key, ip, ctx, tag)
_smoke_test(key, ip, tag, smoke_ctr, smoke_test)
_stream_rootfs(key, ip, tag, export_ctr, base)
finally:
_cleanup()
def _ssh(private_key: Path, guest_ip: str, script: str,
*, timeout: float = 60.0) -> subprocess.CompletedProcess[str]:
return subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [script],
capture_output=True, text=True, timeout=timeout, check=False,
)
def _ssh_streamed(private_key: Path, guest_ip: str, script: str,
*, timeout: float) -> int:
"""Run an SSH command letting the remote's stdout/stderr flow straight to
ours (no capture), for long chatty steps where live progress beats a
silent wait. Returns the exit code."""
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [script],
timeout=timeout, check=False,
)
return proc.returncode
def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path, ctx: str) -> None:
proc = subprocess.run(
util.ssh_base_argv(private_key, guest_ip) + [f"cat > {ctx}/Dockerfile"],
input=dockerfile.read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
die(f"sending Dockerfile to the infra VM failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def _buildah_build(private_key: Path, guest_ip: str, ctx: str, tag: str) -> None:
# Stream buildah's step-by-step output straight to our stderr (like the
# docker backend's `docker build`), so a long first build (base pull +
# apt/npm installs) shows live progress instead of a silent wait. The
# remote stderr is where buildah writes its `STEP i/n` lines.
info(f"buildah build {tag} in the infra VM (streaming output)")
rc = _ssh_streamed(
private_key, guest_ip,
f"buildah build {_BUILD_FLAGS} -t {tag} -f {ctx}/Dockerfile {ctx}/ctx",
timeout=_BUILD_TIMEOUT_SECONDS,
)
if rc != 0:
die(f"buildah build in the infra VM failed (exit {rc}); "
"see the build output above.")
def _smoke_test(private_key: Path, guest_ip: str, tag: str, ctr: str,
argv: tuple[str, ...]) -> None:
"""Run the provider's smoke argv inside the freshly built image
(`buildah run`, which uses the image's own PATH), failing the build
loudly if the CLI is a broken stub. No-op without a declared test. Uses a
named working container (`ctr`) so cleanup is scoped to this build."""
if not argv:
return
cmd = (
f"set -e; buildah from {_STORE_FLAG} --name {ctr} {tag} >/dev/null; "
f"buildah run {_BUILD_FLAGS} {ctr} -- {' '.join(argv)}; rc=$?; "
f"buildah rm {ctr} >/dev/null 2>&1 || true; exit $rc"
)
result = _ssh(private_key, guest_ip, cmd, timeout=120)
if result.returncode != 0:
detail = (result.stdout + result.stderr).strip().splitlines()[-10:]
die(f"agent image failed its post-build smoke test "
f"({' '.join(argv)}):\n" + "\n".join(detail))
def _stream_rootfs(private_key: Path, guest_ip: str, tag: str, ctr: str, base: Path) -> None:
"""`buildah mount` the built image in the infra VM and pipe its rootfs tar
straight into `base` on the host (extracted as the non-root host user, so
uid 0 isn't preserved — the guest init restores /root ownership). Uses a
named working container so cleanup is scoped to this build."""
export = (
f"set -e; buildah from {_STORE_FLAG} --name {ctr} {tag} >/dev/null; "
f"mnt=$(buildah mount {_STORE_FLAG} {ctr}); "
f"tar -C \"$mnt\" -cf - ."
)
ssh_proc = subprocess.Popen(
util.ssh_base_argv(private_key, guest_ip) + [export],
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
)
assert ssh_proc.stdout is not None
untar = subprocess.run(
["tar", "-x", "-C", str(base)], stdin=ssh_proc.stdout, check=False,
)
ssh_proc.stdout.close()
ssh_err = (ssh_proc.stderr.read().decode(errors="replace")
if ssh_proc.stderr else "")
rc = ssh_proc.wait()
if rc != 0 or untar.returncode != 0:
die(f"exporting the built rootfs from the infra VM failed: "
f"{ssh_err.strip() or '<no stderr>'}")
@@ -0,0 +1,199 @@
"""Prebuilt infra-VM rootfs, pulled as an artifact (PRD 0069 Stage 2).
The Firecracker infra VM boots a fixed rootfs (orchestrator control plane +
gateway + buildah, control-plane init as PID 1) that does not vary per launch —
the per-boot bits (authorized_keys, guest IP) ride the kernel cmdline, so one
rootfs boots on any host. Instead of building that rootfs on the launch host
with Docker, we build it **off-host** and publish it as a versioned, ready-to-
boot ext4 (gzip-compressed) to a Gitea **generic package**; the launch host
downloads + verifies + boots it. No Docker, no image tooling on the launch
host — just an HTTP fetch and gunzip.
publish (off-host, see publish_infra.py):
docker build -> rootfs dir -> mke2fs -> gzip -> PUT generic package
pull (this module, launch host):
GET .../rootfs.ext4.gz (+ .sha256) -> verify -> gunzip -> boot
The artifact **version** is a content hash of everything baked into the rootfs
(the shipped bot_bottle package, the three Dockerfiles, and the init), so a
launch host always pulls the artifact matching its code and a content change
can't silently boot a stale rootfs. A checksum mismatch fails closed.
Set `BOT_BOTTLE_INFRA_BUILD=local` to skip the pull and build the rootfs
locally with Docker (dev iteration on the Dockerfiles) — see `infra_vm`.
"""
from __future__ import annotations
import gzip
import hashlib
import os
import shutil
import urllib.error
import urllib.request
from pathlib import Path
from ...log import die, info
from . import util
# Bump if the on-disk artifact *format* changes (compression, layout) so a new
# scheme can't collide with a cached/published artifact of the old one.
_ARTIFACT_FORMAT = "1"
_REPO_ROOT = Path(__file__).resolve().parents[3]
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra")
_DEFAULT_BASE = "https://gitea.dideric.is"
_DEFAULT_OWNER = "didericis"
_PACKAGE = "bot-bottle-firecracker-infra"
# Streaming copy chunk for the (hundreds-of-MB) download.
_CHUNK = 1 << 20
def local_build_requested() -> bool:
"""True when the operator opted into the dev Docker-build path instead of
pulling the published artifact (`BOT_BOTTLE_INFRA_BUILD=local`)."""
return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"
def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) -> str:
"""Content hash (16 hex) of everything baked into the infra rootfs: the
whole shipped `bot_bottle` package, the three fixed Dockerfiles, and the
guest init. Deterministic across the publish host and the launch host when
both run the same checkout, so the tag the launch host pulls is exactly the
tag publish produced.
The package is `COPY bot_bottle /app/bot_bottle`'d wholesale into the image,
so hash *every* regular file under it — not just `*.py`. Non-Python inputs
(e.g. `egress_entrypoint.sh`, `netpool.defaults.env`) are baked in too, and
a change to one must bump the version or a launch host could boot a stale
rootfs whose code differs from its checkout. `__pycache__`/`.pyc` are the
only exclusions — build artifacts, never copied."""
h = hashlib.sha256()
h.update(f"format={_ARTIFACT_FORMAT}\n".encode())
pkg = repo_root / "bot_bottle"
for path in sorted(pkg.rglob("*")):
if not path.is_file():
continue
if "__pycache__" in path.parts or path.suffix == ".pyc":
continue
h.update(str(path.relative_to(repo_root)).encode())
h.update(b"\0")
h.update(path.read_bytes())
for name in _DOCKERFILES:
h.update(name.encode())
h.update(b"\0")
h.update((repo_root / name).read_bytes())
h.update(b"init\0")
h.update(init_script.encode())
return h.hexdigest()[:16]
def _config() -> tuple[str, str, str]:
"""(base_url, owner, token) for the generic-package endpoint. Base + owner
are overridable for other deployments / mirrors; the token comes solely from
`BOT_BOTTLE_INFRA_ARTIFACT_TOKEN` (a dedicated package-scoped token, kept
separate from the general-purpose Gitea token) and is optional — a public
package needs none to pull."""
base = os.environ.get("BOT_BOTTLE_INFRA_ARTIFACT_BASE", _DEFAULT_BASE).rstrip("/")
owner = os.environ.get("BOT_BOTTLE_INFRA_ARTIFACT_OWNER", _DEFAULT_OWNER)
token = os.environ.get("BOT_BOTTLE_INFRA_ARTIFACT_TOKEN", "")
return base, owner, token
def artifact_url(version: str, filename: str) -> str:
"""The generic-package download URL for one file of this version's
artifact (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
base, owner, _ = _config()
return f"{base}/api/packages/{owner}/generic/{_PACKAGE}/{version}/{filename}"
_GZ_NAME = "rootfs.ext4.gz"
_SHA_NAME = "rootfs.ext4.gz.sha256"
def _cache_root(version: str) -> Path:
return util.cache_dir() / "infra-artifact" / version
def _open(url: str) -> urllib.request.Request:
_, _, token = _config()
req = urllib.request.Request(url)
if token:
req.add_header("Authorization", f"token {token}")
return req
def _download(url: str, dest: Path) -> None:
"""Stream `url` to `dest` (atomic via a `.part` sibling)."""
tmp = dest.with_suffix(dest.suffix + ".part")
try:
with urllib.request.urlopen(_open(url)) as resp, open(tmp, "wb") as out:
shutil.copyfileobj(resp, out, _CHUNK)
except urllib.error.HTTPError as e:
tmp.unlink(missing_ok=True)
if e.code == 404:
die(
f"infra artifact not published for this code version.\n"
f" missing: {url}\n"
f" publish it from a build host (Docker):\n"
f" python3 -m bot_bottle.backend.firecracker.publish_infra\n"
f" or build the rootfs locally: BOT_BOTTLE_INFRA_BUILD=local"
)
die(f"downloading infra artifact failed (HTTP {e.code}): {url}")
except urllib.error.URLError as e:
tmp.unlink(missing_ok=True)
die(f"infra artifact registry unreachable: {url} ({e.reason})")
tmp.replace(dest)
def _sha256_file(path: Path) -> str:
h = hashlib.sha256()
with open(path, "rb") as fh:
for chunk in iter(lambda: fh.read(_CHUNK), b""):
h.update(chunk)
return h.hexdigest()
def ensure_artifact_gz(version: str) -> Path:
"""The verified, cached `rootfs.ext4.gz` for `version` — downloading it (and
its `.sha256`) once, then reusing it. Fail-closed on a checksum mismatch:
the partial is removed and we die rather than boot an unverified rootfs."""
root = _cache_root(version)
root.mkdir(parents=True, exist_ok=True)
gz = root / _GZ_NAME
ok = root / ".verified"
if gz.is_file() and ok.is_file():
return gz
info(f"pulling infra rootfs artifact {_PACKAGE}/{version}")
_download(artifact_url(version, _GZ_NAME), gz)
sha = root / _SHA_NAME
_download(artifact_url(version, _SHA_NAME), sha)
expected = sha.read_text().split()[0].strip().lower()
actual = _sha256_file(gz)
if actual != expected:
gz.unlink(missing_ok=True)
sha.unlink(missing_ok=True)
die(
f"infra artifact checksum mismatch for {version}:\n"
f" expected {expected}\n"
f" actual {actual}\n"
f" refusing to boot an unverified rootfs."
)
ok.write_text("ok\n")
return gz
def materialize_ext4(version: str, dest: Path) -> None:
"""Ensure the verified artifact is cached, then gunzip it to `dest` — a
fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
pristine). Atomic via a `.part` sibling."""
gz = ensure_artifact_gz(version)
tmp = dest.with_suffix(dest.suffix + ".part")
info(f"expanding infra rootfs -> {dest}")
with gzip.open(gz, "rb") as src, open(tmp, "wb") as out:
shutil.copyfileobj(src, out, _CHUNK)
tmp.replace(dest)
+440
View File
@@ -0,0 +1,440 @@
"""The per-host infra VM for the Firecracker backend (PRD 0070 Stage B).
A single persistent microVM that runs the orchestrator **control plane** (and,
in a following step, the gateway **data plane**) — the trusted per-host service
the docker backend runs as containers. It boots on the NAT'd orchestrator link
(`netpool.orch_slot()`): the host CLI reaches its control plane over HTTP at the
guest IP, and agent VMs reach its gateway ports over VM-to-VM routing.
Build-from-source (the default while the design churns): the rootfs is exported
from the locally built orchestrator image, which bakes the stdlib-only
control-plane source. A pull-from-registry mode (Gitea's OCI registry) becomes
the default later.
SSH is left enabled for debugging; the control plane is the load-bearing
surface.
"""
from __future__ import annotations
import fcntl
import hashlib
import os
import shlex
import signal
import stat
import subprocess
import time
import urllib.error
import urllib.request
from contextlib import contextmanager
from dataclasses import dataclass
from pathlib import Path
from typing import Generator
from ...log import die, info
from ..docker import util as docker_mod
from ..docker.gateway_provision import GatewayProvisionError
from . import firecracker_vm, infra_artifact, netpool, util
# The single infra-VM image: gateway data plane + baked control-plane source
# (Dockerfile.infra FROM the gateway image). Built from source by default;
# a pull-from-registry mode lands later.
_INFRA_IMAGE = "bot-bottle-infra:latest"
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
_REPO_ROOT = Path(__file__).resolve().parents[3]
CONTROL_PLANE_PORT = 8099
# Gateway data-plane ports (agent-facing): egress proxy, supervise MCP,
# git-http. Reached by agent VMs over VM-to-VM routing (added next).
EGRESS_PORT = 9099
SUPERVISE_PORT = 9100
GIT_HTTP_PORT = 9420
# mitmproxy writes its CA here a beat after start; agents install it to trust
# the gateway's TLS interception.
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
# The infra VM makes direct upstream connections (gateway egress, and buildah
# during builds), and the kernel `ip=` cmdline sets no resolver. Public for
# now; routing DNS through a filtered path is a later refinement.
_INFRA_RESOLVER = "1.1.1.1"
_HEALTH_TIMEOUT_SECONDS = 45.0
_HEALTH_POLL_SECONDS = 0.5
_CA_TIMEOUT_SECONDS = 30.0
@dataclass
class InfraVm:
"""A handle to the per-host infra VM: its guest IP and the stable SSH key
used to fetch the gateway CA / provision git-gate. `vm` is the live VMM
handle when this process booted it, and None when adopting a singleton a
prior launcher started (teardown then goes through the PID file)."""
guest_ip: str
private_key: Path
vm: firecracker_vm.VmHandle | None = None
@property
def control_plane_url(self) -> str:
return f"http://{self.guest_ip}:{CONTROL_PLANE_PORT}"
def terminate(self) -> None:
"""Stop the infra VM — via the live handle if we booted it, else the
PID file (adopting-process case)."""
if self.vm is not None:
self.vm.terminate()
else:
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def gateway_ca_pem(self, *, timeout: float = _CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) that agents install to trust its
TLS interception. Generated a moment after boot, so this polls over
SSH until it appears (mirrors DockerGateway.ca_cert_pem)."""
deadline = time.monotonic() + timeout
while True:
proc = subprocess.run(
util.ssh_base_argv(self.private_key, self.guest_ip)
+ [f"cat {_GATEWAY_CA_PATH}"],
capture_output=True, text=True, timeout=15, check=False,
)
if proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout:
return proc.stdout
if time.monotonic() >= deadline:
die(f"gateway CA not available after {timeout:g}s: "
f"{proc.stderr.strip() or 'empty'}")
time.sleep(_HEALTH_POLL_SECONDS)
def ensure_built() -> None:
"""Ensure the infra rootfs is available before boot.
Default (docker-free, PRD 0069 Stage 2): download + verify the prebuilt
rootfs artifact matching this code version (see `infra_artifact`); the
launch host needs no Docker. `BOT_BOTTLE_INFRA_BUILD=local` instead builds
the three fixed images from source with host Docker — the infra image
`COPY --from`s the orchestrator image and is `FROM` the gateway image, so
both must exist first — for iterating on the Dockerfiles."""
if infra_artifact.local_build_requested():
build_infra_images_with_docker()
return
infra_artifact.ensure_artifact_gz(
infra_artifact.infra_artifact_version(_infra_init()))
def build_infra_images_with_docker() -> None:
"""Build the three fixed images from source with host Docker: orchestrator,
gateway, then the combined infra image (`COPY --from` orchestrator, `FROM`
gateway). The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local`
mode; `publish_infra` uses it off-host to produce the published artifact."""
docker_mod.build_image(
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
docker_mod.build_image(
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra")
def build_infra_rootfs_dir() -> Path:
"""The infra VM's base rootfs: the infra image prepared with the
control-plane + gateway init as PID 1. The init's content is folded into
the cache key so an init change rebuilds the rootfs (the base image digest
alone wouldn't catch it)."""
init = _infra_init()
tag = hashlib.sha256(init.encode()).hexdigest()[:8]
return util.build_base_rootfs_dir(
_INFRA_IMAGE, variant=f"-infra-{tag}", init_script=init,
)
def ensure_running() -> InfraVm:
"""Idempotent per-host singleton. Adopt the infra VM if its control plane
is already healthy (a prior launcher booted it — it outlives short-lived
`start` processes); otherwise clear any stale VM and boot a fresh one.
Returns a handle usable for CA fetch / git-gate provisioning.
Concurrency-safe: the cold stop/build/boot path is serialized by a host
flock, so two simultaneous first launches don't both boot on the same
rootfs/PID. The healthy fast-path takes no lock."""
slot = netpool.orch_slot()
url = f"http://{slot.guest_ip}:{CONTROL_PLANE_PORT}"
key = _infra_dir() / "id_ed25519"
if key.exists() and _health_ok(url):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
with _singleton_lock():
# Re-check under the lock: another launcher may have booted it while
# we waited for the lock (double-checked, so we adopt not re-boot).
if key.exists() and _health_ok(url):
info(f"adopting running infra VM at {url}")
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
stop() # clear a stale/hung VM holding the link before booting fresh
ensure_built()
infra = boot()
wait_for_health(infra)
return infra
@contextmanager
def _singleton_lock() -> Generator[None, None, None]:
"""Host-level exclusive lock serializing the infra VM's cold create path
(`stop`/`ensure_built`/`boot`). flock auto-releases if the launcher
crashes, so the lock is never leaked."""
lock_path = _infra_dir() / "singleton.lock"
handle = open(lock_path, "w", encoding="utf-8")
try:
fcntl.flock(handle, fcntl.LOCK_EX)
yield
finally:
handle.close()
def stop() -> None:
"""Stop the infra VM singleton (idempotent — absent is success)."""
_kill_pidfile()
_pid_file().unlink(missing_ok=True)
def boot() -> InfraVm:
"""Boot the infra VM (detached, so it outlives the launcher) on the
orchestrator link, recording its PID. Prefer `ensure_running`."""
slot = netpool.orch_slot()
if not netpool.tap_present(slot.iface):
die(f"orchestrator link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
run_dir = _infra_dir()
rootfs = run_dir / "rootfs.ext4"
if infra_artifact.local_build_requested():
util.build_rootfs_ext4(build_infra_rootfs_dir(), rootfs, slack_mib=8192)
else:
# Prebuilt artifact already carries the buildah build slack; expand it
# to a fresh writable rootfs for this boot.
infra_artifact.materialize_ext4(
infra_artifact.infra_artifact_version(_infra_init()), rootfs)
private_key, pubkey = _stable_keypair()
info(f"booting infra VM on {slot.iface} (guest {slot.guest_ip})")
vm = firecracker_vm.boot(
name="bot-bottle-infra", rootfs=rootfs, tap=slot.iface,
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
run_dir=run_dir, mem_mib=4096, detached=True,
data_drive=_ensure_registry_volume(),
)
_pid_file().write_text(str(vm.process.pid))
return InfraVm(guest_ip=slot.guest_ip, private_key=private_key, vm=vm)
def _infra_dir() -> Path:
d = util.cache_dir() / "infra"
d.mkdir(parents=True, exist_ok=True)
return d
def _pid_file() -> Path:
return _infra_dir() / "vm.pid"
# The registry "volume": a host-side ext4 file attached to the infra VM as a
# second virtio-block device (guest /dev/vdb), mounted at the control plane's
# DB dir. It outlives the ephemeral rootfs, so the bottle registry survives an
# infra-VM restart — the firecracker analogue of a docker volume. It is a
# plain ext4 file: `sudo mount -o loop <path>` on the host (with the VM
# stopped) to inspect bot-bottle.db directly.
_REGISTRY_SIZE = "512M"
def registry_volume_path() -> Path:
return _infra_dir() / "registry.ext4"
def _ensure_registry_volume() -> Path:
"""Create the empty ext4 registry volume on first use; reuse it after."""
vol = registry_volume_path()
if vol.exists():
return vol
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
proc = subprocess.run(
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
capture_output=True, text=True, check=False,
)
if proc.returncode != 0:
vol.unlink(missing_ok=True)
die(f"creating registry volume failed: {proc.stderr.strip()}")
return vol
def _stable_keypair() -> tuple[Path, str]:
"""The infra VM's SSH keypair — generated once and reused, so any later
launcher can SSH in (fetch CA / provision) even though a different process
booted the VM. The pubkey is re-injected on every boot via the cmdline."""
d = _infra_dir()
key, pub = d / "id_ed25519", d / "id_ed25519.pub"
if key.exists() and pub.exists():
return key, pub.read_text().strip()
key.unlink(missing_ok=True)
pub.unlink(missing_ok=True)
subprocess.run(
["ssh-keygen", "-t", "ed25519", "-N", "", "-q", "-f", str(key),
"-C", "bot-bottle-infra"],
check=True,
)
return key, pub.read_text().strip()
def _kill_pidfile() -> None:
"""SIGTERM (then SIGKILL) the recorded infra VMM, if it's still ours.
Guards against a recycled PID by checking the process is firecracker."""
try:
pid = int(_pid_file().read_text().strip())
except (OSError, ValueError):
return
try:
comm = Path(f"/proc/{pid}/comm").read_text().strip()
except OSError:
return # already gone
if comm != "firecracker":
return # PID recycled by an unrelated process
try:
os.kill(pid, signal.SIGTERM)
for _ in range(50):
if not Path(f"/proc/{pid}").exists():
return
time.sleep(0.1)
os.kill(pid, signal.SIGKILL)
except OSError:
pass
def _health_ok(url: str) -> bool:
try:
with urllib.request.urlopen(f"{url}/health", timeout=1.0) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
class SshGatewayTransport:
"""`GatewayTransport` for the gateway running in the infra VM — the docker
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
def __init__(self, private_key: Path, guest_ip: str) -> None:
self._key = private_key
self._ip = guest_ip
def exec(self, argv: list[str]) -> None:
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
capture_output=True, text=True, timeout=60, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
def cp_into(self, src: str, dest: str) -> None:
# Preserve the source mode (docker cp does): the access-hook is staged
# 0700 and git-http execs it directly — a plain `cat >` would land it
# 0644 and the exec fails with EACCES; keys stay 0600.
mode = stat.S_IMODE(os.stat(src).st_mode)
q = shlex.quote(dest)
proc = subprocess.run(
util.ssh_base_argv(self._key, self._ip)
+ [f"cat > {q} && chmod {mode:o} {q}"],
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
)
if proc.returncode != 0:
raise GatewayProvisionError(
f"infra gateway cp {src} -> {dest} failed: "
f"{proc.stderr.decode(errors='replace').strip()}")
def gateway_transport() -> SshGatewayTransport:
"""git-gate provisioning transport for the gateway in the infra VM, built
from the stable key + the orchestrator link's guest IP. Needs no live VM
handle, so teardown can use it too."""
return SshGatewayTransport(
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
def wait_for_health(
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
) -> None:
"""Poll the control plane's /health until it answers 200 or the deadline
passes. Dies (with the console tail) if the VMM exits early."""
url = f"{infra.control_plane_url}/health"
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if infra.vm is not None and not infra.vm.is_alive():
die(f"infra VM exited during boot (rc={infra.vm.process.returncode}).\n"
f"{firecracker_vm._console_tail(infra.vm.console_log)}")
try:
with urllib.request.urlopen(url, timeout=1.0) as resp:
if resp.status == 200:
info(f"infra control plane healthy at {infra.control_plane_url}")
return
except (urllib.error.URLError, TimeoutError, OSError):
pass
time.sleep(_HEALTH_POLL_SECONDS)
tail = (firecracker_vm._console_tail(infra.vm.console_log)
if infra.vm is not None else "")
die(f"infra control plane at {url} did not become healthy within "
f"{timeout:.0f}s.\n{tail}")
def _infra_init() -> str:
"""PID-1 init for the infra VM: mount the pseudo-filesystems, wire a
resolver, start dropbear (debug SSH), then launch the control plane and
the gateway data plane (multi-tenant against the local control plane)."""
return f"""#!/bin/sh
# bot-bottle Firecracker infra VM init (PID 1).
mount -t proc proc /proc 2>/dev/null
mount -t sysfs sys /sys 2>/dev/null
mount -t devtmpfs dev /dev 2>/dev/null
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
mount -o remount,rw / 2>/dev/null
# Export a real PATH: a bare-init shell resolves its own execs via a
# built-in default path, but that isn't in the *environment*, so
# gateway_init's subprocess daemons (spawned as `python3 ...`) would
# inherit no PATH and fail to find python3. Export it for all children.
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Direct upstream resolver (control-plane / gateway egress + buildah).
printf 'nameserver {_INFRA_RESOLVER}\\n' > /etc/resolv.conf 2>/dev/null
# Debug SSH: install the per-boot pubkey from the kernel cmdline.
KEY=$(sed -n 's/.*bb_pubkey=\\([^ ]*\\).*/\\1/p' /proc/cmdline | base64 -d 2>/dev/null)
if [ -n "$KEY" ]; then
mkdir -p /root/.ssh
printf '%s\\n' "$KEY" > /root/.ssh/authorized_keys
chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys
fi
chown -R 0:0 /root 2>/dev/null || true
mkdir -p /etc/dropbear /run /var/lib/bot-bottle
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted
# at the control plane's DB dir, so bot-bottle.db survives infra-VM restarts.
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
/bb-dropbear -R -E -p 22 &
# Control plane. Source is baked at /app; the package is stdlib-only.
cd /app
BOT_BOTTLE_ROOT=/var/lib/bot-bottle python3 -m bot_bottle.orchestrator \\
--host 0.0.0.0 --port {CONTROL_PLANE_PORT} --broker stub &
# Gateway data plane, multi-tenant: each request resolves source-IP ->
# policy against the local control plane. The VM backend reaches git over
# git-http (9420), so the git:// daemon (git-gate, needs a per-bottle
# entrypoint the consolidated model doesn't use) is left out.
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{CONTROL_PLANE_PORT} \\
SUPERVISE_DB_PATH=/var/lib/bot-bottle/db/bot-bottle.db \\
python3 /app/gateway_init.py &
# Reap as PID 1; children are backgrounded, so `wait` blocks.
while : ; do wait ; done
"""
+121 -265
View File
@@ -1,75 +1,64 @@
"""Launch flow for the Firecracker backend.
"""Launch flow for the Firecracker backend (PRD 0070, consolidated).
Per bottle:
1. mint the egress CA, build the agent image (docker), export it to a
cached ext4 rootfs;
2. claim a free TAP pool slot (rootless flock);
3. bring up the Docker sidecar bundle, publishing egress / git-gate /
supervise on the slot's host-side TAP IP at fixed ports;
4. boot the microVM on that TAP; wait for SSH;
5. provision (CA, prompt, skills, workspace, git, supervise) over SSH.
1. build the agent rootfs in a builder VM (buildah, no host docker), or
resume a frozen bottle from its committed rootfs tar; cache the ext4;
2. ensure the per-host orchestrator + shared gateway are up;
3. claim a free TAP pool slot (rootless flock);
4. register the bottle on the orchestrator by the VM's guest IP (the
attribution key) and provision its git-gate state into the gateway;
5. boot the microVM on that TAP; wait for SSH;
6. provision (shared gateway CA, prompt, skills, workspace, git, supervise)
over SSH.
The per-bottle Docker sidecar bundle is gone. The shared gateway handles
egress / git-gate / supervise for every VM; Docker's PREROUTING DNAT routes
the VMs' traffic to it, and the nft table's `ct status dnat accept` rule
in the forward chain lets it pass. The VM still sends to `host_tap_ip:PORT`
— the address its world is, by nft design, limited to.
Isolation is enforced by the operator-provisioned nft table (checked
fail-closed in preflight): a VM reaches only its sidecar (DNAT'd from
the host TAP IP) and nothing else. The agent's HTTPS_PROXY therefore
points at `http://<host_tap_ip>:9099`, its only route to the world.
fail-closed in preflight): a VM reaches only the sidecar (DNAT'd from
the host TAP IP) and nothing else.
"""
from __future__ import annotations
import dataclasses
import os
import subprocess
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
from ...bottle_state import (
committed_rootfs_path,
egress_state_dir,
git_gate_state_dir,
read_committed_image,
)
from ...egress import (
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_resolve_token_values,
egress_sidecar_env_entries,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...log import die, info, warn
from ...supervise import DB_PATH_IN_CONTAINER, SUPERVISE_PORT
from ...util import expand_tilde
from ..docker.egress import (
EGRESS_CA_IN_CONTAINER,
EGRESS_PORT,
egress_tls_init,
)
from ..docker.git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from ..docker.sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
)
from ...log import info, warn
from ...supervise import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, isolation_probe, netpool, util
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
from .consolidated_launch import (
launch_consolidated,
teardown_consolidated,
)
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
_GIT_HTTP_PORT = 9420
_GIT_GATE_READY_FILE = "/run/git-gate/ready"
def sidecar_container_name(slug: str) -> str:
return f"bot-bottle-sidecars-{slug}"
@contextmanager
@@ -78,6 +67,8 @@ def launch(
*,
provision: Callable[[FirecrackerBottlePlan, "FirecrackerBottle"], str | None],
) -> Generator[FirecrackerBottle, None, None]:
"""Build, launch, and provision a Firecracker bottle via the consolidated
orchestrator. Teardown on exit."""
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
@@ -94,31 +85,87 @@ def launch(
raise teardown_exc
try:
plan = _mint_certs(plan)
plan = _build_agent_image(plan)
# Step 1: agent rootfs. Built from the Dockerfile inside a Firecracker
# builder VM (buildah, no host docker); a committed snapshot is reused
# when present. Returns the base dir the per-bottle ext4 is made from.
plan, agent_base = _build_agent_base(plan)
# Claim a TAP slot; the flock is held until teardown closes it.
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any.
git_gate_plan = plan.git_gate_plan
if git_gate_plan.upstreams:
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, git_gate_plan, git_gate_state_dir(plan.slug),
)
# Step 3: claim a TAP slot; the flock is held until teardown.
slot, lock = netpool.allocate(plan.slug)
stack.callback(lock.close)
info(f"firecracker slot {slot.iface}: host={slot.host_ip} "
f"guest={slot.guest_ip}")
plan = _provision_git_gate_keys(plan)
# Step 4: register on the orchestrator + provision this bottle's
# git-gate state into the shared gateway. The per-bottle egress tokens
# are resolved from the host env now and handed to the orchestrator
# (in memory) for the gateway to inject — the agent never sees them.
# Attribution is by the VM's guest IP (unspoofable via /31 TAP + nft).
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
ctx = launch_consolidated(
plan.egress_plan, git_gate_plan,
guest_ip=slot.guest_ip,
image_ref=plan.image,
tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
)
sidecar_name = sidecar_container_name(plan.slug)
_force_remove_container(sidecar_name)
_start_sidecar_bundle(plan, sidecar_name, slot.host_ip)
stack.callback(_force_remove_container, sidecar_name)
_stage_git_gate(plan, sidecar_name)
# Step 5: install the SHARED gateway CA (replaces the per-bottle CA).
# Write it to a stable host path so the provisioner can copy it over SSH.
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(ctx.gateway_ca_pem)
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=ca_file,
mitmproxy_ca_cert_only_host_path=ca_file,
)
# Point the agent's git-gate insteadOf rewrites and supervise MCP URL
# at the shared gateway (reached at the slot's host TAP IP — the VM
# sends there and Docker DNAT routes to the gateway container).
git_gate_url = (
f"http://{slot.host_ip}:{_GIT_HTTP_PORT}" if git_gate_plan.upstreams else ""
)
supervise_url = (
f"http://{slot.host_ip}:{SUPERVISE_PORT}/"
if plan.supervise_plan is not None else ""
)
plan = dataclasses.replace(
plan,
git_gate_plan=git_gate_plan,
egress_plan=egress_plan,
identity_token=ctx.identity_token,
# Deliver the identity token as egress proxy credentials — clients
# honor `HTTPS_PROXY=http://id:token@gw` without app changes; the
# gateway reads Proxy-Authorization, validates the (source_ip,
# token) pair, and strips it before upstream.
agent_proxy_url=(
f"http://bottle:{ctx.identity_token}"
f"@{slot.host_ip}:{EGRESS_PORT}"
),
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
plan = _stamp_agent_urls(plan, slot.host_ip)
# Build the per-bottle rootfs + SSH key, then boot.
base_dir = util.build_base_rootfs_dir(plan.image)
# Step 6: build the per-bottle rootfs + SSH key, then boot.
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(base_dir, rootfs)
util.build_rootfs_ext4(agent_base, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
vm = firecracker_vm.boot(
@@ -133,8 +180,8 @@ def launch(
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
# Authoritative fail-closed egress-boundary check, before the
# agent runs: prove the VM cannot reach the host directly.
# Authoritative fail-closed egress-boundary check, before the agent
# runs: prove the VM cannot reach the host directly.
isolation_probe.verify_isolation(private_key, slot.guest_ip)
bottle = FirecrackerBottle(
@@ -159,169 +206,24 @@ def launch(
teardown()
def _mint_certs(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
egress_ca_host, egress_ca_cert_only = egress_tls_init(egress_state_dir(plan.slug))
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
)
return dataclasses.replace(plan, egress_plan=egress_plan)
def _build_agent_image(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
_docker_build(SIDECAR_BUNDLE_IMAGE, _REPO_DIR, dockerfile=SIDECAR_BUNDLE_DOCKERFILE)
committed = read_committed_image(plan.slug)
if committed and _image_exists(committed):
info(f"using committed image {committed!r}")
return dataclasses.replace(
plan,
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
)
_docker_build(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
return plan
def _provision_git_gate_keys(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
if not plan.git_gate_plan.upstreams:
return plan
git_gate_plan = provision_git_gate_dynamic_keys(
plan.manifest.bottle, plan.git_gate_plan, git_gate_state_dir(plan.slug),
)
return dataclasses.replace(plan, git_gate_plan=git_gate_plan)
def _stamp_agent_urls(
plan: FirecrackerBottlePlan, host_ip: str,
) -> FirecrackerBottlePlan:
proxy_url = f"http://{host_ip}:{EGRESS_PORT}"
supervise_url = (
f"http://{host_ip}:{SUPERVISE_PORT}/" if plan.supervise_plan is not None else ""
)
git_gate_url = (
f"http://{host_ip}:{_GIT_HTTP_PORT}" if plan.git_gate_plan.upstreams else ""
)
return dataclasses.replace(
plan,
agent_proxy_url=proxy_url,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
# --- sidecar bundle (Docker) ----------------------------------------
def _start_sidecar_bundle(
plan: FirecrackerBottlePlan, sidecar_name: str, host_ip: str,
) -> None:
argv = ["docker", "run", "--name", sidecar_name, "--detach", "--rm",
"-e", f"BOT_BOTTLE_SIDECAR_DAEMONS={','.join(_sidecar_daemons(plan))}"]
for entry in _sidecar_env_entries(plan):
argv += ["-e", entry]
for host_path, container_path, read_only in _sidecar_mounts(plan):
argv += ["-v", f"{host_path}:{container_path}{':ro' if read_only else ''}"]
# Publish on the slot's host TAP IP at fixed ports — each bottle
# has a distinct host_ip, so fixed ports never collide, and the VM
# reaches them at a stable, well-known address (its only route out).
for port in _sidecar_ports(plan):
argv += ["-p", f"{host_ip}:{port}:{port}"]
argv.append(SIDECAR_BUNDLE_IMAGE)
effective_env = {**dict(os.environ), **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
env = {**os.environ, **token_values}
info(f"docker run sidecar bundle {sidecar_name} (published on {host_ip})")
result = subprocess.run(argv, capture_output=True, text=True, env=env, check=False)
if result.returncode != 0:
die(f"docker run for sidecar bundle {sidecar_name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
def _sidecar_daemons(plan: FirecrackerBottlePlan) -> tuple[str, ...]:
daemons = ["egress"]
if plan.git_gate_plan.upstreams:
daemons += ["git-gate", "git-http"]
if plan.supervise_plan is not None:
daemons.append("supervise")
return tuple(daemons)
def _sidecar_ports(plan: FirecrackerBottlePlan) -> tuple[int, ...]:
ports = [EGRESS_PORT]
if plan.git_gate_plan.upstreams:
ports.append(_GIT_HTTP_PORT)
if plan.supervise_plan is not None:
ports.append(SUPERVISE_PORT)
return tuple(ports)
def _sidecar_env_entries(plan: FirecrackerBottlePlan) -> tuple[str, ...]:
env: list[str] = list(egress_sidecar_env_entries(plan.egress_plan))
if plan.git_gate_plan.upstreams:
env.append(f"BOT_BOTTLE_GIT_GATE_READY_FILE={_GIT_GATE_READY_FILE}")
if plan.supervise_plan is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
return tuple(env)
def _sidecar_mounts(
def _build_agent_base(
plan: FirecrackerBottlePlan,
) -> tuple[tuple[str, str, bool], ...]:
mounts: list[tuple[str, str, bool]] = []
ep = plan.egress_plan
mounts.append((str(ep.mitmproxy_ca_host_path.parent),
str(Path(EGRESS_CA_IN_CONTAINER).parent), False))
if ep.routes:
mounts.append((str(ep.routes_path.parent),
str(Path(EGRESS_ROUTES_IN_CONTAINER).parent), True))
sp = plan.supervise_plan
if sp is not None:
mounts.append((str(sp.db_path.parent),
str(Path(DB_PATH_IN_CONTAINER).parent), False))
return tuple(mounts)
def _stage_git_gate(plan: FirecrackerBottlePlan, sidecar_name: str) -> None:
gp = plan.git_gate_plan
if not gp.upstreams:
return
_docker_exec(sidecar_name, [
"mkdir", "-p",
str(Path(GIT_GATE_HOOK_IN_CONTAINER).parent),
GIT_GATE_CREDS_DIR_IN_CONTAINER, "/git",
str(Path(_GIT_GATE_READY_FILE).parent),
])
for host_path, container_path in _git_gate_files(plan):
_docker_cp(host_path, f"{sidecar_name}:{container_path}")
_docker_exec(sidecar_name, [
"sh", "-c",
f"chmod 755 {GIT_GATE_ENTRYPOINT_IN_CONTAINER} "
f"{GIT_GATE_HOOK_IN_CONTAINER} {GIT_GATE_ACCESS_HOOK_IN_CONTAINER} && "
f"chmod 600 {GIT_GATE_CREDS_DIR_IN_CONTAINER}/* && "
f"touch {_GIT_GATE_READY_FILE}",
])
def _git_gate_files(plan: FirecrackerBottlePlan) -> tuple[tuple[str, str], ...]:
gp = plan.git_gate_plan
files: list[tuple[str, str]] = [
(str(gp.entrypoint_script), GIT_GATE_ENTRYPOINT_IN_CONTAINER),
(str(gp.hook_script), GIT_GATE_HOOK_IN_CONTAINER),
(str(gp.access_hook_script), GIT_GATE_ACCESS_HOOK_IN_CONTAINER),
]
for upstream in gp.upstreams:
files.append((expand_tilde(upstream.identity_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-key"))
if upstream.known_hosts_file:
files.append((str(upstream.known_hosts_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-known_hosts"))
return tuple(files)
) -> tuple[FirecrackerBottlePlan, Path]:
"""Produce the agent's base rootfs dir. Primary path: build the Dockerfile
inside a Firecracker builder VM (buildah, no host docker), smoke-testing
the image before export. A committed snapshot (freeze/migrate) is resumed
directly from the rootfs tar the freezer wrote — no host docker either."""
committed = read_committed_image(plan.slug)
committed_tar = committed_rootfs_path(plan.slug)
if committed and committed_tar.is_file():
info(f"resuming from committed rootfs {committed_tar}")
return plan, util.build_committed_rootfs_dir(committed_tar)
base = image_builder.build_agent_rootfs_dir(
Path(plan.dockerfile_path),
image_tag=plan.image,
smoke_test=runtime_for(plan.agent_provider_template).smoke_test,
)
return plan, base
# --- agent guest env -------------------------------------------------
@@ -330,7 +232,8 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
"""Env injected into every agent/exec call over SSH. The VM has no
baked process env (it just runs init), so the proxy/CA/git/supervise
wiring is applied per-invocation."""
proxy_url = f"http://{host_ip}:{EGRESS_PORT}"
# Carries the identity token as proxy credentials (set in `launch`).
proxy_url = plan.agent_proxy_url or f"http://{host_ip}:{EGRESS_PORT}"
no_proxy = f"localhost,127.0.0.1,{host_ip}"
env: dict[str, str] = {
"HTTPS_PROXY": proxy_url, "HTTP_PROXY": proxy_url,
@@ -355,50 +258,3 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
if value is not None:
env[name] = value
return env
# --- docker helpers --------------------------------------------------
def _docker_build(ref: str, context: str, *, dockerfile: str = "") -> None:
info(f"docker build {ref}")
args = ["docker", "build", "-t", ref]
if dockerfile:
if not os.path.isabs(dockerfile):
dockerfile = os.path.join(context, dockerfile)
args += ["-f", dockerfile]
args.append(context)
result = subprocess.run(args, check=False)
if result.returncode != 0:
die(f"docker build for {ref!r} failed")
def _image_exists(ref: str) -> bool:
return subprocess.run(
["docker", "image", "inspect", ref],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
).returncode == 0
def _force_remove_container(name: str) -> None:
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
def _docker_exec(name: str, argv: list[str]) -> None:
result = subprocess.run(
["docker", "exec", name, *argv], capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"docker exec in {name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
def _docker_cp(host_path: str, dest: str) -> None:
result = subprocess.run(
["docker", "cp", host_path, dest], capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"docker cp {host_path} -> {dest} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}")
@@ -15,3 +15,11 @@ BOT_BOTTLE_FC_POOL_SIZE=8
BOT_BOTTLE_FC_IP_BASE=10.243.0.0
BOT_BOTTLE_FC_IFACE_PREFIX=bbfc
BOT_BOTTLE_FC_NFT_TABLE=bot_bottle_fc
# The orchestrator/gateway VM's own TAP — a dedicated link OUTSIDE the
# bbfc* agent pool. Unlike agent VMs (which reach only their gateway),
# the orchestrator is trusted infra that needs real NAT'd internet
# egress: to FROM-pull + apt/npm during in-VM agent-image builds
# (buildah) and to forward agent egress upstream (Stage B gateway). Its
# /31 is the top of the IP_BASE /16 (host x.y.255.0, guest x.y.255.1),
# clear of the pool near the bottom of the block.
BOT_BOTTLE_FC_ORCH_IFACE=bborch0
+28 -3
View File
@@ -18,7 +18,7 @@ Topology (per slot i):
* a /31 host<->guest link: host = base + 2i (the gateway the VM
routes through), guest = base + 2i + 1 (the VM's address).
* isolation via ``table inet bot_bottle_fc``: a VM reaches only its
own sidecar (DNAT'd from the host TAP IP) and nothing else.
own gateway (DNAT'd from the host TAP IP) and nothing else.
The default IP block is ``10.243.0.0/16`` — an intentionally obscure
corner of RFC-1918 private space. RFC-1918 is the range *designated*
@@ -79,6 +79,12 @@ def _cfg(key: str) -> str:
IFACE_PREFIX = _cfg("BOT_BOTTLE_FC_IFACE_PREFIX")
NFT_TABLE = _cfg("BOT_BOTTLE_FC_NFT_TABLE")
# The orchestrator/gateway VM's dedicated TAP — outside the bbfc* agent
# pool and, unlike it, NAT'd to the internet (see `orch_slot`). The
# orchestrator is trusted infra: it builds agent images in-VM (buildah
# needs to FROM-pull + apt/npm) and forwards agent egress upstream.
ORCH_IFACE = _cfg("BOT_BOTTLE_FC_ORCH_IFACE")
def pool_size() -> int:
return int(_cfg("BOT_BOTTLE_FC_POOL_SIZE"))
@@ -88,10 +94,10 @@ def ip_base() -> str:
return _cfg("BOT_BOTTLE_FC_IP_BASE")
# Sidecar ports the VM reaches at its host-side TAP IP. Kept in sync
# Gateway ports the VM reaches at its host-side TAP IP. Kept in sync
# with the backend constants (egress 9099, supervise 9100, git-http
# 9420); rendered into the setup output for operator visibility.
SIDECAR_PORTS = (9099, 9100, 9420)
GATEWAY_PORTS = (9099, 9100, 9420)
@dataclass(frozen=True)
@@ -123,6 +129,25 @@ def all_slots() -> list[Slot]:
return [slot(i) for i in range(pool_size())]
def orch_slot() -> Slot:
"""The orchestrator/gateway VM's dedicated link — its own TAP
(`ORCH_IFACE`) on a /31 at the TOP of the IP_BASE /16 (host
x.y.255.0, guest x.y.255.1), well clear of the agent pool near the
bottom of the block. Unlike a pool `Slot`, this link is NAT'd out to
the internet by the setup (the orchestrator is trusted infra), so it
is deliberately *not* one of the isolated `bbfc*` slots.
`index` is -1 (sentinel: not a pool index)."""
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
host = base16 + 0xFF00
return Slot(
index=-1,
iface=ORCH_IFACE,
host_ip=str(ipaddress.IPv4Address(host)),
guest_ip=str(ipaddress.IPv4Address(host + 1)),
)
# --- fail-closed verification ---------------------------------------
def _run_ok(argv: list[str]) -> bool:
@@ -0,0 +1,169 @@
"""Build the infra rootfs and publish it as a Gitea generic package.
The off-host (build / CI) half of PRD 0069 Stage 2: this DOES use Docker, but
never on the launch host. It runs the same pipeline the launch host used to run
locally — `docker build` the three fixed images, export to a rootfs dir, inject
the guest boot, `mke2fs` to an ext4 with the buildah build slack — then gzips
the ext4 and PUTs it (plus a `.sha256`) to
`…/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<version>/`.
The `<version>` is `infra_artifact.infra_artifact_version(...)`, the content
hash of the rootfs inputs, so a launch host at the same code checkout resolves
the exact artifact this produced.
python3 -m bot_bottle.backend.firecracker.publish_infra [--dry-run] [--force]
Auth: a token with `write:package` on the target owner, from
`BOT_BOTTLE_INFRA_ARTIFACT_TOKEN`.
"""
from __future__ import annotations
import argparse
import gzip
import hashlib
import shutil
import sys
import tempfile
import urllib.error
import urllib.request
from pathlib import Path
from . import infra_artifact, infra_vm, util
_CHUNK = 1 << 20
# A human-readable description shipped alongside the artifact — generic packages
# have no description field, so this file *is* the description on the package
# page. Uploaded on every publish so it never goes stale.
_ABOUT_NAME = "about.txt"
_ABOUT_TEXT = (
"bot-bottle infra rootfs for the Firecracker backend (PRD 0069 Stage 2, "
"#348): the per-host infra VM (orchestrator control plane + gateway + "
"buildah). Prebuilt off-host, gzip ext4; the launch host downloads + "
"sha256-verifies + boots it, no host Docker. The version tag is a content "
"hash of the rootfs inputs. Files: rootfs.ext4.gz + rootfs.ext4.gz.sha256.\n"
)
def _gzip(src: Path, dest: Path) -> None:
with open(src, "rb") as fh, gzip.open(dest, "wb") as out:
shutil.copyfileobj(fh, out, _CHUNK)
def _sha256(path: Path) -> str:
h = hashlib.sha256()
with open(path, "rb") as fh:
for chunk in iter(lambda: fh.read(_CHUNK), b""):
h.update(chunk)
return h.hexdigest()
def _put(url: str, body: "bytes | Path", token: str) -> None:
"""PUT `body` (raw bytes, or a Path streamed from disk) to `url`. The rootfs
is hundreds of MB, so it is passed as a Path and streamed — `urlopen` reads
the open file in blocks rather than materializing it in memory (with an
explicit Content-Length, which Gitea requires and which also stops urllib
from `len()`-ing a non-bytes body)."""
handle = None
if isinstance(body, Path):
length = body.stat().st_size
handle = open(body, "rb")
data: object = handle
else:
length = len(body)
data = body
req = urllib.request.Request(url, data=data, method="PUT") # type: ignore[arg-type]
req.add_header("Content-Length", str(length))
if token:
req.add_header("Authorization", f"token {token}")
req.add_header("Content-Type", "application/octet-stream")
try:
with urllib.request.urlopen(req) as resp:
print(f" uploaded {url} (HTTP {resp.status})")
except urllib.error.HTTPError as e:
if e.code == 409:
raise SystemExit(
f"artifact already published at {url} (HTTP 409); "
f"bump the code version or pass --force to overwrite"
)
raise SystemExit(f"upload failed (HTTP {e.code}): {url}\n{e.read().decode(errors='replace')}")
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
finally:
if handle is not None:
handle.close()
def _delete(url: str, token: str) -> None:
req = urllib.request.Request(url, method="DELETE")
if token:
req.add_header("Authorization", f"token {token}")
try:
with urllib.request.urlopen(req):
pass
except urllib.error.HTTPError as e:
if e.code != 404:
raise SystemExit(f"could not overwrite existing artifact (HTTP {e.code}): {url}")
except urllib.error.URLError as e:
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
def build_artifact(out_dir: Path) -> tuple[str, Path, Path]:
"""Build the infra rootfs ext4, gzip it, and write the checksum. Returns
`(version, gz_path, sha_path)`. Uses host Docker (off-host / CI)."""
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
print(f"building infra rootfs artifact {version} (docker)")
infra_vm.build_infra_images_with_docker()
base = infra_vm.build_infra_rootfs_dir()
ext4 = out_dir / "rootfs.ext4"
util.build_rootfs_ext4(base, ext4, slack_mib=8192)
gz = out_dir / "rootfs.ext4.gz"
print("compressing rootfs")
_gzip(ext4, gz)
ext4.unlink(missing_ok=True)
sha = out_dir / "rootfs.ext4.gz.sha256"
digest = _sha256(gz)
sha.write_text(f"{digest} rootfs.ext4.gz\n")
print(f" {gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
return version, gz, sha
def main(argv: list[str] | None = None) -> int:
parser = argparse.ArgumentParser(
prog="publish_infra", description="Build + publish the infra rootfs artifact.")
parser.add_argument("--dry-run", action="store_true",
help="build the artifact but do not upload")
parser.add_argument("--force", action="store_true",
help="overwrite an already-published artifact of this version")
args = parser.parse_args(argv)
_, _, token = infra_artifact._config()
if not args.dry_run and not token:
raise SystemExit(
"no publish token: set BOT_BOTTLE_INFRA_ARTIFACT_TOKEN to a token "
"with write:package")
with tempfile.TemporaryDirectory(prefix="bb-publish-infra.") as tmp:
version, gz, sha = build_artifact(Path(tmp))
gz_url = infra_artifact.artifact_url(version, gz.name)
sha_url = infra_artifact.artifact_url(version, sha.name)
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME)
if args.dry_run:
print(f"dry-run: would upload -> {gz_url}")
return 0
if args.force:
_delete(gz_url, token)
_delete(sha_url, token)
_delete(about_url, token)
_put(gz_url, gz, token) # streamed from disk (hundreds of MB)
_put(sha_url, sha.read_bytes(), token) # tiny, in-memory is fine
_put(about_url, _ABOUT_TEXT.encode(), token) # package description
print(f"published infra rootfs {version}")
return 0
if __name__ == "__main__":
sys.exit(main())
+87 -12
View File
@@ -14,6 +14,7 @@ and `./cli.py backend setup --backend=firecracker`.
from __future__ import annotations
import hashlib
import os
import platform
import shutil
@@ -159,15 +160,22 @@ def docker_image_id(ref: str) -> str:
return result.stdout.strip().replace("sha256:", "")[:16]
def build_base_rootfs_dir(image_ref: str) -> Path:
"""Export the agent image's filesystem and inject the guest init +
static dropbear. Cached by image digest the per-bottle bits
def build_base_rootfs_dir(
image_ref: str, *, variant: str = "", init_script: str | None = None,
) -> Path:
"""Export the image's filesystem and inject the guest init + static
dropbear. Cached by image digest the per-bottle bits
(authorized_keys, IP) are passed at boot via the kernel cmdline, so
this tree carries nothing bottle-specific and is safely shared.
`variant` suffixes the cache key so the same image can be prepared
with a different `init_script` (e.g. the infra VM boots the same
orchestrator image as the builder but runs the control plane as
PID 1, not the SSH-only agent init) without a cache collision.
Returns the prepared directory (read as the `mke2fs -d` source)."""
digest = docker_image_id(image_ref)
base = cache_dir() / "rootfs" / digest
base = cache_dir() / "rootfs" / f"{digest}{variant}"
ready = base / ".bb-ready"
if ready.is_file():
return base
@@ -200,18 +208,85 @@ def build_base_rootfs_dir(image_ref: str) -> Path:
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
)
_inject_guest_boot(base)
inject_guest_boot(base, init_script=init_script)
ready.write_text("ok\n")
return base
def _inject_guest_boot(rootfs: Path) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs."""
shutil.copy2(dropbear_path(), rootfs / "bb-dropbear")
os.chmod(rootfs / "bb-dropbear", 0o755)
init = rootfs / "bb-init"
init.write_text(_GUEST_INIT)
os.chmod(init, 0o755)
def build_committed_rootfs_dir(tar_path: Path) -> Path:
"""Prepare a base rootfs dir from a frozen-bottle snapshot tar (the
freeze/resume path no Docker). Extracts the snapshot, recreates the
virtual mount points the freezer excluded, and injects the guest init +
static dropbear, mirroring `build_base_rootfs_dir` but sourced from a tar
we control rather than a Docker image.
Cached under the rootfs cache, keyed by the tar's size+mtime so a
re-freeze re-extracts but repeated resumes of the same snapshot don't.
Returns the prepared directory (read as the `mke2fs -d` source)."""
st = tar_path.stat()
fingerprint = hashlib.sha256(
f"{tar_path}:{st.st_size}:{st.st_mtime_ns}".encode()
).hexdigest()[:16]
base = cache_dir() / "rootfs" / f"committed-{fingerprint}"
ready = base / ".bb-ready"
if ready.is_file():
return base
if base.exists():
shutil.rmtree(base, ignore_errors=True)
base.mkdir(parents=True)
info(f"extracting committed rootfs {tar_path} -> {base}")
result = subprocess.run(
["tar", "-x", "-f", str(tar_path), "-C", str(base)],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
die(f"extracting committed rootfs {tar_path} failed: "
f"{result.stderr.strip() or '<no stderr>'}")
# The freezer excludes the live/virtual filesystems from the snapshot;
# recreate them as empty mount points so the guest init can mount
# proc/sys/dev and dropbear has a writable /run.
for mount_point in ("proc", "sys", "dev", "run"):
(base / mount_point).mkdir(mode=0o755, exist_ok=True)
inject_guest_boot(base)
ready.write_text("ok\n")
return base
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
"""Drop the static dropbear and the PID-1 init into the rootfs.
`init_script` defaults to the SSH-only agent init; the infra VM
passes its own (control plane + gateway) init.
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
may already exist as symlinks aimed at a host file (e.g. bb-init ->
~/.bashrc). Replace whatever is there and create the files with
O_EXCL|O_NOFOLLOW so the write always lands a fresh regular file in
the staging tree and never follows a planted symlink out of it."""
_write_staged_file(rootfs / "bb-dropbear", dropbear_path().read_bytes())
_write_staged_file(rootfs / "bb-init", (init_script or _GUEST_INIT).encode())
def _write_staged_file(path: Path, data: bytes) -> None:
"""Write `data` to `path` (mode 0755) as a fresh regular file inside a
staging rootfs, replacing any pre-existing entry without following a
symlink at `path`. Fails closed on anything unexpected there."""
if path.is_symlink() or path.exists():
if path.is_dir() and not path.is_symlink():
shutil.rmtree(path)
else:
path.unlink()
fd = os.open(
path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o755
)
try:
os.write(fd, data)
finally:
os.close(fd)
os.chmod(path, 0o755)
def build_rootfs_ext4(base_dir: Path, out_path: Path, *, slack_mib: int = 1024) -> None:
@@ -2,7 +2,7 @@
Selectable via `BOT_BOTTLE_BACKEND=macos-container`. This package owns
the Apple `container` CLI integration; launch remains gated until the
sidecar network enforcement shape is implemented.
gateway network enforcement shape is implemented.
"""
from .backend import MacosContainerBottleBackend
@@ -89,6 +89,14 @@ class MacosContainerBottleBackend(
with _launch.launch(plan, provision=self.provision) as bottle:
yield bottle
def ensure_orchestrator(self) -> str:
"""Bring up the per-host infra container (control plane + gateway) and
return its control-plane URL the on-demand entry point operator tools
(`supervise`) call when no control plane is running yet. Mirrors
firecracker's infra-VM bring-up."""
from .infra import MacosInfraService
return MacosInfraService().ensure_running().control_plane_url
def prepare_cleanup(self) -> MacosContainerBottleCleanupPlan:
return _cleanup.prepare_cleanup()
+32 -4
View File
@@ -52,6 +52,7 @@ class MacosContainerBottle(Bottle):
terminal_title: str = "",
terminal_color: str = "",
agent_workdir: str = "/home/node",
exec_env: dict[str, str] | None = None,
):
self.name = container
self._teardown = teardown
@@ -62,6 +63,15 @@ class MacosContainerBottle(Bottle):
self.terminal_color = terminal_color
self.agent_provider_template = agent_provider_template
self.agent_workdir = agent_workdir
# Env applied to the agent process at `container exec` time, on top of
# what the container was run with. This is how the identity token
# reaches the agent (PRD 0070): registration mints it *after* the
# container exists — its source IP is the registration key and Apple
# Container assigns that by DHCP — so it cannot be in the run-time env
# the way docker's compose spec does it. `container exec --env` wins
# over the run-time value, so the token-bearing proxy URL set here
# supersedes the token-less one baked in at launch.
self._exec_env = dict(exec_env or {})
self._closed = False
def agent_argv(self, argv: list[str], *, tty: bool = True) -> list[str]:
@@ -74,6 +84,12 @@ class MacosContainerBottle(Bottle):
)
)
container_exec = ["container", "exec"]
# Bare env names, same rule as the terminal hints below: the value
# stays in the child env `exec_agent` builds and never reaches argv —
# the proxy URL here carries the identity token, which `ps` would
# otherwise expose to every process on the host.
for name in sorted(self._exec_env):
container_exec.extend(["--env", name])
if tty:
container_exec.extend(["--interactive", "--tty"])
# Forward terminal capability hints so TUIs can enable modified-key
@@ -94,21 +110,33 @@ class MacosContainerBottle(Bottle):
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
agent_argv = self.agent_argv(argv, tty=tty)
# The values behind the bare `--env` names in `agent_argv`. `sh -lc`
# below is in this process tree, so the child env reaches `container
# exec` either way.
env = {**os.environ, **self._exec_env} if self._exec_env else None
script = (
exec_shell_script(agent_argv, self.terminal_title, self.terminal_color)
if tty else None
)
if script is None:
return subprocess.run(agent_argv, check=False).returncode
return subprocess.run(["sh", "-lc", script], check=False).returncode
return subprocess.run(agent_argv, env=env, check=False).returncode
return subprocess.run(["sh", "-lc", script], env=env, check=False).returncode
def exec(self, script: str, *, user: str = "node") -> ExecResult:
# Carry the same exec env the agent gets: provisioning steps run
# through here, and a provider whose provision step fetches anything
# would egress without the identity token and be denied by /resolve.
# Bare `--env NAME` again, so the token stays off argv.
argv = ["container", "exec", "--user", user, "--interactive"]
for name in sorted(self._exec_env):
argv.extend(["--env", name])
argv.extend([self.name, "sh", "-s"])
result = subprocess.run(
["container", "exec", "--user", user, "--interactive",
self.name, "sh", "-s"],
argv,
input=script,
capture_output=True,
text=True,
env={**os.environ, **self._exec_env} if self._exec_env else None,
check=False,
)
return ExecResult(
@@ -13,9 +13,13 @@ from .. import BottlePlan
class MacosContainerBottlePlan(BottlePlan):
slug: str
forwarded_env: dict[str, str] = field(repr=False)
agent_proxy_url: str = ""
agent_git_gate_url: str = ""
agent_supervise_url: str = ""
# Read by provision-time consumers (git extraHeader, supervise MCP header)
# via getattr(plan, "identity_token", ""); stamped in launch after the
# bottle is registered. See launch.py's stamp for why it lives here and not
# only in the exec-time proxy env.
identity_token: str = ""
@property
def container_name(self) -> str:
@@ -9,7 +9,6 @@ from . import util as container_mod
from .bottle_cleanup_plan import MacosContainerBottleCleanupPlan
_PREFIX = "bot-bottle-"
_BUNDLE_PREFIX = "bot-bottle-sidecars-"
def _list_prefixed_containers() -> list[str]:
@@ -24,7 +23,7 @@ def _list_prefixed_containers() -> list[str]:
return []
return sorted(
name for name in (line.strip() for line in result.stdout.splitlines())
if name.startswith(_PREFIX) or name.startswith(_BUNDLE_PREFIX)
if name.startswith(_PREFIX)
)
@@ -0,0 +1,144 @@
"""Consolidated bottle launch sequence for the macOS backend (PRD 0070).
The docker backend allocates a free address, pins the agent to it with
`--ip`, registers it, *then* starts the agent registration precedes launch
because the pinned address is known up front.
**Apple Container 1.0.0 has no `--ip`.** The `--network` flag takes only
`<name>[,mac=][,mtu=]`; the address is assigned by vmnet's DHCP and is
knowable only once the container is running. So the macOS order inverts:
ensure_gateway() -> caller starts the agent -> register_agent(source_ip)
That is why this module exposes two functions where docker has one the
caller has to start the agent in between. `ensure_gateway` runs first because
the agent's proxy env needs the gateway's address at `container run` time; the
agent's *own* address (the attribution key) only exists afterwards.
The control plane and the gateway are one **infra container** here (see
`infra`), so `gateway_ip` and the control-plane host are the same address.
The consequence for the identity token: it is minted by registration, i.e.
*after* the agent container exists, so it cannot be baked into the run-time
env the way docker's compose spec does. It is delivered at `container exec`
time instead see `bottle.MacosContainerBottle`.
That delivery is load-bearing, not a nicety: `/resolve` requires a matching
`(source_ip, identity_token)` pair and fail-closes with no source-IP-only
fallback (#366). So egress that does not carry the token is denied — which is
the safe direction, and is why the agent's init process is a bare `sleep` and
every real command arrives through `container exec`.
"""
from __future__ import annotations
from dataclasses import dataclass
from ...egress import EgressPlan
from ...git_gate import GitGatePlan
from ...orchestrator.client import OrchestratorClient
from ...orchestrator.registration import registration_inputs
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
from .gateway import GATEWAY_NETWORK
from .gateway_provision import AppleGatewayTransport
from .infra import MacosInfraService, OrchestratorStartError
class ConsolidatedLaunchError(RuntimeError):
"""The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
class GatewayEndpoint:
"""What the agent `container run` needs to reach the shared gateway (the
infra container). `gateway_ip` is that container's host-only address, the
same host the control-plane URL points at."""
orchestrator_url: str
gateway_ip: str # the gateway's address — the agent's proxy target
gateway_ca_pem: str # the shared CA the provisioner installs
network: str # the shared host-only network to attach to
@dataclass(frozen=True)
class LaunchContext:
"""What the running agent needs once it has been registered."""
bottle_id: str
identity_token: str
source_ip: str # the agent's DHCP-assigned address (attribution key)
gateway_ip: str
network: str
orchestrator_url: str
def ensure_gateway(
*, service: MacosInfraService | None = None,
) -> GatewayEndpoint:
"""Ensure the per-host infra container (control plane + gateway) is up and
report how to reach it. Idempotent one singleton, so N bottle launches
share it. Call before starting the agent container: the agent's proxy env
needs `gateway_ip` at run time."""
service = service or MacosInfraService()
infra = service.ensure_running()
return GatewayEndpoint(
orchestrator_url=infra.control_plane_url,
gateway_ip=infra.gateway_ip,
gateway_ca_pem=service.ca_cert_pem(),
network=service.network,
)
def register_agent(
egress_plan: EgressPlan,
git_gate_plan: GitGatePlan,
*,
source_ip: str,
endpoint: GatewayEndpoint,
image_ref: str = "",
tokens: dict[str, str] | None = None,
) -> LaunchContext:
"""Register the (already running) agent by its address and provision its
git-gate state into the gateway. `source_ip` must be read from the live
container it is the attribution key the gateway resolves policy by.
Raises on failure; the caller tears down."""
client = OrchestratorClient(endpoint.orchestrator_url)
inputs = registration_inputs(egress_plan)
reg = client.register_bottle(
source_ip, image_ref=image_ref, policy=inputs.policy,
metadata=inputs.metadata, tokens=tokens,
)
try:
provision_git_gate(AppleGatewayTransport(), reg.bottle_id, git_gate_plan)
except Exception:
# Roll the registration back so a provisioning failure leaves no orphan.
client.teardown_bottle(reg.bottle_id)
raise
return LaunchContext(
bottle_id=reg.bottle_id,
identity_token=reg.identity_token,
source_ip=source_ip,
gateway_ip=endpoint.gateway_ip,
network=endpoint.network,
orchestrator_url=endpoint.orchestrator_url,
)
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
"""Deregister the bottle and remove its git-gate state from the gateway.
Both steps are idempotent so this is safe from a cleanup trap. Does NOT
stop the gateway it's a persistent per-host singleton."""
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
deprovision_git_gate(AppleGatewayTransport(), bottle_id)
__all__ = [
"GatewayEndpoint",
"LaunchContext",
"ensure_gateway",
"register_agent",
"teardown_consolidated",
"ConsolidatedLaunchError",
"OrchestratorStartError",
"GATEWAY_NETWORK",
]
@@ -1,36 +1,26 @@
"""Host-side egress apply for the macos-container backend.
"""Host-side egress route-apply for the macos-container backend.
Uses `container kill --signal HUP` (Apple Container framework) instead
of `docker kill` to signal the sidecar bundle.
The per-bottle companion container this used to signal (`container kill
--signal HUP <container>`) was removed in the companion-container removal
(#385). In the consolidated model the shared gateway resolves egress policy
per-request against the orchestrator rather than reloading a per-bottle routes
file, so the live per-bottle reload is not supported here and fails closed
until the gateway-side apply lands same posture as the docker backend.
"""
from __future__ import annotations
import os
import subprocess
from ...log import warn
from ..egress_apply import EgressApplicator, EgressApplyError
from .launch import sidecar_container_name
class MacOSContainerEgressApplicator(EgressApplicator):
def _signal_bundle_reload(self, slug: str) -> None:
container = sidecar_container_name(slug)
result = subprocess.run(
["container", "kill", "--signal", "HUP", container],
capture_output=True, text=True, check=False, env=os.environ,
del slug
raise EgressApplyError(
"live egress route-apply was removed with the per-bottle "
"companion container (#385); route changes will flow through "
"the consolidated gateway in a follow-up."
)
if result.returncode != 0:
last_error = (result.stderr or "").strip() or (result.stdout or "").strip()
warn(
f"egress: routes updated on disk for {slug}, but bundle reload failed: "
f"{last_error or 'container kill failed'}"
)
raise EgressApplyError(
f"could not reload egress bundle {container}: "
f"{last_error or 'container kill failed'}"
)
applicator = MacOSContainerEgressApplicator()
@@ -6,9 +6,13 @@ import subprocess
from ...bottle_state import read_metadata
from .. import ActiveAgent
from .infra import INFRA_NAME
_PREFIX = "bot-bottle-"
_SIDECAR_PREFIX = "bot-bottle-sidecars-"
# The shared per-host infra container carries the same prefix as agent
# containers but is infrastructure, not a bottle — one control plane + gateway
# serves every agent, so listing it as an agent would invent one per host.
_INFRA_NAMES = frozenset({INFRA_NAME})
def enumerate_active() -> list[ActiveAgent]:
@@ -22,9 +26,7 @@ def enumerate_active() -> list[ActiveAgent]:
return []
out: list[ActiveAgent] = []
for name in sorted(line.strip() for line in result.stdout.splitlines()):
if not name.startswith(_PREFIX):
continue
if name.startswith(_SIDECAR_PREFIX):
if not name.startswith(_PREFIX) or name in _INFRA_NAMES:
continue
slug = name[len(_PREFIX):]
metadata = read_metadata(slug)
@@ -0,0 +1,46 @@
"""Shared network/image constants for the macOS consolidated infra container.
The gateway data plane no longer runs as its own Apple container it shares a
single per-host **infra container** with the control plane (see `infra`),
because two Apple-Container guests writing one `bot-bottle.db` over virtiofs
would race incoherent `fcntl` locks. This module holds the pieces both the
infra service and the launch/provision glue need: the network names, the
gateway image, and the network-creation helper.
"""
from __future__ import annotations
import os
from ...orchestrator.gateway import GatewayError
from . import util as container_mod
# The shared host-only network the infra container and every agent bottle sit
# on. The agent's address here is the attribution key. Distinct from the docker
# names so both backends can coexist on one host.
GATEWAY_NETWORK = "bot-bottle-mac-gateway"
# The NAT network that gives the infra container (and only it) a route out.
GATEWAY_EGRESS_NETWORK = "bot-bottle-mac-egress"
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
DEFAULT_CA_TIMEOUT_SECONDS = 30.0
def ensure_networks(
network: str = GATEWAY_NETWORK, egress_network: str = GATEWAY_EGRESS_NETWORK,
) -> None:
"""Create the shared host-only network + the NAT egress network. Idempotent
`create_network` tolerates 'already exists'."""
container_mod.create_network(egress_network)
container_mod.create_network(network, internal=True)
__all__ = [
"GATEWAY_NETWORK",
"GATEWAY_EGRESS_NETWORK",
"GATEWAY_IMAGE",
"GatewayError",
"DEFAULT_CA_TIMEOUT_SECONDS",
"ensure_networks",
]
@@ -0,0 +1,44 @@
"""`GatewayTransport` for the Apple infra container (PRD 0070).
The provisioning *logic* (per-bottle creds dirs, namespaced repo init) is
backend-neutral and lives in `backend.docker.gateway_provision`; this is only
the transport how files and commands reach the running gateway. Docker uses
`docker exec`/`docker cp` and Firecracker uses SSH; Apple uses the `container`
CLI's equivalents against the infra container that hosts the gateway daemons.
"""
from __future__ import annotations
from ..docker.gateway_provision import GatewayProvisionError
from . import util as container_mod
from .infra import INFRA_NAME
class AppleGatewayTransport:
"""`GatewayTransport` for the gateway daemons in the Apple infra container."""
def __init__(self, gateway: str = INFRA_NAME) -> None:
self.gateway = gateway
def exec(self, argv: list[str]) -> None:
result = container_mod.run_container_argv(
["container", "exec", self.gateway, *argv]
)
if result.returncode != 0:
raise GatewayProvisionError(
f"gateway exec {argv!r} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def cp_into(self, src: str, dest: str) -> None:
result = container_mod.run_container_argv(
["container", "cp", src, f"{self.gateway}:{dest}"]
)
if result.returncode != 0:
raise GatewayProvisionError(
f"gateway cp {src} -> {dest} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
__all__ = ["AppleGatewayTransport", "GatewayProvisionError"]
+292
View File
@@ -0,0 +1,292 @@
"""The per-host infra container for the macOS backend (PRD 0070).
A single persistent Apple container that runs BOTH the orchestrator control
plane and the gateway data plane the macOS analogue of the Firecracker infra
VM (`backend/firecracker/infra_vm.py`), not the docker backend's two separate
containers.
Why one container, not two: Apple Containers are lightweight VMs, each with its
own kernel. The docker backend runs the orchestrator and gateway as two
containers safely because they share the host kernel, so their concurrent
writes to the one `bot-bottle.db` (the orchestrator's registry + the gateway
supervise daemon's queue) are serialized by coherent `fcntl` locks. Across two
*guest* kernels sharing a virtiofs-mounted DB those locks are not coherent, and
concurrent writers can corrupt the file. Firecracker solved this by putting
both services in one guest with the DB on a device only that guest mounts; this
does the same with Apple primitives.
Two consequences fall out of the single container, both simplifications:
- **No DNS dance.** The control plane and the gateway daemons reach each other
over `127.0.0.1`, so nothing depends on Apple's (absent) container DNS and
there is no orchestrator-before-gateway ordering to get right.
- **The DB is never host-shared.** It lives on a container-only volume, so no
host process opens the live file. The host CLI reaches registry + supervise
state through the control-plane HTTP surface (`cli/supervise.py` already uses
`OrchestratorClient`), exactly as it does for firecracker.
The control-plane source is bind-mounted (like the docker orchestrator), so a
code change takes effect on the next launch without an image rebuild; the
gateway daemons are baked in the gateway image and rebuild through its own
digest check.
"""
from __future__ import annotations
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from pathlib import Path
from ... import log
from ...orchestrator.gateway import GATEWAY_CA_CERT
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
OrchestratorStartError,
source_hash,
)
from ...paths import HOST_DB_FILENAME
from . import util as container_mod
from .gateway import (
DEFAULT_CA_TIMEOUT_SECONDS,
GATEWAY_EGRESS_NETWORK,
GATEWAY_IMAGE,
GATEWAY_NETWORK,
GatewayError,
ensure_networks,
)
# The one per-host infra container: control plane + gateway data plane.
INFRA_NAME = "bot-bottle-mac-infra"
INFRA_LABEL = "bot-bottle-mac-infra=1"
# Container-only volume holding bot-bottle.db. No host bind-mount, so the DB is
# written by exactly one kernel (this container's). Survives recreation.
INFRA_DB_VOLUME = "bot-bottle-mac-db"
# BOT_BOTTLE_ROOT inside the container; host_db_path() resolves the DB to
# <root>/db/<filename> and the supervise daemon writes the same file.
_DB_ROOT_IN_CONTAINER = "/var/lib/bot-bottle"
_DB_PATH_IN_CONTAINER = f"{_DB_ROOT_IN_CONTAINER}/db/{HOST_DB_FILENAME}"
_SRC_IN_CONTAINER = "/bot-bottle-src"
_REPO_ROOT = Path(__file__).resolve().parents[3]
_HEALTH_POLL_SECONDS = 0.25
_HEALTH_REQUEST_TIMEOUT_SECONDS = 1.0
_CA_POLL_SECONDS = 0.5
# The gateway subset the consolidated model runs (no per-bottle git:// daemon).
_GATEWAY_DAEMONS = "egress,git-http,supervise"
def _init_script(port: int) -> str:
"""PID-1 init: start the control plane and the gateway daemons, both in
this container, reaching each other over loopback. Backgrounded so `wait`
reaps as PID 1. No `set -e` a transient daemon failure must not kill the
whole container (gateway_init applies the same 'stay up' policy)."""
return (
"export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n"
f"mkdir -p $(dirname {_DB_PATH_IN_CONTAINER})\n"
# Control plane, from the bind-mounted source (stdlib-only package).
f"( cd {_SRC_IN_CONTAINER} && BOT_BOTTLE_ROOT={_DB_ROOT_IN_CONTAINER} "
f"python3 -m bot_bottle.orchestrator --host 0.0.0.0 --port {port} "
"--broker stub ) &\n"
# Gateway data plane, multi-tenant against the local control plane.
f"( cd /app && BOT_BOTTLE_GATEWAY_DAEMONS={_GATEWAY_DAEMONS} "
f"BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{port} "
f"SUPERVISE_DB_PATH={_DB_PATH_IN_CONTAINER} python3 /app/gateway_init.py ) &\n"
"while : ; do wait ; done\n"
)
@dataclass(frozen=True)
class InfraEndpoint:
"""How to reach the running infra container. The control plane and the
gateway are the same container, so one address serves both."""
control_plane_url: str # http://<infra ip>:8099 — host CLI + registration
gateway_ip: str # same container; agents' proxy / git-http / MCP target
class MacosInfraService:
"""Manages the single per-host infra container. Callers use
`ensure_running()` (returns the endpoint) and `ca_cert_pem()`."""
def __init__(
self,
*,
port: int = DEFAULT_PORT,
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
image: str = GATEWAY_IMAGE,
repo_root: Path = _REPO_ROOT,
name: str = INFRA_NAME,
db_volume: str = INFRA_DB_VOLUME,
) -> None:
self.port = port
self.network = network
self.egress_network = egress_network
self.image = image
self._repo_root = repo_root
self._name = name
self._db_volume = db_volume
def _resolve_url(self) -> str:
"""The control-plane URL, or "" while the container has no address."""
ip = container_mod.try_container_ipv4_on_network(self._name, self.network)
return f"http://{ip}:{self.port}" if ip else ""
def is_healthy(
self, url: str, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS,
) -> bool:
if not url:
return False
try:
with urllib.request.urlopen(f"{url}/health", timeout=timeout) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
def _source_current(self, current_hash: str) -> bool:
"""True iff the running infra container was created from the current
bind-mounted control-plane source. The control-plane process loads that
code at startup and won't reload it, so a stale container keeps serving
OLD code."""
if not container_mod.container_is_running(self._name):
return False
env = container_mod.container_env(self._name)
if not env:
return True # can't compare → don't churn a working container
return env.get("BOT_BOTTLE_SOURCE_HASH") == current_hash
def _running_healthy_endpoint(self, current_hash: str) -> InfraEndpoint | None:
"""The endpoint if the running container is BOTH source-current and
answering /health, else None ( recreate). Health, not just the source
label, is what lets a wedged-but-current container self-heal instead of
being polled to death forever."""
if not self._source_current(current_hash):
return None
url = self._resolve_url()
if url and self.is_healthy(url):
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
return None
def ensure_built(self) -> None:
"""Ensure the gateway data-plane image exists. The control-plane source
is bind-mounted, not baked, so only the gateway image needs building."""
container_mod.build_image(
self.image, str(self._repo_root), dockerfile="Dockerfile.gateway",
)
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> InfraEndpoint:
"""Ensure the single infra container is up; return how to reach it.
Idempotent per-host singleton a healthy container on current source
is left untouched, so N launches share the one control plane + gateway.
Raises `OrchestratorStartError` on startup timeout."""
current_hash = source_hash(self._repo_root)
endpoint = self._running_healthy_endpoint(current_hash)
if endpoint is not None:
return endpoint
self.ensure_built()
log.info("starting infra container", context={"name": self._name})
self._run_container(current_hash)
return self._wait_healthy(startup_timeout)
def _run_container(self, current_hash: str) -> None:
ensure_networks(self.network, self.egress_network)
container_mod.force_remove_container(self._name)
argv = [
"container", "run", "--detach",
"--name", self._name,
"--label", "bot-bottle.backend=macos-container",
"--label", INFRA_LABEL,
# NAT network FIRST so the gateway's egress has a default route;
# the host-only network is where agents (and the host CLI) reach it.
"--network", self.egress_network,
"--network", self.network,
"--dns", container_mod.dns_server(),
# Container-only DB volume: one kernel writes bot-bottle.db, never
# shared with the host or another guest.
"--volume", f"{self._db_volume}:{_DB_ROOT_IN_CONTAINER}",
# Bind-mount the control-plane source (read-only); a code change
# takes effect on relaunch with no image rebuild.
"--mount",
container_mod.bind_mount_spec(
str(self._repo_root), _SRC_IN_CONTAINER, readonly=True),
# Baked onto the container so `_source_current` can detect a real
# control-plane code change and recreate.
"--env", f"BOT_BOTTLE_SOURCE_HASH={current_hash}",
"--entrypoint", "sh",
self.image,
"-c", _init_script(self.port),
]
result = container_mod.run_container_argv(argv)
if result.returncode != 0:
raise OrchestratorStartError(
f"infra container failed to start: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _wait_healthy(self, startup_timeout: float) -> InfraEndpoint:
deadline = time.monotonic() + startup_timeout
while True:
url = self._resolve_url()
if url and self.is_healthy(url):
log.info("infra container healthy", context={"url": url})
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
if time.monotonic() >= deadline:
raise OrchestratorStartError(
f"infra container did not become healthy within "
f"{startup_timeout:g}s"
)
time.sleep(_HEALTH_POLL_SECONDS)
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
interception. Read out of the container (the CA lives on a
container-internal path, not a host mount); polls because mitmproxy
writes it a beat after start."""
deadline = time.monotonic() + timeout
while True:
result = container_mod.run_container_argv(
["container", "exec", self._name, "cat", GATEWAY_CA_CERT])
if result.returncode == 0 and result.stdout.strip():
return result.stdout
if time.monotonic() >= deadline:
raise GatewayError(
f"gateway CA not available in {self._name} after {timeout:g}s: "
f"{(result.stderr or '').strip() or 'empty'}"
)
time.sleep(_CA_POLL_SECONDS)
def stop(self) -> None:
"""Remove the infra container (idempotent). The DB volume persists."""
container_mod.force_remove_container(self._name)
def _ip_of(url: str) -> str:
"""The host from an http://host:port URL."""
return url.split("://", 1)[-1].rsplit(":", 1)[0]
def probe_control_plane_url(port: int = DEFAULT_PORT) -> str:
"""The running infra container's control-plane URL, or "" if it isn't up.
Used by host-side control-plane discovery (`discover_orchestrator_url`);
safe to call on any host returns "" when the container or the `container`
CLI isn't present."""
ip = container_mod.try_container_ipv4_on_network(INFRA_NAME, GATEWAY_NETWORK)
return f"http://{ip}:{port}" if ip else ""
__all__ = [
"MacosInfraService",
"InfraEndpoint",
"OrchestratorStartError",
"GatewayError",
"INFRA_NAME",
"INFRA_DB_VOLUME",
]
+198 -306
View File
@@ -1,11 +1,33 @@
"""Launch flow for the macOS Apple Container backend.
"""Launch flow for the macOS Apple Container backend (PRD 0070).
This backend keeps the explicit proxy-env enforcement model for v1:
the agent container is attached only to a host-only Apple Container
network, while the sidecar bundle is attached to a NAT network first
and the host-only network second. The sidecar's host-only IP is
discovered from `container inspect` and stamped into the agent's
HTTP_PROXY / HTTPS_PROXY env vars.
The agent container attaches to the **shared host-only gateway network** and
proxies egress through the one per-host gateway, replacing the per-bottle
companion container removed in #385.
The order differs from docker's, forced by Apple Container 1.0.0 having no
`--ip` (see `consolidated_launch`): the agent is started *before* it is
registered, because its DHCP-assigned address the attribution key does not
exist until then.
gateway up -> run agent -> read its IP -> register it -> provision
Two things follow from that inversion:
- The **identity token** is minted by registration and so cannot be in the
agent's run-time env; it rides the proxy URL applied at `container exec`
time (`bottle.MacosContainerBottle`). `/resolve` requires it (#366), so
egress without it is denied hence the bare `sleep` init: every real agent
command goes through exec and therefore carries the token.
- The agent is run with `--cap-drop CAP_NET_RAW`. Apple Container grants
NET_RAW by default, which would let an agent open a raw socket and forge a
neighbour's source address on the shared segment. NET_ADMIN is already
absent (the agent cannot change its own address or route), so dropping
NET_RAW is what closes the source-address half of PRD 0070's invariant:
"a packet's source address, as seen by the orchestrator, provably identifies
the originating bottle." The identity token is the other half — an attacker
would need to forge the address *and* steal the token but the invariant is
a stated precondition of consolidation, so it is enforced on its own terms
rather than left to the token.
"""
from __future__ import annotations
@@ -23,52 +45,30 @@ from ...bottle_state import (
read_committed_image,
)
from ...egress import (
EGRESS_ROUTES_IN_CONTAINER,
egress_agent_env_entries,
egress_resolve_token_values,
egress_sidecar_env_entries,
)
from ...git_gate import (
provision_git_gate_dynamic_keys,
revoke_git_gate_provisioned_keys,
)
from ...git_http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
from ...log import die, info, warn
from ...supervise import DB_PATH_IN_CONTAINER, SUPERVISE_PORT
from ...util import expand_tilde
from ..docker.egress import EGRESS_CA_IN_CONTAINER, EGRESS_PORT
from ..docker.git_gate import (
GIT_GATE_ACCESS_HOOK_IN_CONTAINER,
GIT_GATE_CREDS_DIR_IN_CONTAINER,
GIT_GATE_ENTRYPOINT_IN_CONTAINER,
GIT_GATE_HOOK_IN_CONTAINER,
)
from ..docker.sidecar_bundle import (
SIDECAR_BUNDLE_DOCKERFILE,
SIDECAR_BUNDLE_IMAGE,
)
from ..docker.egress import egress_tls_init
from ...supervise import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import util as container_mod
from .bottle import MacosContainerBottle
from .bottle_plan import MacosContainerBottlePlan
from .consolidated_launch import (
GatewayEndpoint,
ensure_gateway,
register_agent,
teardown_consolidated,
)
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
_AGENT_SLEEP_SECONDS = "2147483647"
_GIT_HTTP_PORT = 9420
_GIT_GATE_READY_FILE = "/run/git-gate/ready"
def internal_network_name(slug: str) -> str:
return f"bot-bottle-net-{slug}"
def egress_network_name(slug: str) -> str:
return f"bot-bottle-egress-{slug}"
def sidecar_container_name(slug: str) -> str:
return f"bot-bottle-sidecars-{slug}"
@contextmanager
@@ -77,7 +77,8 @@ def launch(
*,
provision: Callable[[MacosContainerBottlePlan, "MacosContainerBottle"], str | None],
) -> Generator[MacosContainerBottle, None, None]:
"""Build, run, provision, and yield an Apple Container bottle."""
"""Build, run, register, provision, and yield an Apple Container bottle on
the shared per-host gateway."""
stack = ExitStack()
bottle_for_revoke = plan.manifest.bottle
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
@@ -94,30 +95,66 @@ def launch(
raise teardown_exc
try:
plan = _mint_certs(plan)
plan = _build_images(plan)
internal_network = internal_network_name(plan.slug)
egress_network = egress_network_name(plan.slug)
_create_networks(internal_network, egress_network, stack)
# Step 1: the per-host singletons. Must precede the agent run — its
# proxy env needs the gateway's address at `container run` time.
endpoint = ensure_gateway()
# Step 2: mint this bottle's deploy keys, then point it at the SHARED
# gateway's CA + git-http/supervise ports.
plan = _provision_git_gate_keys(plan)
plan = _install_gateway_ca(plan, endpoint)
plan = _stamp_agent_urls(plan, endpoint)
sidecar_name = sidecar_container_name(plan.slug)
container_mod.force_remove_container(sidecar_name)
_start_sidecar_bundle(plan, sidecar_name, internal_network, egress_network)
stack.callback(container_mod.force_remove_container, sidecar_name)
_stage_git_gate(plan, sidecar_name)
sidecar_ip = container_mod.container_ipv4_on_network(
sidecar_name, internal_network,
)
plan = _stamp_agent_urls(plan, sidecar_ip)
# Step 3: run the agent. It has no identity token yet — registration
# needs the address this run assigns.
container_mod.force_remove_container(plan.container_name)
_start_agent(plan, internal_network, sidecar_ip)
_start_agent(plan, endpoint)
stack.callback(container_mod.force_remove_container, plan.container_name)
# Step 4: read the assigned address and register by it. This is the
# attribution key; `--cap-drop CAP_NET_RAW` at run is what makes it
# unforgeable. Poll: `container run --detach` can return before vmnet's
# DHCP has assigned the address.
source_ip = container_mod.wait_container_ipv4_on_network(
plan.container_name, endpoint.network,
)
if not source_ip:
die(
f"agent {plan.container_name} never got an address on "
f"{endpoint.network}"
)
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
ctx = register_agent(
plan.egress_plan,
plan.git_gate_plan,
source_ip=source_ip,
endpoint=endpoint,
image_ref=plan.image,
tokens=token_values,
)
stack.callback(
teardown_consolidated, ctx.bottle_id,
orchestrator_url=ctx.orchestrator_url,
)
info(
f"agent {plan.container_name} registered "
f"(gateway {endpoint.gateway_ip}, ip {source_ip})"
)
# Stamp the token onto the plan so provision-time consumers can read it,
# not only the exec-time egress proxy. git-gate's gitconfig extraHeader
# and the supervise MCP --header both reach the gateway on NO_PROXY (they
# bypass the egress proxy that carries the token), so without this the
# gateway's /resolve fail-closes and every git fetch/push and supervise
# call from the bottle is denied. Registration already produced the
# token above, so — unlike the run-time env — the plan CAN carry it.
plan = dataclasses.replace(plan, identity_token=ctx.identity_token)
bottle = MacosContainerBottle(
plan.container_name,
teardown,
@@ -125,9 +162,13 @@ def launch(
agent_command=plan.agent_command,
agent_prompt_mode=plan.agent_prompt_mode,
agent_provider_template=plan.agent_provider_template,
terminal_title=f"{plan.spec.label} ({plan.spec.agent_name})" if plan.spec.label else plan.spec.agent_name,
terminal_title=(
f"{plan.spec.label} ({plan.spec.agent_name})"
if plan.spec.label else plan.spec.agent_name
),
terminal_color=plan.spec.color,
agent_workdir=plan.workspace_plan.workdir,
exec_env=_identity_proxy_env(endpoint, ctx.identity_token),
)
bottle.prompt_path = provision(plan, bottle)
@@ -136,116 +177,24 @@ def launch(
teardown()
def _mint_certs(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
egress_ca_host, egress_ca_cert_only = egress_tls_init(
egress_state_dir(plan.slug),
)
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=egress_ca_host,
mitmproxy_ca_cert_only_host_path=egress_ca_cert_only,
)
return dataclasses.replace(plan, egress_plan=egress_plan)
def _build_images(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
container_mod.build_image(
SIDECAR_BUNDLE_IMAGE,
_REPO_DIR,
dockerfile=SIDECAR_BUNDLE_DOCKERFILE,
)
"""Build the agent image. The gateway's own image is built by
`ensure_gateway` it belongs to the shared singleton, not to a bottle."""
committed = read_committed_image(plan.slug)
if committed and container_mod.image_exists(committed):
info(f"using committed image {committed!r}")
return dataclasses.replace(
plan,
agent_provision=dataclasses.replace(
plan.agent_provision,
image=committed,
plan.agent_provision, image=committed,
),
)
container_mod.build_image(
plan.image,
_REPO_DIR,
dockerfile=plan.dockerfile_path,
plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path,
)
return plan
def _create_networks(
internal_network: str,
egress_network: str,
stack: ExitStack,
) -> None:
container_mod.create_network(internal_network, internal=True)
stack.callback(container_mod.remove_network, internal_network)
container_mod.create_network(egress_network)
stack.callback(container_mod.remove_network, egress_network)
def _start_sidecar_bundle(
plan: MacosContainerBottlePlan,
sidecar_name: str,
internal_network: str,
egress_network: str,
) -> None:
argv = _sidecar_run_argv(plan, sidecar_name, internal_network, egress_network)
effective_env = {**dict(os.environ), **plan.agent_provision.provisioned_env}
token_values = egress_resolve_token_values(
plan.egress_plan.token_env_map, effective_env,
)
env = {**os.environ, **token_values}
info(f"container run sidecar bundle {sidecar_name}")
result = subprocess.run(
argv, capture_output=True, text=True, env=env, check=False,
)
if result.returncode != 0:
die(
f"container run for sidecar bundle {sidecar_name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _start_agent(
plan: MacosContainerBottlePlan,
internal_network: str,
sidecar_ip: str,
) -> None:
argv = _agent_run_argv(plan, internal_network, sidecar_ip)
env = {
**os.environ,
**plan.forwarded_env,
}
info(f"container run agent {plan.container_name}")
result = subprocess.run(
argv, capture_output=True, text=True, env=env, check=False,
)
if result.returncode != 0:
die(
f"container run for agent {plan.container_name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
def _stamp_agent_urls(
plan: MacosContainerBottlePlan,
sidecar_ip: str,
) -> MacosContainerBottlePlan:
proxy_url = f"http://{sidecar_ip}:{EGRESS_PORT}"
supervise_url = ""
if plan.supervise_plan is not None:
supervise_url = f"http://{sidecar_ip}:{SUPERVISE_PORT}/"
git_gate_url = ""
if plan.git_gate_plan.upstreams:
git_gate_url = f"http://{sidecar_ip}:{_GIT_HTTP_PORT}"
return dataclasses.replace(
plan,
agent_proxy_url=proxy_url,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
def _provision_git_gate_keys(
plan: MacosContainerBottlePlan,
) -> MacosContainerBottlePlan:
@@ -259,177 +208,120 @@ def _provision_git_gate_keys(
return dataclasses.replace(plan, git_gate_plan=git_gate_plan)
def _stage_git_gate(plan: MacosContainerBottlePlan, sidecar_name: str) -> None:
gp = plan.git_gate_plan
if not gp.upstreams:
return
def _install_gateway_ca(
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
) -> MacosContainerBottlePlan:
"""Stage the SHARED gateway's CA for the provisioner to install, replacing
the per-bottle CA the companion container used to mint. Every bottle on
this host trusts this one CA."""
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
ca_dir.mkdir(parents=True, exist_ok=True)
ca_file = ca_dir / "gateway-ca.pem"
ca_file.write_text(endpoint.gateway_ca_pem)
egress_plan = dataclasses.replace(
plan.egress_plan,
mitmproxy_ca_host_path=ca_file,
mitmproxy_ca_cert_only_host_path=ca_file,
)
return dataclasses.replace(plan, egress_plan=egress_plan)
container_mod.exec_container(
sidecar_name,
[
"mkdir",
"-p",
str(Path(GIT_GATE_HOOK_IN_CONTAINER).parent),
GIT_GATE_CREDS_DIR_IN_CONTAINER,
"/git",
str(Path(_GIT_GATE_READY_FILE).parent),
],
def _stamp_agent_urls(
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
) -> MacosContainerBottlePlan:
"""Point the agent's git-gate insteadOf rewrites + supervise MCP at the
shared gateway's ports. Both bypass the egress proxy (NO_PROXY covers the
gateway address)."""
git_gate_url = (
f"http://{endpoint.gateway_ip}:{_GIT_HTTP_PORT}"
if plan.git_gate_plan.upstreams else ""
)
supervise_url = (
f"http://{endpoint.gateway_ip}:{SUPERVISE_PORT}/"
if plan.supervise_plan is not None else ""
)
return dataclasses.replace(
plan,
agent_git_gate_url=git_gate_url,
agent_supervise_url=supervise_url,
)
for host_path, container_path in _git_gate_files(plan):
container_mod.copy_into_container(
sidecar_name, host_path, container_path,
def _proxy_url(gateway_ip: str, identity_token: str = "") -> str:
"""The agent's egress proxy URL. The identity token rides as proxy
credentials the gateway reads Proxy-Authorization, resolves the
(source_ip, token) pair against the control plane, and strips it before
upstream. Without a valid pair `/resolve` denies the request (#366)."""
cred = f"bottle:{identity_token}@" if identity_token else ""
return f"http://{cred}{gateway_ip}:{EGRESS_PORT}"
def _no_proxy(gateway_ip: str) -> str:
# git-http + supervise live on the gateway and must NOT go through the
# egress proxy — the agent reaches them directly by its address.
return f"localhost,127.0.0.1,{gateway_ip}"
def _identity_proxy_env(
endpoint: GatewayEndpoint, identity_token: str,
) -> dict[str, str]:
"""The token-bearing proxy env applied at `container exec`. It supersedes
the token-less run-time value (exec `--env` wins), which is the only way to
get the token in: it does not exist until after the container runs."""
if not identity_token:
return {}
url = _proxy_url(endpoint.gateway_ip, identity_token)
return {
"HTTPS_PROXY": url, "HTTP_PROXY": url,
"https_proxy": url, "http_proxy": url,
}
def _start_agent(plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint) -> None:
argv = _agent_run_argv(plan, endpoint)
env = {**os.environ, **plan.forwarded_env}
info(f"container run agent {plan.container_name}")
result = subprocess.run(
argv, capture_output=True, text=True, env=env, check=False,
)
if result.returncode != 0:
die(
f"container run for agent {plan.container_name} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
container_mod.exec_container(
sidecar_name,
[
"sh",
"-c",
"chmod 755 "
f"{GIT_GATE_ENTRYPOINT_IN_CONTAINER} "
f"{GIT_GATE_HOOK_IN_CONTAINER} "
f"{GIT_GATE_ACCESS_HOOK_IN_CONTAINER} && "
f"chmod 600 {GIT_GATE_CREDS_DIR_IN_CONTAINER}/* && "
f"touch {_GIT_GATE_READY_FILE}",
],
)
def _git_gate_files(
plan: MacosContainerBottlePlan,
) -> tuple[tuple[str, str], ...]:
gp = plan.git_gate_plan
files: list[tuple[str, str]] = [
(str(gp.entrypoint_script), GIT_GATE_ENTRYPOINT_IN_CONTAINER),
(str(gp.hook_script), GIT_GATE_HOOK_IN_CONTAINER),
(str(gp.access_hook_script), GIT_GATE_ACCESS_HOOK_IN_CONTAINER),
]
for upstream in gp.upstreams:
files.append((
expand_tilde(upstream.identity_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-key",
))
if upstream.known_hosts_file:
files.append((
str(upstream.known_hosts_file),
f"{GIT_GATE_CREDS_DIR_IN_CONTAINER}/{upstream.name}-known_hosts",
))
return tuple(files)
def _sidecar_run_argv(
plan: MacosContainerBottlePlan,
sidecar_name: str,
internal_network: str,
egress_network: str,
) -> list[str]:
argv = [
"container", "run",
"--name", sidecar_name,
"--detach",
"--rm",
"--network", egress_network,
"--network", internal_network,
"--dns", _sidecar_dns(),
"--env", f"BOT_BOTTLE_SIDECAR_DAEMONS={','.join(_sidecar_daemons(plan))}",
]
for entry in _sidecar_env_entries(plan):
argv += ["--env", entry]
for host_path, container_path, read_only in _sidecar_mounts(plan):
argv += ["--mount", _mount_spec(host_path, container_path, read_only)]
argv.append(SIDECAR_BUNDLE_IMAGE)
return argv
def _agent_run_argv(
plan: MacosContainerBottlePlan,
internal_network: str,
sidecar_ip: str,
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
) -> list[str]:
argv = [
"container", "run",
"--name", plan.container_name,
"--detach",
"--network", internal_network,
"--label", "bot-bottle.backend=macos-container",
"--network", endpoint.network,
# The attribution invariant: without NET_RAW the agent cannot open a
# raw socket, so it cannot source-IP-spoof its neighbours on the shared
# segment. NET_ADMIN is not granted by default, so its address and
# route are already fixed. See the module docstring.
"--cap-drop", "CAP_NET_RAW",
]
for entry in _agent_env_entries(plan, sidecar_ip):
for entry in _agent_env_entries(plan, endpoint):
argv += ["--env", entry]
# The init process is a no-op: every agent command arrives via
# `container exec`, which is also how the identity token gets in.
argv += [plan.image, "sleep", _AGENT_SLEEP_SECONDS]
return argv
def _sidecar_dns() -> str:
return container_mod.dns_server()
def _sidecar_daemons(plan: MacosContainerBottlePlan) -> tuple[str, ...]:
daemons = ["egress"]
if plan.git_gate_plan.upstreams:
daemons += ["git-gate", "git-http"]
if plan.supervise_plan is not None:
daemons.append("supervise")
return tuple(daemons)
def _sidecar_env_entries(plan: MacosContainerBottlePlan) -> tuple[str, ...]:
env: list[str] = list(egress_sidecar_env_entries(plan.egress_plan))
if plan.git_gate_plan.upstreams:
env.append(f"BOT_BOTTLE_GIT_GATE_READY_FILE={_GIT_GATE_READY_FILE}")
if plan.supervise_plan is not None:
env += [
f"SUPERVISE_BOTTLE_SLUG={plan.slug}",
f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
f"SUPERVISE_PORT={SUPERVISE_PORT}",
]
return tuple(env)
def _sidecar_mounts(
plan: MacosContainerBottlePlan,
) -> tuple[tuple[str, str, bool], ...]:
mounts: list[tuple[str, str, bool]] = []
ep = plan.egress_plan
mounts.append((
str(ep.mitmproxy_ca_host_path.parent),
str(Path(EGRESS_CA_IN_CONTAINER).parent),
False,
))
if ep.routes:
mounts.append((
str(ep.routes_path.parent),
str(Path(EGRESS_ROUTES_IN_CONTAINER).parent),
True,
))
sp = plan.supervise_plan
if sp is not None:
# `container run --mount type=bind` only accepts directory
# sources (a file source fails with "is not a directory") —
# mount db_path's dedicated parent dir instead of the file
# itself, same as the CA/routes mounts above.
mounts.append((
str(sp.db_path.parent),
str(Path(DB_PATH_IN_CONTAINER).parent),
False,
))
return tuple(mounts)
def _mount_spec(host_path: str, container_path: str, read_only: bool) -> str:
spec = f"type=bind,source={host_path},target={container_path}"
if read_only:
spec += ",readonly"
return spec
def _agent_env_entries(
plan: MacosContainerBottlePlan,
sidecar_ip: str,
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
) -> tuple[str, ...]:
proxy_url = f"http://{sidecar_ip}:{EGRESS_PORT}"
no_proxy = _agent_no_proxy(plan, sidecar_ip)
# Token-less at run time — the token does not exist yet (see
# `_identity_proxy_env`). Anything egressing before the exec-time override
# is denied by `/resolve`, which is the safe direction.
proxy_url = _proxy_url(endpoint.gateway_ip)
no_proxy = _no_proxy(endpoint.gateway_ip)
env = [
f"HTTPS_PROXY={proxy_url}",
f"HTTP_PROXY={proxy_url}",
@@ -447,12 +339,12 @@ def _agent_env_entries(
env.append(f"MCP_SUPERVISE_URL={plan.agent_supervise_url}")
for name, value in sorted(plan.agent_provision.guest_env.items()):
env.append(f"{name}={value}")
# Forwarded vars: bare name → inherits from the `container run` process env
# so the secret value never lands on argv.
for name in sorted(plan.forwarded_env.keys()):
env.append(name)
env.extend(egress_agent_env_entries(plan.egress_plan))
return tuple(env)
def _agent_no_proxy(plan: MacosContainerBottlePlan, sidecar_ip: str) -> str:
hosts = ["localhost", "127.0.0.1", sidecar_ip]
return ",".join(hosts)
__all__ = ["launch"]
+157 -12
View File
@@ -60,13 +60,21 @@ def dns_server() -> str:
def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
"""Build an OCI image with Apple's BuildKit-backed `container build`."""
"""Build an OCI image with Apple's BuildKit-backed `container build`.
Set `BOT_BOTTLE_NO_CACHE=1` (the `start --no-cache` flag) to force
`--no-cache`. The npm/curl installers some provider Dockerfiles
shell out to can silently no-op on a transient network failure
e.g. an `optionalDependencies` fetch for a platform-native binary
and the builder will then cache that broken layer indefinitely."""
info(
f"building image {ref} from {context} with Apple Container "
"(layer cache keeps repeat builds fast)"
)
_ensure_builder_dns()
args = [_CONTAINER, "build", "-t", ref, "--dns", dns_server()]
if os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
args.append("--no-cache")
if dockerfile:
# `container build` resolves -f relative to the current working
# directory, not the build context. Anchor a relative Dockerfile to
@@ -78,6 +86,28 @@ def build_image(ref: str, context: str, *, dockerfile: str = "") -> None:
subprocess.run(args, check=True)
def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
"""Run `argv` inside a throwaway container of a freshly built agent
image and die loudly if it fails, instead of shipping an image
whose CLI only breaks at first real use. No-op when the provider
hasn't declared a smoke test (`AgentProviderRuntime.smoke_test`)."""
if not argv:
return
result = subprocess.run(
[_CONTAINER, "run", "--rm", "--entrypoint", argv[0], image, *argv[1:]],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
detail = (result.stderr or result.stdout or "").strip()
die(
f"agent image {image!r} failed its post-build smoke test "
f"({' '.join(argv)}): {detail}\n"
f"Try rebuilding from scratch: bot-bottle start --no-cache"
)
def commit_container(container_name: str, image_tag: str) -> None:
"""Snapshot a running Apple Container as a local image.
@@ -407,22 +437,137 @@ def inspect_container(name: str) -> dict[str, object]:
def container_ipv4_on_network(name: str, network: str) -> str:
data = inspect_container(name)
status = data.get("status")
"""The container's IPv4 address on `network`. Fatal if absent — callers
that can tolerate "not yet" want `try_container_ipv4_on_network`."""
ip = try_container_ipv4_on_network(name, network)
if not ip:
die(f"container {name} has no IPv4 address on {network}")
return ip
def run_container_argv(argv: list[str]) -> subprocess.CompletedProcess[str]:
"""Run a `container` command, returning the result for the caller to
interpret. Unlike the `die`-on-failure helpers above, this lets callers
that raise their own typed errors (the gateway / orchestrator lifecycle)
keep control of the failure path."""
return subprocess.run(argv, capture_output=True, text=True, check=False)
def bind_mount_spec(source: str, target: str, *, readonly: bool = False) -> str:
"""A `container run --mount` bind spec. One definition so the gateway and
orchestrator emit an identical string a divergence here would silently
break one backend's mounts while the other kept working."""
spec = f"type=bind,source={source},target={target}"
if readonly:
spec += ",readonly"
return spec
def _normalize_digest(value: str) -> str:
return value.split(":", 1)[1] if ":" in value else value
def _inspect_first(argv: list[str]) -> dict[str, object]:
"""Run an inspect command and return its first JSON object, or {} on any
failure (non-zero exit, malformed JSON, unexpected shape). {} is the shared
'don't know' signal all the non-fatal inspect readers below build on — a
caller comparing against it treats it as 'leave the working container
alone', never as a mismatch."""
result = run_container_argv(argv)
if result.returncode != 0:
return {}
try:
data = json.loads(result.stdout or "[]")
except json.JSONDecodeError:
return {}
if isinstance(data, list):
data = data[0] if data else {}
return data if isinstance(data, dict) else {}
def _descriptor_digest(node: object) -> str:
"""The normalized digest under a `{... "descriptor": {"digest": ...}}`
node, or "". Both the image and container inspect shapes nest the image's
identity this way, so the digest readers stay symmetric a difference
between them is what would spuriously recreate a container."""
if not isinstance(node, dict):
return ""
descriptor = node.get("descriptor")
if isinstance(descriptor, dict) and descriptor.get("digest"):
return _normalize_digest(str(descriptor["digest"]))
return ""
def image_digest(ref: str) -> str:
"""The digest of image `ref`, or "" if it can't be read. Reads exactly the
field `container_image_digest` reads (`configuration.descriptor.digest`) so
the two are comparable; "" means 'don't know' → callers don't churn."""
data = _inspect_first([_CONTAINER, "image", "inspect", ref])
return _descriptor_digest(data.get("configuration"))
def container_image_digest(name: str) -> str:
"""The digest of the image container `name` was created from, or "" if it
can't be read. Compare with `image_digest(ref)` to tell whether a running
container predates an image rebuild."""
config = _inspect_first([_CONTAINER, "inspect", name]).get("configuration")
image = config.get("image") if isinstance(config, dict) else None
return _descriptor_digest(image)
def container_env(name: str) -> dict[str, str]:
"""The env container `name` was started with, or {} if unreadable. Lets a
caller tell whether a running container's baked-in configuration still
matches what it would pass today."""
config = _inspect_first([_CONTAINER, "inspect", name]).get("configuration")
init = config.get("initProcess") if isinstance(config, dict) else None
entries = init.get("environment") if isinstance(init, dict) else None
if not isinstance(entries, list):
return {}
env: dict[str, str] = {}
for entry in entries:
if isinstance(entry, str) and "=" in entry:
key, value = entry.split("=", 1)
env[key] = value
return env
def try_container_ipv4_on_network(name: str, network: str) -> str:
"""`container_ipv4_on_network` without the fatal exit: "" when the address
isn't readable yet. For pollers — a container is created before it has an
address, so "not yet" is an expected state there, not an error."""
status = _inspect_first([_CONTAINER, "inspect", name]).get("status")
networks = status.get("networks") if isinstance(status, dict) else None
if not isinstance(networks, list):
die(f"container inspect {name} did not include status.networks")
return ""
for entry in networks:
if not isinstance(entry, dict):
continue
if entry.get("network") != network:
if not isinstance(entry, dict) or entry.get("network") != network:
continue
raw = entry.get("ipv4Address")
if not isinstance(raw, str) or not raw:
die(f"container {name} has no IPv4 address on {network}")
return raw.split("/", 1)[0]
die(f"container {name} is not attached to network {network}")
raise AssertionError("unreachable")
if isinstance(raw, str) and raw:
return raw.split("/", 1)[0]
return ""
def wait_container_ipv4_on_network(
name: str, network: str, *, timeout: float = 15.0, poll: float = 0.25,
) -> str:
"""Poll for the container's DHCP-assigned address on `network`, returning
it once available or "" on timeout.
Apple Container has no `--ip`: `container run --detach` can return before
vmnet's DHCP has populated `status.networks[].ipv4Address`, so a bare read
right after start races the assignment. Callers that need the address (the
attribution key, the gateway's proxy target) poll through here instead of
the fatal `container_ipv4_on_network`."""
deadline = time.monotonic() + timeout
while True:
ip = try_container_ipv4_on_network(name, network)
if ip:
return ip
if time.monotonic() >= deadline:
return ""
time.sleep(poll)
def image_id(ref: str) -> str:
+1 -1
View File
@@ -94,7 +94,7 @@ def prepare_egress(
def prepare_supervise(bottle: ManifestBottle, slug: str) -> SupervisePlan | None:
"""Prepare the supervise sidecar state dir. Returns None when
"""Prepare the supervise daemon state dir. Returns None when
bottle.supervise is falsy."""
if not bottle.supervise:
return None
+30 -9
View File
@@ -31,6 +31,7 @@ from __future__ import annotations
import dataclasses
import json
import secrets
import socket
import string
from dataclasses import dataclass
from pathlib import Path
@@ -43,17 +44,18 @@ from .paths import bot_bottle_root
_STATE_SUBDIR = "state"
_PER_BOTTLE_DOCKERFILE_NAME = "Dockerfile"
_COMMITTED_IMAGE_NAME = "committed-image"
_COMMITTED_ROOTFS_NAME = "committed-rootfs.tar"
_TRANSCRIPT_SUBDIR = "transcript"
# Per-sidecar scratch subdirs. PRD 0018 chunk 2: bind-mount sources
# Per-daemon scratch subdirs. PRD 0018 chunk 2: bind-mount sources
# live here so chunk 3's `docker compose up` can find them at stable
# paths. Each sidecar's `prepare()` writes config + CAs into its own
# paths. Each daemon's `prepare()` writes config + CAs into its own
# subdir; the launch step is unchanged today (still `docker cp`).
_EGRESS_SUBDIR = "egress"
_GIT_GATE_SUBDIR = "git-gate"
_SUPERVISE_SUBDIR = "supervise"
_AGENT_SUBDIR = "agent"
_METADATA_NAME = "metadata.json"
# Live-config dir bind-mounted into the supervise sidecar (read-only).
# Live-config dir bind-mounted into the supervise daemon (read-only).
# Host's apply paths keep these files fresh so supervise's
# `list-egress-routes` MCP tool returns the current state —
# not a snapshot from launch time.
@@ -87,6 +89,14 @@ def bottle_identity(agent_name: str) -> str:
return f"{slug}-{suffix}"
def globalize_slug(slug: str) -> str:
"""Return a globally-unique slug qualified with the current hostname.
Assumes slug is a value returned from mint_slug. Use wherever a slug
must be unique across hosts (e.g. deploy-key titles)."""
return f"{socket.gethostname()}-{slug}"
@dataclass(frozen=True)
class BottleMetadata:
"""Persistent record of how a bottle was launched, written at
@@ -191,6 +201,15 @@ def committed_image_path(identity: str) -> Path:
return bottle_state_dir(identity) / _COMMITTED_IMAGE_NAME
def committed_rootfs_path(identity: str) -> Path:
"""Where the Firecracker freezer stores a snapshot of the bottle's
guest rootfs (a plain tar). This is the resumable/migratable artifact
the Firecracker backend boots from no Docker image involved. The
matching `committed-image` state file records that a snapshot exists
(and its path); `resume` boots from this tar when both are present."""
return bottle_state_dir(identity) / _COMMITTED_ROOTFS_NAME
def write_committed_image(identity: str, image_tag: str) -> Path:
"""Persist the committed image tag for `identity`. The next
`cli.py resume <identity>` will boot from this image instead of
@@ -222,7 +241,7 @@ def per_bottle_image_tag(identity: str) -> str:
def live_config_dir(identity: str) -> Path:
"""Per-bottle live-config dir. Bind-mounted read-only into the
supervise sidecar; the host's apply paths refresh the files on
supervise daemon; the host's apply paths refresh the files on
every operator approval so the agent's `list-*` MCP tools always
return current state."""
return bottle_state_dir(identity) / _LIVE_CONFIG_SUBDIR
@@ -260,9 +279,9 @@ def transcript_snapshot_dir(identity: str) -> Path:
return bottle_state_dir(identity) / _TRANSCRIPT_SUBDIR
# --- Per-sidecar scratch subdirs (PRD 0018 chunk 2) ------------------------
# --- Per-daemon scratch subdirs (PRD 0018 chunk 2) ------------------------
#
# Each sidecar gets its own subdir under the bottle's state dir for
# Each daemon gets its own subdir under the bottle's state dir for
# bind-mount sources (config, CAs, hooks, etc.). Prepare-time writes
# land here; the state dir's normal cleanup (`cleanup_state`) reaps
# them along with everything else when the bottle session ends and
@@ -270,20 +289,20 @@ def transcript_snapshot_dir(identity: str) -> Path:
def egress_state_dir(identity: str) -> Path:
"""State subdir for the egress sidecar: routes.yaml + the
"""State subdir for the egress daemon: routes.yaml + the
per-bottle mitmproxy CA. Bind-mount source from chunk 3 onward."""
return bottle_state_dir(identity) / _EGRESS_SUBDIR
def git_gate_state_dir(identity: str) -> Path:
"""State subdir for the git-gate sidecar: entrypoint + hooks +
"""State subdir for the git-gate daemon: entrypoint + hooks +
per-upstream known_hosts. Bind-mount source from chunk 3
onward."""
return bottle_state_dir(identity) / _GIT_GATE_SUBDIR
def supervise_state_dir(identity: str) -> Path:
"""State subdir reserved for supervise sidecar bind-mount sources.
"""State subdir reserved for supervise daemon bind-mount sources.
Runtime queue/audit rows live in the host-level bot-bottle SQLite
database, so they survive state-dir cleanup."""
return bottle_state_dir(identity) / _SUPERVISE_SUBDIR
@@ -340,10 +359,12 @@ __all__ = [
"BottleMetadata",
"agent_state_dir",
"bottle_identity",
"globalize_slug",
"bottle_state_dir",
"cleanup_state",
"clear_preserve_marker",
"committed_image_path",
"committed_rootfs_path",
"egress_state_dir",
"git_gate_state_dir",
"is_preserved",
+2 -2
View File
@@ -2,8 +2,8 @@
Walks every registered backend (docker, firecracker, macos-container)
so a single `./cli.py cleanup` reaps every backend's leftovers — a
firecracker bottle's sidecars won't survive a docker-only cleanup pass
(issue addressed alongside #77).
firecracker bottle's VM processes and run dirs won't survive a
docker-only cleanup pass (issue addressed alongside #77).
Each backend's `prepare_cleanup` enumerates its own resources;
docker's `_list_orphan_state_dirs` consults
+16
View File
@@ -46,6 +46,17 @@ def cmd_start(argv: list[str]) -> int:
parser = argparse.ArgumentParser(prog=f"{PROG} start", add_help=True)
parser.add_argument("--dry-run", action="store_true")
parser.add_argument("--cwd", action="store_true", help="copy host cwd into the running bottle")
parser.add_argument(
"--no-cache",
action="store_true",
help=(
"rebuild agent/sidecar images from scratch, bypassing the "
"build layer cache. Use when an image looks broken after a "
"dependency bump — e.g. an installer's optionalDependencies "
"fetch silently no-op'd on a transient failure and got baked "
"into a cached layer."
),
)
parser.add_argument(
"--backend",
choices=known_backend_names(),
@@ -97,6 +108,11 @@ def cmd_start(argv: list[str]) -> int:
args = parser.parse_args(argv)
dry_run = args.dry_run or os.environ.get("BOT_BOTTLE_DRY_RUN") == "1"
if args.no_cache or os.environ.get("BOT_BOTTLE_NO_CACHE") == "1":
# Read by build_image() in each backend's util module — set here
# so both the interactive and --headless paths pick it up without
# threading a no_cache field through every backend's plan dataclass.
os.environ["BOT_BOTTLE_NO_CACHE"] = "1"
manifest = ManifestIndex.resolve(USER_CWD)
backend_name: str | None = args.backend
+84 -87
View File
@@ -20,32 +20,19 @@ from datetime import datetime, timezone
from pathlib import Path
from ..paths import bot_bottle_root
from ..bottle_state import read_metadata
from ..backend.docker.egress_apply import (
EgressApplyError,
applicator as _docker_applicator,
)
from ..backend.macos_container.egress_apply import (
applicator as _macos_applicator,
)
from ..log import Die, error, info
from ..orchestrator.client import (
OrchestratorClient,
OrchestratorClientError,
discover_orchestrator_url,
)
from ..supervise import (
COMPONENT_FOR_TOOL,
AuditEntry,
Proposal,
Response,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_GITLEAKS_ALLOW,
TOOL_EGRESS_TOKEN_ALLOW,
list_all_pending_proposals,
render_diff,
write_audit_entry,
write_response,
)
from ._common import PROG
@@ -60,30 +47,61 @@ _REPORT_ONLY_TOOLS: tuple[str, ...] = (TOOL_GITLEAKS_ALLOW, TOOL_EGRESS_TOKEN_AL
@dataclass(frozen=True)
class QueuedProposal:
"""A pending proposal from the supervise queue."""
"""A pending proposal from the supervise queue.
`label` is the operator-facing bottle name (the human slug the
orchestrator resolved from the registry); `proposal.bottle_slug` is the
opaque bottle_id every operator action is keyed by. Display uses `label`;
respond calls use `proposal.bottle_slug`."""
proposal: Proposal
label: str = ""
# Errors any remediation engine may raise. Caught by the TUI key
# handlers and surfaced in the status line so a failed apply keeps
# the proposal pending rather than crashing curses.
ApplyError = (EgressApplyError,)
# A failed operator action (orchestrator unreachable, bottle torn down,
# 409) is caught by the TUI key handlers and surfaced in the status line so
# the proposal stays pending rather than crashing curses.
ApplyError = (OrchestratorClientError,)
def apply_routes_change(slug: str, content: str) -> tuple[str, str]:
meta = read_metadata(slug)
backend = meta.backend if meta is not None else ""
if backend == "macos-container":
return _macos_applicator.apply_routes_change(slug, content)
return _docker_applicator.apply_routes_change(slug, content)
# The one per-host orchestrator, discovered lazily on first use. Every
# operator action — list, approve, reject — goes through its HTTP control
# plane (the orchestrator owns the single DB + live policy); there is no
# direct-DB path and no backend branching here.
_client_instance: OrchestratorClient | None = None
def _resolve_orchestrator_url() -> str:
"""URL of the running orchestrator control plane, starting one on demand.
Supervise is often the first thing an operator runs before any bottle
has booted the control plane. So when discovery finds nothing, bring up
the selected backend's orchestrator + gateway (idempotent) rather than
failing with "launch a bottle first"."""
try:
return discover_orchestrator_url()
except OrchestratorClientError:
from ..backend import get_bottle_backend
backend = get_bottle_backend()
info(f"no orchestrator control plane running; starting one ({backend.name})…")
return backend.ensure_orchestrator()
def _client() -> OrchestratorClient:
global _client_instance # noqa: PLW0603 — CLI-session singleton
if _client_instance is None:
_client_instance = OrchestratorClient(_resolve_orchestrator_url())
return _client_instance
def discover_pending() -> list[QueuedProposal]:
"""Collect pending proposals across bottles."""
"""Collect pending proposals across bottles from the orchestrator."""
out = [
QueuedProposal(proposal=proposal)
for proposal in list_all_pending_proposals()
QueuedProposal(
proposal=Proposal.from_dict(d),
label=str(d.get("bottle_label") or d.get("bottle_slug") or ""),
)
for d in _client().supervise_pending()
]
out.sort(key=lambda q: q.proposal.arrival_timestamp)
return out
@@ -91,8 +109,8 @@ def discover_pending() -> list[QueuedProposal]:
def _approval_status(qp: QueuedProposal, verb: str) -> str:
"""Status-line text after a successful approval."""
base = f"{verb} {qp.proposal.tool} for [{qp.proposal.bottle_slug}]"
return f"{base}; resume: ./cli.py resume {qp.proposal.bottle_slug}"
base = f"{verb} {qp.proposal.tool} for [{qp.label}]"
return f"{base}; resume: ./cli.py resume {qp.label}"
def _detail_lines(
@@ -103,7 +121,7 @@ def _detail_lines(
"""Return the detail-view body as (text, curses-attr) tuples."""
p = qp.proposal
out: list[tuple[str, int]] = [
(f"bottle: {p.bottle_slug}", 0),
(f"bottle: {qp.label}", 0),
(f"tool: {p.tool}", 0),
(f"id: {p.id}", 0),
(f"arrived: {p.arrival_timestamp}", 0),
@@ -136,39 +154,27 @@ def approve(
notes: str = "",
final_file: str | None = None,
) -> None:
"""Apply the proposal, write the waiting response, and audit it."""
status = STATUS_MODIFIED if final_file is not None else STATUS_APPROVED
file_to_apply = final_file if final_file is not None else qp.proposal.proposed_file
diff_before, diff_after = "", ""
if qp.proposal.tool in (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK):
diff_before, diff_after = apply_routes_change(
qp.proposal.bottle_slug,
file_to_apply,
)
response = Response(
proposal_id=qp.proposal.id,
status=status,
"""Approve (or, with `final_file`, modify-then-approve) via the
orchestrator: it applies the route change to the bottle's live policy,
writes the response that unblocks the agent, and audits it one atomic
server-side op. Raises `OrchestratorClientError` on failure."""
_client().supervise_respond(
qp.proposal.id,
bottle_slug=qp.proposal.bottle_slug,
decision="modify" if final_file is not None else "approve",
notes=notes,
final_file=final_file,
)
write_response(qp.proposal.bottle_slug, response)
_write_audit(
qp, action=status, notes=notes,
diff_before=diff_before, diff_after=diff_after,
)
def reject(qp: QueuedProposal, *, reason: str) -> None:
"""Write a rejection response and an audit entry."""
response = Response(
proposal_id=qp.proposal.id,
status=STATUS_REJECTED,
"""Reject via the orchestrator (writes the response + audit)."""
_client().supervise_respond(
qp.proposal.id,
bottle_slug=qp.proposal.bottle_slug,
decision="reject",
notes=reason,
final_file=None,
)
write_response(qp.proposal.bottle_slug, response)
_write_audit(qp, action=STATUS_REJECTED, notes=reason, diff_before="", diff_after="")
def _approve_from_tui(
@@ -188,29 +194,6 @@ def _approve_from_tui(
return _approval_status(qp, verb)
def _write_audit(
qp: QueuedProposal,
*,
action: str,
notes: str,
diff_before: str,
diff_after: str,
) -> None:
"""Audit log for egress tool."""
component = COMPONENT_FOR_TOOL.get(qp.proposal.tool)
if component is None:
return
write_audit_entry(AuditEntry(
timestamp=datetime.now(timezone.utc).isoformat(),
bottle_slug=qp.proposal.bottle_slug,
component=component,
operator_action=action,
operator_notes=notes,
justification=qp.proposal.justification,
diff=render_diff(diff_before, diff_after, label=component),
))
# --- $EDITOR integration --------------------------------------------------
@@ -245,6 +228,20 @@ def cmd_supervise(argv: list[str]) -> int:
)
args = parser.parse_args(argv)
# Establish the orchestrator connection up front so a missing control
# plane is a clean one-line error, not a curses crash mid-loop. This also
# starts the orchestrator on demand when none is running (see `_client`).
try:
_client()
except OrchestratorClientError as e:
error(str(e))
return 1
except Die as e:
# Backend has no orchestrator to start (e.g. macos-container).
if e.message:
error(e.message)
return e.code if isinstance(e.code, int) else 1
if args.once:
return _list_once()
try:
@@ -299,7 +296,7 @@ def _list_once() -> int:
for qp in pending:
sys.stdout.write(
f"{qp.proposal.arrival_timestamp} "
f"[{qp.proposal.bottle_slug}] "
f"[{qp.label}] "
f"{qp.proposal.tool} "
f"{qp.proposal.id}\n"
)
@@ -396,7 +393,7 @@ def _main_loop(stdscr: "curses._CursesWindow") -> None: # type: ignore # pragm
reason = _prompt(stdscr, "reject reason: ")
if reason:
reject(qp, reason=reason)
status_line = f"rejected {qp.proposal.tool} for [{qp.proposal.bottle_slug}]"
status_line = f"rejected {qp.proposal.tool} for [{qp.label}]"
else:
status_line = "reject aborted (empty reason)"
@@ -435,7 +432,7 @@ def _render(
cursor = "> " if i == selected else " "
line = (
f"{cursor}{ts_short} "
f"[{p.bottle_slug}] {p.tool:<18} {p.id[:8]}"
f"[{qp.label}] {p.tool:<18} {p.id[:8]}"
)
attr = curses.A_REVERSE if i == selected else curses.A_NORMAL
stdscr.addnstr(row, 0, line, w - 1, attr)
+12 -3
View File
@@ -4,7 +4,7 @@ The Claude-specific behavior previously inlined under
`agent_provider.agent_provision_plan` (claude.json trust marker,
api.anthropic.com egress route, OAuth-token placeholder), plus
the `claude mcp add` invocation that registers the supervise
sidecar in claude-code's user config (PRD 0013)."""
gateway in claude-code's user config (PRD 0013)."""
from __future__ import annotations
@@ -32,6 +32,8 @@ if TYPE_CHECKING:
_SUPERVISE_MCP_NAME = "supervise"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
_IDENTITY_HEADER = "x-bot-bottle-identity"
def _skills_dir(guest_home: str) -> str:
@@ -91,6 +93,7 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="append_file",
bypass_args=("--dangerously-skip-permissions",),
resume_args=("--continue",),
smoke_test=("claude", "--version"),
)
@@ -293,16 +296,22 @@ class ClaudeAgentProvider(AgentProvider):
supervise_url: str,
) -> None:
"""Run `claude mcp add` inside the agent guest to register the
supervise sidecar in claude-code's user config (~/.claude.json).
supervise daemon in claude-code's user config (~/.claude.json).
Failure is logged but not fatal the bottle still works without
the entry; the operator can register it manually."""
if plan.supervise_plan is None:
return
info(f"registering supervise MCP server in agent claude config → {supervise_url}")
# Deliver the identity token as an MCP request header — the supervise
# daemon requires it (mandatory (source_ip, token) attribution).
token = getattr(plan, "identity_token", "")
header = (
f" --header {shlex.quote(f'{_IDENTITY_HEADER}: {token}')}" if token else ""
)
r = bottle.exec(
f"claude mcp add --scope user --transport http "
f"{_SUPERVISE_MCP_NAME} {supervise_url}",
f"{_SUPERVISE_MCP_NAME} {supervise_url}{header}",
user="node",
)
if r.returncode != 0:
+68 -15
View File
@@ -4,11 +4,12 @@ The Codex-specific behavior previously inlined under
`agent_provider.agent_provision_plan` (config.toml trust marker,
chatgpt.com / api.openai.com egress routes, optional host-credential
forwarding with dummy-auth.json + verify), plus the `codex mcp add`
invocation that registers the supervise sidecar in Codex's
invocation that registers the supervise daemon in Codex's
~/.codex/config.toml (PRD 0050)."""
from __future__ import annotations
import base64
import os
import shlex
from pathlib import Path
@@ -26,7 +27,7 @@ from ...agent_provider import (
)
from .codex_auth import codex_host_access_token, write_codex_dummy_auth_file
from ...egress import CODEX_HOST_CREDENTIAL_TOKEN_REF, EgressRoute
from ...log import die, info, warn
from ...log import die, info
if TYPE_CHECKING:
@@ -34,6 +35,8 @@ if TYPE_CHECKING:
_SUPERVISE_MCP_NAME = "supervise"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
_IDENTITY_HEADER = "x-bot-bottle-identity"
_CODEX_CLI = "/home/node/.codex/packages/standalone/current/bin/codex"
_CODEX_CLI_PATH = (
"/home/node/.local/bin:"
@@ -42,6 +45,41 @@ _CODEX_CLI_PATH = (
)
def _toml_basic_string(value: str) -> str:
"""Quote `value` as a TOML basic (double-quoted) string."""
escaped = (
value.replace("\\", "\\\\")
.replace('"', '\\"')
.replace("\n", "\\n")
.replace("\t", "\\t")
)
return f'"{escaped}"'
def _supervise_mcp_config_toml(supervise_url: str, token: str) -> str:
"""Render the `[mcp_servers.supervise]` streamable-HTTP entry for
Codex's `config.toml`.
The Codex CLI has no `mcp add --header` flag; a static request
header on an HTTP MCP server is only expressible via the
`http_headers` config key (see `RawMcpServerConfig` /
`McpServerTransportConfig::StreamableHttp`). We deliver the
mandatory identity token (source_ip, token attribution) that way.
Only Codex-supported streamable-HTTP keys (`url`, `http_headers`)
are emitted."""
lines = [
"",
f"[mcp_servers.{_SUPERVISE_MCP_NAME}]",
f"url = {_toml_basic_string(supervise_url)}",
]
if token:
key = _toml_basic_string(_IDENTITY_HEADER)
val = _toml_basic_string(token)
lines.append(f"http_headers = {{ {key} = {val} }}")
lines.append("")
return "\n".join(lines)
def _skills_dir(guest_home: str) -> str:
# Codex agents still read skills from the claude-code convention
# (~/.claude/skills/) — the bot-bottle-codex image follows the
@@ -61,6 +99,7 @@ _RUNTIME = AgentProviderRuntime(
prompt_mode="read_prompt_file",
bypass_args=("--dangerously-bypass-approvals-and-sandbox",),
resume_args=("resume", "--last"),
smoke_test=(_CODEX_CLI, "--version"),
)
@@ -265,25 +304,39 @@ class CodexAgentProvider(AgentProvider):
bottle: "Bottle",
supervise_url: str,
) -> None:
"""Run `codex mcp add` inside the agent guest to register the
supervise sidecar in Codex's user config (~/.codex/config.toml).
"""Register the supervise daemon as a streamable-HTTP MCP
server in Codex's user config (`~/.codex/config.toml`).
Mirrors the Claude provider's `claude mcp add` flow — failure
is logged but not fatal."""
We write the `[mcp_servers.supervise]` entry directly rather
than shelling out to `codex mcp add`: the CLI's `add` has no
way to attach a static request header, and the identity token
(mandatory (source_ip, token) attribution) MUST ride on the
MCP request as `http_headers`. Failure is FATAL when supervise
is enabled a silently-unregistered server leaves the agent
with no supervise access and, under mandatory attribution, no
way to recover from inside the bottle."""
if plan.supervise_plan is None:
return
info(f"registering supervise MCP server in agent codex config → {supervise_url}")
r = bottle.exec(
f"{shlex.quote(_CODEX_CLI)} mcp add {_SUPERVISE_MCP_NAME} --url "
f"{shlex.quote(supervise_url)}",
user="node",
token = getattr(plan, "identity_token", "")
block = _supervise_mcp_config_toml(supervise_url, token)
auth_dir = plan.agent_provision.guest_env.get("CODEX_HOME") \
or f"{plan.guest_home}/.codex"
config_path = f"{auth_dir}/config.toml"
# Append via base64 so the TOML payload never has to survive a
# shell-quoting round trip. node owns the config file, so append
# as node to preserve ownership/mode.
payload = base64.b64encode(block.encode()).decode()
script = (
f"printf %s {shlex.quote(payload)} | base64 -d "
f">> {shlex.quote(config_path)}"
)
r = bottle.exec(script, user="node")
if r.returncode != 0:
warn(
f"`codex mcp add supervise` failed (exit {r.returncode}): "
f"{(r.stderr or r.stdout or '').strip()}. Inside the bottle, "
f"register manually with: "
f"codex mcp add supervise --url {shlex.quote(supervise_url)}"
die(
"agent provider provisioning: could not register supervise "
f"MCP server in {config_path}: "
f"{(r.stderr or r.stdout or '').strip()}"
)
def headless_prompt(self, prompt: str) -> list[str]:
+1 -1
View File
@@ -3,7 +3,7 @@
Pure Python, no mitmproxy dependency. Each detector is a module-level
function returning `ScanResult | None`.
Ships flat into the sidecar bundle image alongside
Ships flat into the gateway image alongside
`egress_addon_core.py` both this file and the package source use
the same try/except import shim pattern.
"""
+6 -6
View File
@@ -2,7 +2,7 @@
This module defines the abstract proxy (`Egress`), its plan
dataclass (`EgressPlan`), and the resolved per-route shape
(`EgressRoute`). The sidecar's start/stop lifecycle is backend-
(`EgressRoute`). The gateway's start/stop lifecycle is backend-
specific and lives on concrete subclasses (see
`bot_bottle/backend/docker/egress.py`).
"""
@@ -63,8 +63,8 @@ def _random_canary_env() -> str:
return f"{first}_{second}_SECRET"
def egress_sidecar_env_entries(plan: "EgressPlan") -> tuple[str, ...]:
"""Return sidecar env entries needed by egress across all backends."""
def egress_gateway_env_entries(plan: "EgressPlan") -> tuple[str, ...]:
"""Return gateway env entries needed by egress across all backends."""
env: list[str] = []
if plan.routes:
env.extend(sorted(plan.token_env_map.keys()))
@@ -87,7 +87,7 @@ class EgressRoute(Route):
Inherits `host`, `matches`, `auth_scheme`, and `token_env`
from `egress_addon_core.Route` those are the fields that cross the
YAML wire into the sidecar. The fields below are host-only and
YAML wire into the gateway. The fields below are host-only and
are never serialised to the addon.
`token_ref` is the host env var the CLI reads at launch and forwards
@@ -386,7 +386,7 @@ class Egress(ABC):
routes_path.write_text(egress_render_routes(routes, log=log))
routes_path.chmod(0o600)
# Generate a per-session fake secret under a plausible random env name.
# The sidecar marks that exact env name as sensitive for known-secret
# The gateway marks that exact env name as sensitive for known-secret
# scanning; the agent receives the same name/value as exfil bait.
canary = secrets.token_urlsafe(32)
return EgressPlan(
@@ -412,6 +412,6 @@ __all__ = [
"egress_resolve_token_values",
"egress_routes_for_bottle",
"egress_agent_env_entries",
"egress_sidecar_env_entries",
"egress_gateway_env_entries",
"egress_token_env_map",
]
+107 -36
View File
@@ -6,10 +6,13 @@ egress container."""
from __future__ import annotations
import asyncio
import base64
import binascii
import json
import os
import signal
import sys
import typing
from pathlib import Path
from mitmproxy import http # type: ignore[import-not-found] # pylint: disable=import-error
@@ -68,10 +71,28 @@ INTROSPECT_HOST = "_egress.local"
# → legacy per-bottle single-tenant mode (unchanged).
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
# App-layer identity token (defense-in-depth over the source-IP invariant);
# the agent injects it, the addon strips it so it never leaks upstream.
# App-layer identity token. Delivered as proxy credentials
# (`HTTPS_PROXY=http://<bottle_id>:<token>@gw`): clients honor it as part of
# the proxy protocol without app changes, and the addon reads + strips it so
# it never leaks upstream. The legacy `x-bot-bottle-identity` request header
# is still stripped defensively (git-http uses that header on its own port).
IDENTITY_HEADER = "x-bot-bottle-identity"
def _token_from_proxy_auth(header: str) -> str:
"""Extract the identity token (the password) from a `Proxy-Authorization:
Basic base64(<bottle_id>:<token>)` header. Empty on any malformed value
the mandatory `/resolve` then fail-closes on the empty token."""
scheme, _, encoded = header.partition(" ")
if scheme.lower() != "basic" or not encoded:
return ""
try:
decoded = base64.b64decode(encoded, validate=True).decode("utf-8")
except (binascii.Error, ValueError, UnicodeDecodeError):
return ""
_, _, password = decoded.partition(":")
return password
# Seconds the egress proxy holds a token-blocked request open waiting for the
# operator's supervisor decision (PRD 0062), overridable via env.
DEFAULT_TOKEN_ALLOW_TIMEOUT_SECONDS = 300.0
@@ -91,6 +112,10 @@ class EgressAddon:
# Class default so addons built via __new__ (e.g. in tests) default to
# single-tenant; __init__ sets the instance attribute for real runs.
_resolver: "PolicyResolver | None" = None
# Class default so __new__-built addons have it (real runs get a fresh
# per-instance dict in __init__; only http_connect mutates it, which the
# request-flow tests don't exercise).
_conn_tokens: "dict[str, str]" = {}
def __init__(self) -> None:
self.routes_path = os.environ.get("EGRESS_ROUTES", DEFAULT_ROUTES_PATH)
@@ -105,6 +130,10 @@ class EgressAddon:
# scan. In-memory only (a restart re-prompts); mutated only from the
# asyncio loop that runs the addon hooks, so no lock is needed.
self._safe_tokens: dict[str, set[str]] = {}
# Per-client-connection identity token captured from the CONNECT's
# `Proxy-Authorization` (HTTPS tunnels don't repeat it on the bumped
# inner requests). Keyed by client_conn.id; cleared on disconnect.
self._conn_tokens: dict[str, str] = {}
self._supervise_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "").strip()
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
self._reload(initial=True)
@@ -230,21 +259,57 @@ class EgressAddon:
+ "\n"
)
def _resolve_flow(self, flow: http.HTTPFlow) -> "tuple[Config, str]":
"""The `(Config, supervise slug)` to apply to this request. Single-tenant
the static `self.config` and the env slug. Consolidated the calling
bottle's Config and its bottle id, resolved by source IP in one
round-trip (fail-closed to deny-all + empty slug if unattributed). The
identity token, if the agent injected one, is read then stripped so it
never leaks upstream. The slug keys both the proposal queue and the
per-bottle safelist, so an approval only ever affects its own bottle."""
def _resolve_flow(
self, flow: http.HTTPFlow,
) -> "tuple[Config, str, typing.Mapping[str, str]]":
"""The `(Config, supervise slug, env)` to apply to this request.
Single-tenant the static `self.config`, the env slug, and the process
env. Consolidated the calling bottle's Config + bottle id + auth
tokens, resolved by source IP in one round-trip (fail-closed to deny-all
+ empty slug if unattributed); `env` is the process env overlaid with
the bottle's tokens, so upstream-auth injection (and DLP) use *this*
bottle's credentials — exactly what the per-bottle gateway daemon's env did.
The identity token, if the agent injected one, is read then stripped so
it never leaks upstream."""
if self._resolver is None:
return self.config, self._supervise_slug
return self.config, self._supervise_slug, os.environ
conn = flow.client_conn
client_ip = conn.peername[0] if conn and conn.peername else ""
token = flow.request.headers.get(IDENTITY_HEADER, "")
token = self._request_token(flow)
config, slug, tokens = resolve_client_context(self._resolver, client_ip, token)
env = {**os.environ, **tokens} if tokens else os.environ
return config, slug, env
def _request_token(self, flow: http.HTTPFlow) -> str:
"""The per-bottle identity token for this request, from the proxy
credentials the delivery mechanism (`HTTPS_PROXY=http://id:token@gw`)
that clients honor without app changes. Plain-HTTP requests carry
`Proxy-Authorization` directly; HTTPS bumped requests inherit the token
captured from their tunnel's CONNECT. Read then stripped so it never
leaks upstream (also strips the legacy header, if present)."""
token = _token_from_proxy_auth(
flow.request.headers.get("Proxy-Authorization", ""))
flow.request.headers.pop("Proxy-Authorization", None)
flow.request.headers.pop(IDENTITY_HEADER, None)
return resolve_client_context(self._resolver, client_ip, token)
conn = flow.client_conn
if not token and conn is not None:
token = self._conn_tokens.get(getattr(conn, "id", ""), "")
return token
def http_connect(self, flow: http.HTTPFlow) -> None:
"""Capture the identity token from an HTTPS tunnel's CONNECT (the inner
bumped requests won't carry `Proxy-Authorization`), keyed by client
connection, and strip it so it never reaches upstream."""
token = _token_from_proxy_auth(
flow.request.headers.get("Proxy-Authorization", ""))
flow.request.headers.pop("Proxy-Authorization", None)
conn = flow.client_conn
if conn is not None and getattr(conn, "id", ""):
self._conn_tokens[conn.id] = token
def client_disconnected(self, client: typing.Any) -> None:
"""Drop the per-connection token when the client goes away."""
self._conn_tokens.pop(getattr(client, "id", ""), None)
async def request(self, flow: http.HTTPFlow) -> None:
request_path, _, query = flow.request.path.partition("?")
@@ -253,14 +318,14 @@ class EgressAddon:
self._serve_introspection(flow, request_path)
return
config, slug = self._resolve_flow(flow)
config, slug, env = self._resolve_flow(flow)
# DLP outbound scan BEFORE stripping auth — catches tokens the
# agent tried to smuggle in any header, path, query param, or body.
# Hostname is included to catch DNS-tunnelling exfiltration attempts.
route = match_route(config.routes, flow.request.pretty_host)
if route is not None:
if not await self._handle_outbound_dlp(flow, route, slug):
if not await self._handle_outbound_dlp(flow, route, slug, env):
return
# The redact policy may have rewritten the request line; recompute
# the path/query the git checks below rely on.
@@ -299,7 +364,7 @@ class EgressAddon:
config.routes,
flow.request.pretty_host,
request_path,
os.environ,
env,
request_method=flow.request.method,
request_headers=req_headers,
)
@@ -325,10 +390,12 @@ class EgressAddon:
flow: http.HTTPFlow,
route: Route,
slug: str,
env: "typing.Mapping[str, str]",
) -> bool:
"""Scan the outbound request and apply the route's on-match policy
(PRD 0062). Returns True if the request may be forwarded, False if a
403 response has been written to `flow`.
(PRD 0062). `env` is the per-bottle env overlay (process env + this
bottle's tokens) used for DLP detection. Returns True if the request may
be forwarded, False if a 403 response has been written to `flow`.
Loops so the supervise policy can re-scan after each approval a
second, un-approved token in the same request is still caught."""
@@ -345,7 +412,7 @@ class EgressAddon:
flow.request.pretty_host, request_path, query, headers, "",
)
result = scan_outbound(
route, scan_text, os.environ,
route, scan_text, env,
safe_tokens=self._safe_tokens_for(slug), crlf_text=crlf_text,
)
if result is None or result.severity != "block":
@@ -356,7 +423,7 @@ class EgressAddon:
# redact scrubs every detection (tokens and structural CRLF) and
# forwards; it fails closed only if a match survives the scrub.
if policy == ON_MATCH_REDACT:
if self._redact_outbound(flow, route):
if self._redact_outbound(flow, route, env):
if self.config.log >= LOG_BLOCKS:
sys.stderr.write(json.dumps({
"event": "egress_redacted",
@@ -384,29 +451,31 @@ class EgressAddon:
if not self._supervise_available(slug):
self._block_dlp(flow, result)
return False
approved = await self._supervise_token_block(flow, request_path, result, slug)
approved = await self._supervise_token_block(flow, request_path, result, slug, env)
if not approved:
return False # _supervise_token_block wrote the 403 response
# loop: the approved value is now in safe_tokens; re-scan.
def _redact_outbound(self, flow: http.HTTPFlow, route: Route) -> bool:
def _redact_outbound(
self, flow: http.HTTPFlow, route: Route, env: "typing.Mapping[str, str]",
) -> bool:
"""Scrub detected tokens (and CRLF injection sequences) from the mutable
request surfaces (body, headers, path/query) and re-scan. Returns True
if the request is now clean; False if a block-severity match remains on
a surface redaction cannot rewrite (the hostname) so the caller fails
closed."""
request surfaces (body, headers, path/query) and re-scan. `env` is the
per-bottle env overlay. Returns True if the request is now clean; False
if a block-severity match remains on a surface redaction cannot rewrite
(the hostname) so the caller fails closed."""
body = flow.request.get_text(strict=False)
if body:
redacted_body = redact_tokens(body, env=os.environ)
redacted_body = redact_tokens(body, env=env)
if redacted_body != body:
flow.request.text = redacted_body
for name, value in list(flow.request.headers.items()):
if name.lower() == "host":
continue # routing-critical; never a legitimate token
redacted = strip_crlf(redact_tokens(value, env=os.environ))
redacted = strip_crlf(redact_tokens(value, env=env))
if redacted != value:
flow.request.headers[name] = redacted
redacted_path = strip_crlf(redact_tokens(flow.request.path, env=os.environ))
redacted_path = strip_crlf(redact_tokens(flow.request.path, env=env))
if redacted_path != flow.request.path:
flow.request.path = redacted_path
@@ -419,7 +488,7 @@ class EgressAddon:
crlf_text = build_outbound_scan_text(
flow.request.pretty_host, request_path, query, headers, "",
)
result = scan_outbound(route, scan_text, os.environ, crlf_text=crlf_text)
result = scan_outbound(route, scan_text, env, crlf_text=crlf_text)
return result is None or result.severity != "block"
async def _supervise_token_block(
@@ -428,20 +497,22 @@ class EgressAddon:
request_path: str,
result: ScanResult,
slug: str,
env: "typing.Mapping[str, str]",
) -> bool:
"""Route a token DLP block to the operator's supervisor queue and wait.
`slug` attributes the proposal to the calling bottle (its own queue +
safelist) in the shared gateway this is what keeps one bottle's
approval from unblocking another's request. Returns True if the operator
approved (the matched value is added to that bottle's safelist and the
caller re-scans); False if the request must be blocked (a 403 response
has been written to `flow`)."""
approval from unblocking another's request. `env` is the per-bottle env
overlay used to redact secrets from the proposal text. Returns True if
the operator approved (the matched value is added to that bottle's
safelist and the caller re-scans); False if the request must be blocked
(a 403 response has been written to `flow`)."""
host = flow.request.pretty_host
payload = build_token_allow_payload(
redact_tokens(host, env=os.environ),
redact_tokens(host, env=env),
flow.request.method,
redact_tokens(request_path, env=os.environ),
redact_tokens(request_path, env=env),
result,
)
proposal = _sv.Proposal.new(
+16 -13
View File
@@ -7,8 +7,8 @@ exercise the parse + decision functions without depending on the
container.
Imports: stdlib + `yaml_subset` (which is itself stdlib-only and
ships flat into the sidecar bundle image alongside this file
see `Dockerfile.sidecars`)."""
ships flat into the gateway image alongside this file
see `Dockerfile.gateway`)."""
from __future__ import annotations
@@ -22,7 +22,7 @@ except ImportError: # pragma: no cover - host-side path
from .yaml_subset import YamlSubsetError, parse_yaml_subset
# DLP detector-config parsing lives in a sibling module (also flat-bundled
# into the gateway — see Dockerfile.sidecars). Re-exported below so existing
# into the gateway — see Dockerfile.gateway). Re-exported below so existing
# `from egress_addon_core import ON_MATCH_*` callers keep working.
try:
from egress_dlp_config import ( # type: ignore[import-not-found]
@@ -450,28 +450,31 @@ def resolve_client_config(
class ContextResolverLike(typing.Protocol):
"""The bit of `policy_resolver.PolicyResolver` `resolve_client_context`
needs one round-trip returning both policy and bottle id."""
needs one round-trip returning policy, bottle id, and auth tokens."""
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = ...,
) -> "tuple[str | None, str | None]":
) -> "tuple[str | None, str | None, dict[str, str]]":
...
def resolve_client_context(
resolver: ContextResolverLike, client_ip: str, identity_token: str = "",
) -> "tuple[Config, str]":
"""The calling client's `(Config, bottle_id)` in one round-trip —
) -> "tuple[Config, str, dict[str, str]]":
"""The calling client's `(Config, bottle_id, tokens)` in one round-trip —
**fail-closed**. The Config follows `resolve_client_config`'s deny-all
rules; the bottle id is `""` whenever unattributed or the orchestrator
errored, which the caller treats as "supervise unavailable for this
bottle" (never another bottle's queue). One `/resolve` keys both the
per-request egress policy and the per-bottle supervise queue + safelist."""
errored (caller treats as "supervise unavailable", never another bottle's
queue); `tokens` are the per-bottle upstream auth values the addon injects.
One `/resolve` keys the egress policy, the supervise queue + safelist, and
auth injection."""
try:
policy, bottle_id = resolver.resolve_policy_and_bottle_id(client_ip, identity_token)
policy, bottle_id, tokens = resolver.resolve_policy_and_bottle_id(
client_ip, identity_token,
)
except Exception: # noqa: BLE001 # pylint: disable=broad-exception-caught
return Config(routes=()), "" # orchestrator unreachable/errored → deny
return _config_from_policy(policy), (bottle_id or "")
return Config(routes=()), "", {} # orchestrator unreachable/errored → deny
return _config_from_policy(policy), (bottle_id or ""), tokens
# ---------------------------------------------------------------------------
+2 -2
View File
@@ -6,8 +6,8 @@ and what the proxy does when an outbound detector matches a token
kept apart from the request-time scan/decision flow in `egress_addon_core`
so each half reads top-to-bottom without scrolling past the other.
Stdlib-only; ships flat into the sidecar bundle image alongside
`egress_addon_core.py` see `Dockerfile.sidecars`."""
Stdlib-only; ships flat into the gateway image alongside
`egress_addon_core.py` see `Dockerfile.gateway`."""
from __future__ import annotations
+3 -3
View File
@@ -1,8 +1,8 @@
#!/bin/sh
# Egress daemon entrypoint inside the sidecar bundle (PRD 0024).
# Egress daemon entrypoint inside the gateway (PRD 0024).
#
# Extracted verbatim from Dockerfile.egress's prior inline `sh -c`
# ENTRYPOINT so the supervisor in bot_bottle/sidecar_init.py can
# ENTRYPOINT so the supervisor in bot_bottle/gateway_init.py can
# call it as a normal child. Behavior is unchanged:
#
# * Upstream proxy: when EGRESS_UPSTREAM_PROXY is set, switch
@@ -22,7 +22,7 @@ set -e
# Pin mitmproxy's config dir to the bind-mount location of its CA
# regardless of which user mitmdump runs as. In the legacy
# four-sidecar setup (Dockerfile.egress, USER mitmproxy) this
# four-daemon setup (Dockerfile.egress, USER mitmproxy) this
# resolved naturally to `~mitmproxy/.mitmproxy`. In the PRD 0024
# bundle (USER root) `~root/.mitmproxy` is empty, so without this
# flag mitmdump would generate a fresh CA on the wrong path and
@@ -1,26 +1,26 @@
"""Per-bottle sidecar supervisor (PRD 0024 chunk 1).
"""Gateway data-plane supervisor (PRD 0070; PRD 0024 bundle shape).
PID 1 inside the `bot-bottle-sidecars` bundle image. Spawns
PID 1 inside the `bot-bottle-gateway` data-plane image. Spawns
the configured daemons (egress, git-gate, supervise),
forwards SIGTERM/SIGINT to each child, and propagates per-daemon
stdout+stderr to the container log with a `[name] ` prefix.
Failure policy (interim): when a child dies unexpectedly, the
supervisor logs the death and leaves the surviving children
running. The bundle stays up; whatever the dead daemon served
running. The gateway stays up; whatever the dead daemon served
will start failing, surfacing in the agent's own error path.
The supervisor itself exits only when (a) the operator/compose
sends SIGTERM/SIGINT, or (b) every child has died.
The supervisor itself exits only when (a) the operator sends
SIGTERM/SIGINT, or (b) every child has died.
Failure policy (eventual): on unexpected death, the supervisor
restarts the daemon and emits a notification to the supervise
sidecar so the operator sees the event. That lands in a later
PR; the interim policy is "don't take the bundle down for one
daemon so the operator sees the event. That lands in a later
PR; the interim policy is "don't take the gateway down for one
sick daemon."
Daemon subset is env-driven. The compose renderer narrows it via
`BOT_BOTTLE_SIDECAR_DAEMONS=egress` for bottles that
don't use git-gate or supervise. Default: all daemons.
Daemon subset is env-driven via `BOT_BOTTLE_GATEWAY_DAEMONS=egress`
for callers that don't use git-gate or supervise. Default: all
daemons.
Stdlib-only by design adding supervisord/s6/runit for four
daemons is heavier than this script.
@@ -103,8 +103,8 @@ def _selected_daemons(
env: dict[str, str],
all_daemons: Sequence[_DaemonSpec] | None = None,
) -> tuple[_DaemonSpec, ...]:
"""Filter the daemon set by the BOT_BOTTLE_SIDECAR_DAEMONS env
var. Unknown names in the list are ignored the renderer is the
"""Filter the daemon set by the BOT_BOTTLE_GATEWAY_DAEMONS env
var. Unknown names in the list are ignored the caller is the
source of truth for which daemons are wired.
`all_daemons` defaults to `_DAEMONS` resolved at call time (not
@@ -112,7 +112,7 @@ def _selected_daemons(
`_DAEMONS` and have the new value take effect."""
if all_daemons is None:
all_daemons = _DAEMONS
raw = env.get("BOT_BOTTLE_SIDECAR_DAEMONS", "").strip()
raw = env.get("BOT_BOTTLE_GATEWAY_DAEMONS", "").strip()
if not raw:
return tuple(all_daemons)
wanted = {n.strip() for n in raw.split(",") if n.strip()}
@@ -120,7 +120,7 @@ def _selected_daemons(
def _log(msg: str) -> None:
sys.stdout.write(f"sidecar-init: {msg}\n")
sys.stdout.write(f"gateway-init: {msg}\n")
sys.stdout.flush()
+4 -4
View File
@@ -1,6 +1,6 @@
"""Per-agent git-gate (PRD 0008).
A third per-agent sidecar that fronts the bottle's declared git
A third per-agent daemon that fronts the bottle's declared git
upstreams as a transparent mirror. Each `bottle.git` entry maps to
a bare repo on the gate; `git daemon` serves the bare repos over
`git://<gate>/<name>.git`. Two hooks make the mirror bidirectional:
@@ -15,7 +15,7 @@ a bare repo on the gate; `git daemon` serves the bare repos over
The agent never sees the upstream credential under either path.
Why a separate sidecar (not folded into egress or ssh-gate): the
Why a separate daemon (not folded into egress or ssh-gate): the
gate is the only one of the three that holds upstream push
credentials. Mixing it with egress would put push creds in the
same blast radius as internet-facing TLS interception; mixing it
@@ -23,7 +23,7 @@ with ssh-gate would force ssh-gate above L4 and into git-protocol
land. See `docs/prds/0008-git-gate.md`.
This module defines the abstract gate (`GitGate`) and its plan
dataclass (`GitGatePlan`). The sidecar's start/stop lifecycle is
dataclass (`GitGatePlan`). The gateway's start/stop lifecycle is
backend-specific and lives on concrete subclasses (see
`bot_bottle/backend/docker/git_gate.py`)."""
@@ -86,7 +86,7 @@ class GitGatePlan:
class GitGate(ABC):
"""The per-agent git-gate. Encapsulates the host-side prepare
(upstream lift + entrypoint/hook render); the sidecar's
(upstream lift + entrypoint/hook render); the gateway's
start/stop lifecycle is backend-specific and lives on concrete
subclasses."""
+2 -1
View File
@@ -13,6 +13,7 @@ import dataclasses
from pathlib import Path
from typing import TYPE_CHECKING
from .bottle_state import globalize_slug
from .errors import MissingEnvVarError
from .log import info
from .manifest import ManifestBottle, ManifestGitEntry
@@ -46,7 +47,7 @@ def _provision_dynamic_key(
owner_repo = entry.UpstreamPath
if owner_repo.endswith(".git"):
owner_repo = owner_repo[:-4]
title = f"bot-bottle:{slug}:{entry.Name}"
title = f"bot-bottle:{globalize_slug(slug)}:{entry.Name}"
info(f"provisioning deploy key for git-gate.repos[{entry.Name!r}]")
key_id, private_key_bytes = provisioner.create(owner_repo, title)
+16 -3
View File
@@ -1,7 +1,7 @@
"""Pure host-side rendering for the per-agent git-gate (PRD 0008).
Builds the agent's `.gitconfig` insteadOf rewrites, the known_hosts
line, and the entrypoint / pre-receive / access-hook scripts the sidecar
line, and the entrypoint / pre-receive / access-hook scripts the gateway
runs. No docker or forge calls exposed for tests and reuse across
backends. Split out of `git_gate.py` so the control surface (`GitGate`)
and the deploy-key lifecycle (`git_gate_provision`) each read on their
@@ -16,9 +16,12 @@ from pathlib import Path
from .manifest import ManifestBottle, ManifestGitEntry
# Short network alias for git-gate inside the sidecar bundle. The
# Short network alias for git-gate inside the gateway. The
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
GIT_GATE_HOSTNAME = "git-gate"
# App-layer identity token header the agent's git sends to git-http and the
# gateway validates (mirrors egress_addon / git_http_backend IDENTITY_HEADER).
IDENTITY_HEADER = "x-bot-bottle-identity"
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
# git daemon (--timeout/--init-timeout), the access-hook subprocess in
# git_http_backend, and the git http-backend CGI subprocess.
@@ -38,7 +41,7 @@ class GitGateUpstream:
KnownHostKey string from the manifest; the gate's start step
materialises it into a known_hosts file if non-empty.
the gate credential paths inside the running sidecar."""
the gate credential paths inside the running gateway."""
name: str
upstream_url: str
@@ -75,6 +78,7 @@ def _gitconfig_validate_value(field: str, value: str) -> None:
def git_gate_render_gitconfig(
entries: tuple[ManifestGitEntry, ...], gate_host: str, *, scheme: str = "git",
identity_token: str = "",
) -> str:
"""Render the agent's ~/.gitconfig content for git-gate
`insteadOf` rewrites. Pure host-side, no docker / VM;
@@ -96,6 +100,15 @@ def git_gate_render_gitconfig(
"# the upstream bidirectionally (gitleaks-scanned push;\n",
"# fetch-from-upstream-before-every-upload-pack via access-hook).\n",
]
# Over the smart-HTTP transport (VM backends), attach the per-bottle
# identity token as a request header on requests to the gate, scoped to
# its URL so it never goes to any other remote. git-http requires it (the
# gateway's mandatory (source_ip, token) attribution). git:// (single-tenant
# docker) carries no header — attribution there is the network alias.
if identity_token and scheme == "http":
_gitconfig_validate_value("identity_token", identity_token)
out.append(f'[http "http://{gate_host}/"]\n')
out.append(f"\textraHeader = {IDENTITY_HEADER}: {identity_token}\n")
for entry in entries:
_gitconfig_validate_value(f"repos[{entry.Name!r}].url", entry.Upstream)
out.append(f'[url "{scheme}://{gate_host}/{entry.Name}.git"]\n')
+5 -5
View File
@@ -2,7 +2,7 @@
Used where `git://` push traffic over a host-published Docker port can
hang before receive-pack reaches hooks (e.g. the firecracker backend,
where the guest reaches the sidecar over the point-to-point TAP). The
where the guest reaches the gateway over the point-to-point TAP). The
wrapper serves the same `/git/*.git` bare repos through
`git http-backend`, so pre-receive and upstream forwarding remain the
git-gate enforcement point.
@@ -27,8 +27,8 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlsplit
# policy_resolver ships flat alongside this file in the sidecar bundle
# image (see Dockerfile.sidecars); the bot_bottle.* fallback is the
# policy_resolver ships flat alongside this file in the gateway
# image (see Dockerfile.gateway); the bot_bottle.* fallback is the
# host-side / test path. Mirrors egress_addon's import shape.
try:
from policy_resolver import ( # type: ignore[import-not-found]
@@ -103,8 +103,8 @@ def resolve_sandbox_root(
return namespace
# Mirrors git_gate_render.GIT_GATE_TIMEOUT_SECS. Duplicated rather than
# imported: this module ships as a flat top-level sibling in the sidecar
# bundle image (see Dockerfile.sidecars), not as part of the bot_bottle
# imported: this module ships as a flat top-level sibling in the gateway
# bundle image (see Dockerfile.gateway), not as part of the bot_bottle
# package, so `bot_bottle.git_gate` and its dependency chain aren't
# available at runtime.
GIT_GATE_TIMEOUT_SECS = 15
+2 -2
View File
@@ -17,7 +17,7 @@ class ManifestAgentProvider:
`template` selects a built-in launch/runtime contract. `dockerfile`
optionally points at a custom agent-image Dockerfile while leaving
bot-bottle's sidecar infrastructure intact.
bot-bottle's gateway infrastructure intact.
`auth_token` names the host env var that holds the provider's OAuth
token (Claude only). The provisioner injects a provider-owned egress
@@ -26,7 +26,7 @@ class ManifestAgentProvider:
so the Claude Code CLI starts.
`forward_host_credentials` forwards the host Codex auth token into
the egress sidecar (Codex only).
the egress daemon (Codex only).
"""
template: str = "claude"
+4 -4
View File
@@ -39,10 +39,10 @@ class ManifestBottle:
# identity without any git-gate.repos upstreams, and vice versa.
git_user: ManifestGitUser = field(default_factory=ManifestGitUser)
egress: ManifestEgressConfig = field(default_factory=ManifestEgressConfig)
# Per-bottle stuck-recovery sidecar (PRD 0013). When true (the
# Per-bottle stuck-recovery daemon (PRD 0013). When true (the
# default, issue #249), the launch step brings up a supervise
# sidecar that exposes egress MCP tools to the agent. Set
# `supervise: false` to skip the sidecar.
# daemon that exposes egress MCP tools to the agent. Set
# `supervise: false` to skip the gateway.
supervise: bool = True
@classmethod
@@ -61,7 +61,7 @@ class ManifestBottle:
raise ManifestError(
f"bottle '{name}' has an 'ssh' field, which has been removed "
f"(PRD 0009). Declare upstreams under 'git-gate.repos' with "
f"url + identity + host_key; the git-gate sidecar (PRD 0008) "
f"url + identity + host_key; the git-gate daemon (PRD 0008) "
f"holds the credential and gitleaks-scans pushes."
)
+14 -2
View File
@@ -16,6 +16,7 @@ import secrets
from pathlib import Path
from .. import log
from ..store_manager import StoreManager
from .broker import LaunchBroker, StubBroker
from .control_plane import make_server
from .docker_broker import DockerBroker
@@ -39,19 +40,30 @@ def main(argv: list[str] | None = None) -> int:
)
parser.add_argument(
"--gateway", action="store_true",
help="run one consolidated per-host sidecar bundle (build-if-missing)",
help="run one consolidated per-host gateway (build-if-missing)",
)
args = parser.parse_args(argv)
registry = RegistryStore(args.db)
registry.migrate()
# One DB per host: the supervise queue + audit tables live in the SAME
# SQLite file the registry owns, so the control plane is the single
# source of truth. The in-VM supervise daemon writes here; the host
# operator reaches it over HTTP (never a second, disconnected DB).
StoreManager(registry.db_path).migrate()
# An ephemeral signing secret ties the orchestrator (signer) to its
# broker (verifier). 'stub' records launches instead of starting
# anything; 'docker' runs real containers (firecracker drops in later).
secret = secrets.token_bytes(32)
broker: LaunchBroker = DockerBroker(secret) if args.broker == "docker" else StubBroker(secret)
gateway: Gateway | None = DockerGateway() if args.gateway else None
# Standalone `--gateway` (not the consolidated flow, where the host
# lifecycle runs the gateway). The gateway resolves against this same
# process; the URL is only reachable when they share a docker network.
gateway: Gateway | None = (
DockerGateway(orchestrator_url=f"http://{args.host}:{args.port}")
if args.gateway else None
)
orchestrator = Orchestrator(registry, broker, secret, gateway)
# One persistent per-host gateway, shared by every bottle: build the
+74 -2
View File
@@ -90,14 +90,18 @@ class OrchestratorClient:
image_ref: str = "",
metadata: str = "",
policy: str = "",
tokens: dict[str, str] | None = None,
) -> RegisteredBottle:
"""Register a bottle and broker its launch (`POST /bottles`). Returns
its minted id + identity token."""
"""Register a bottle and broker its launch (`POST /bottles`). `tokens`
are the per-bottle egress auth values (env_name -> value) the
orchestrator holds in memory for the gateway to inject. Returns the
minted id + identity token."""
payload = self._ok("POST", "/bottles", {
"source_ip": source_ip,
"image_ref": image_ref,
"metadata": metadata,
"policy": policy,
"tokens": tokens or {},
})
bottle_id = payload.get("bottle_id")
token = payload.get("identity_token")
@@ -131,10 +135,78 @@ class OrchestratorClient:
bottles = payload.get("bottles")
return bottles if isinstance(bottles, list) else []
# --- supervise queue (operator TUI) ------------------------------------
def supervise_pending(self) -> list[dict[str, object]]:
"""Pending supervise proposals across all bottles
(`GET /supervise/proposals`)."""
payload = self._ok("GET", "/supervise/proposals")
proposals = payload.get("proposals")
return proposals if isinstance(proposals, list) else []
def supervise_respond(
self,
proposal_id: str,
*,
bottle_slug: str,
decision: str,
notes: str = "",
final_file: str | None = None,
) -> None:
"""Record an operator decision (`POST /supervise/respond`). `decision`
is approve/modify/reject. Raises `OrchestratorClientError` if the
proposal is gone or the bottle can no longer be applied to (409)."""
body: dict[str, object] = {
"proposal_id": proposal_id,
"bottle_slug": bottle_slug,
"decision": decision,
"notes": notes,
}
if final_file is not None:
body["final_file"] = final_file
self._ok("POST", "/supervise/respond", body)
def discover_orchestrator_url(*, timeout: float = 2.0) -> str:
"""The URL of the one running per-host orchestrator control plane, probing
the backends' well-known control-plane addresses (both on port 8099):
docker publishes it on loopback; the firecracker infra VM serves it on the
orchestrator TAP. Returns the first that answers `/health`; raises if none
do (no orchestrator up launch a bottle first)."""
candidates: list[str] = []
try: # docker: loopback-published control plane
from .lifecycle import DEFAULT_PORT as _DOCKER_PORT
candidates.append(f"http://127.0.0.1:{_DOCKER_PORT}")
except Exception: # noqa: BLE001 — backend optional
candidates.append("http://127.0.0.1:8099")
try: # firecracker: infra VM control plane on the orchestrator TAP
from ..backend.firecracker import netpool
from ..backend.firecracker.infra_vm import CONTROL_PLANE_PORT
candidates.append(
f"http://{netpool.orch_slot().guest_ip}:{CONTROL_PLANE_PORT}")
except Exception: # noqa: BLE001 — backend optional / not firecracker
pass
try: # macOS: infra container control plane on its host-only address
from ..backend.macos_container.infra import probe_control_plane_url
url = probe_control_plane_url()
if url:
candidates.append(url)
except Exception: # noqa: BLE001 — backend optional / not macOS
pass
for url in candidates:
if OrchestratorClient(url, timeout=timeout).health():
return url
raise OrchestratorClientError(
"no running orchestrator control plane found (tried "
+ ", ".join(candidates)
+ "); launch a bottle first"
)
__all__ = [
"OrchestratorClient",
"OrchestratorClientError",
"RegisteredBottle",
"DEFAULT_TIMEOUT_SECONDS",
"discover_orchestrator_url",
]
+63 -5
View File
@@ -16,6 +16,10 @@ vsock / unix-socket portability caveats):
POST /attribute -> 200 {"bottle_id"} | 403
POST /resolve -> 200 {"bottle_id","policy"} | 403
body: {"source_ip","identity_token"}
GET /supervise/proposals -> 200 {"proposals": [ <proposal>, ...]}
POST /supervise/respond -> 200 {"responded": true} | 409 (operator)
body: {"proposal_id","bottle_slug",
"decision", ["notes"],["final_file"]}
`POST /bottles` / `DELETE` drive the full launch lifecycle: they mint (or
tear down) the bottle in the registry AND broker the backend-native launch
@@ -34,6 +38,7 @@ import http.server
import json
import os
import socketserver
import sys
import typing
from urllib.parse import urlsplit
@@ -80,11 +85,16 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
image_ref = data.get("image_ref")
metadata = data.get("metadata")
policy = data.get("policy")
raw_tokens = data.get("tokens")
tokens = {
k: v for k, v in raw_tokens.items() if isinstance(k, str) and isinstance(v, str)
} if isinstance(raw_tokens, dict) else {}
rec = orch.launch_bottle(
source_ip,
image_ref=image_ref if isinstance(image_ref, str) else "",
metadata=metadata if isinstance(metadata, str) else "",
policy=policy if isinstance(policy, str) else "",
tokens=tokens,
)
return 201, {"bottle_id": rec.bottle_id, "identity_token": rec.identity_token}
@@ -121,10 +131,44 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
return 403, {"error": "unattributed"}
return 200, {"bottle_id": rec.bottle_id}
if method == "GET" and route == "/supervise/proposals":
# Operator TUI: pending supervise proposals across all bottles.
return 200, {"proposals": orch.supervise_pending()}
if method == "POST" and route == "/supervise/respond":
# Operator decision: apply (approve/modify rewrites egress policy),
# write the queued response, audit — all server-side on the one DB.
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
proposal_id = data.get("proposal_id")
bottle_slug = data.get("bottle_slug")
decision = data.get("decision")
if not (isinstance(proposal_id, str) and proposal_id):
return 400, {"error": "proposal_id (string) is required"}
if not (isinstance(bottle_slug, str) and bottle_slug):
return 400, {"error": "bottle_slug (string) is required"}
if not (isinstance(decision, str) and decision):
return 400, {"error": "decision (string) is required"}
notes = data.get("notes")
final_file = data.get("final_file")
ok, err = orch.supervise_respond(
proposal_id,
bottle_slug=bottle_slug,
decision=decision,
notes=notes if isinstance(notes, str) else "",
final_file=final_file if isinstance(final_file, str) else None,
)
if ok:
return 200, {"responded": True}
return 409, {"error": err}
if method == "POST" and route == "/resolve":
# The per-request lookup the multi-tenant gateway makes: returns the
# bottle's policy. identity_token is OPTIONAL — absent means resolve
# by source IP alone (network-layer attribution).
# bottle's policy. Requires a matching (source_ip, identity_token)
# pair — a missing/empty/mismatched token fail-closes (403), no
# source-IP-only fallback.
try:
data = _parse_json_object(body)
except ValueError as e:
@@ -136,7 +180,13 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
rec = orch.resolve(source_ip, token if isinstance(token, str) else "")
if rec is None:
return 403, {"error": "unattributed"}
return 200, {"bottle_id": rec.bottle_id, "policy": rec.policy}
# tokens are the in-memory per-bottle egress auth values the gateway
# injects; served here, never persisted.
return 200, {
"bottle_id": rec.bottle_id,
"policy": rec.policy,
"tokens": orch.tokens_for(rec.bottle_id),
}
return 404, {"error": "not found"}
@@ -151,12 +201,20 @@ class Handler(http.server.BaseHTTPRequestHandler):
super().log_message(format, *args)
def _serve(self, method: str) -> None:
"""Read the request body, dispatch it, and write the JSON reply."""
"""Read the request body, dispatch it, and write the JSON reply. A
dispatch failure (e.g. a broker error) returns a 500 rather than
crashing the connection, so one bad request can't take the control
plane down for the caller."""
server = self.server
assert isinstance(server, ControlPlaneServer)
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length > 0 else b""
status, payload = dispatch(server.orchestrator, method, self.path, body)
try:
status, payload = dispatch(server.orchestrator, method, self.path, body)
except Exception as e: # noqa: BLE001 — the control plane must stay up
sys.stderr.write(f"orchestrator: {method} {self.path} failed: {e!r}\n")
sys.stderr.flush()
status, payload = 500, {"error": f"internal error: {e}"}
data = json.dumps(payload).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
+2 -2
View File
@@ -2,12 +2,12 @@
On a verified launch request it starts a Docker container; on teardown it
removes it. This proves the orchestrator -> backend seam on the cheapest
backend (the sidecar bundle is already containers). Only the request's
backend (the gateway is already containers). Only the request's
static ids/flags reach `docker`, so nothing free-form crosses the boundary.
Slice 3 launches a single container from the request's `image_ref`, named
after the bottle id and labelled for cleanup. Wiring the full agent +
sidecar bundle (networks, mounts, the consolidated sidecar) is a later
gateway (networks, mounts, the consolidated gateway) is a later
slice this is the seam, not the finished launcher.
"""
+119 -24
View File
@@ -1,7 +1,7 @@
"""The consolidated per-host gateway (PRD 0070).
The core consolidation win: **one** persistent gateway per host, shared by
every bottle, instead of a sidecar bundle per bottle. It's safe to share
every bottle, instead of a gateway per bottle. It's safe to share
because the attribution invariant (source IP + identity token, see
`registry`) lets the gateway attribute each request to the right bottle
so per-bottle policy lives in one long-lived process keyed on who's calling.
@@ -19,9 +19,22 @@ from __future__ import annotations
import abc
import os
import time
from pathlib import Path
from ..docker_cmd import run_docker
from ..paths import host_db_path
from ..supervise import DB_PATH_IN_CONTAINER
# The host DB dir is bind-mounted here so the gateway's supervise daemon
# writes its queued proposals into the ONE host DB (the same file the
# orchestrator container opens and the operator reaches over HTTP).
_SUPERVISE_DB_DIR_IN_CONTAINER = os.path.dirname(DB_PATH_IN_CONTAINER)
# The gateway's mitmproxy writes its CA a beat after the container starts, so
# reads poll for it rather than assuming it's there on a fresh launch.
_CA_POLL_SECONDS = 0.5
DEFAULT_CA_TIMEOUT_SECONDS = 30.0
GATEWAY_NAME = "bot-bottle-orch-gateway"
GATEWAY_LABEL = "bot-bottle-orch-gateway=1"
@@ -31,15 +44,31 @@ GATEWAY_LABEL = "bot-bottle-orch-gateway=1"
# the source IP the gateway attributes by is the address on this network.
GATEWAY_NETWORK = "bot-bottle-gateway"
# The real sidecar-bundle image + its Dockerfile. Kept as a local constant
# rather than imported from backend.docker.sidecar_bundle, which would drag
# mitmproxy's CA dir in the bundle. A persistent named volume here keeps the
# gateway's self-generated CA STABLE across container recreation — every agent
# installs this one CA to trust the shared gateway's TLS interception, so it
# must not rotate when the gateway restarts.
MITMPROXY_HOME = "/home/mitmproxy/.mitmproxy"
GATEWAY_CA_VOLUME = "bot-bottle-gateway-mitmproxy"
GATEWAY_CA_CERT = f"{MITMPROXY_HOME}/mitmproxy-ca-cert.pem"
# The gateway data-plane image + its Dockerfile. Kept as a local constant
# rather than imported from the backend layer, which would drag
# the whole backend layer into the lean orchestrator (see #359); unify when
# that lands. Env override matches the backend's BOT_BOTTLE_SIDECAR_IMAGE.
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_SIDECAR_IMAGE", "bot-bottle-sidecars:latest")
GATEWAY_DOCKERFILE = "Dockerfile.sidecars"
# that lands. Env override matches the backend's BOT_BOTTLE_GATEWAY_IMAGE.
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
GATEWAY_DOCKERFILE = "Dockerfile.gateway"
_REPO_ROOT = Path(__file__).resolve().parents[2]
def _host_db_dir() -> str:
"""The host DB directory (created if missing), for the gateway's
supervise-DB bind-mount."""
db_dir = host_db_path().parent
db_dir.mkdir(parents=True, exist_ok=True)
return str(db_dir)
class GatewayError(Exception):
"""The shared gateway failed to build/start/stop (non-zero `docker` exit)."""
@@ -72,8 +101,8 @@ class Gateway(abc.ABC):
class DockerGateway(Gateway):
"""The consolidated gateway as a single, fixed-name Docker container.
`image_ref` defaults to the real sidecar-bundle image; `ensure_built`
builds it from `Dockerfile.sidecars` when it's missing. (Note: slice 5
`image_ref` defaults to the gateway data-plane image; `ensure_built`
builds it from `Dockerfile.gateway` when it's missing. (Note: slice 5
builds + launches the bundle container; wiring its per-bottle,
source-IP-keyed config is a later slice see PRD 0070.)"""
@@ -83,30 +112,46 @@ class DockerGateway(Gateway):
*,
name: str = GATEWAY_NAME,
network: str = GATEWAY_NETWORK,
orchestrator_url: str = "",
build_context: Path | None = None,
dockerfile: str | None = GATEWAY_DOCKERFILE,
host_port_bindings: tuple[int, ...] = (),
) -> None:
self.image_ref = image_ref
self.name = name
self.network = network
# The control-plane URL the gateway's data plane resolves per bottle
# against — reached by container name over docker DNS on the shared
# network (container↔container, no host firewall). Empty → single-tenant.
self._orchestrator_url = orchestrator_url
self._build_context = build_context or _REPO_ROOT
self._dockerfile = dockerfile
# Ports published on the host (0.0.0.0). Used by the Firecracker
# backend's dev-harness gateway so VMs can reach it via their TAP link;
# Docker's DNAT + the nft `ct status dnat accept` rule handle the rest.
self._host_port_bindings = host_port_bindings
def image_exists(self) -> bool:
return run_docker(["docker", "image", "inspect", self.image_ref]).returncode == 0
def ensure_built(self) -> None:
"""Build the bundle image from its Dockerfile when it's missing.
No-op when the image is already present, or when no dockerfile is
configured (e.g. a pre-pulled image)."""
if self._dockerfile is None or self.image_exists():
"""Build the bundle image from its Dockerfile, **cache-aware** — cheap
(a cache check) when nothing changed, a real rebuild when the flat
sources (egress addon / git-http / policy_resolver / supervise) moved.
This deliberately builds every time rather than build-if-missing: the
per-bottle model kept the image fresh via compose's `build:` on up, and
a stale image silently runs the OLD single-tenant daemons. No-op only
when no dockerfile is configured (a pre-pulled image). BOT_BOTTLE_NO_CACHE
forces a full rebuild (parity with `start --no-cache`)."""
if self._dockerfile is None:
return
proc = run_docker([
"docker", "build",
"-t", self.image_ref,
"-f", str(self._build_context / self._dockerfile),
str(self._build_context),
])
argv = ["docker", "build", "-t", self.image_ref,
"-f", str(self._build_context / self._dockerfile),
str(self._build_context)]
if os.environ.get("BOT_BOTTLE_NO_CACHE"):
argv.insert(2, "--no-cache")
proc = run_docker(argv)
if proc.returncode != 0:
raise GatewayError(f"gateway image build failed: {proc.stderr.strip()}")
@@ -119,6 +164,18 @@ class DockerGateway(Gateway):
])
return self.name in proc.stdout.split()
def _running_image_is_current(self) -> bool:
"""True iff the running gateway was created from the *current*
`image_ref`. When `ensure_built` rebuilds the image (a source change),
the running container is still the OLD image running the OLD flat
daemons so this is how a rebuild actually takes effect: a mismatch
means recreate."""
running = run_docker(["docker", "inspect", "--format", "{{.Image}}", self.name])
current = run_docker(["docker", "image", "inspect", "--format", "{{.Id}}", self.image_ref])
if running.returncode != 0 or current.returncode != 0:
return True # can't compare → don't churn a working container
return running.stdout.strip() == current.stdout.strip()
def _ensure_network(self) -> None:
"""Create the shared gateway network if it doesn't exist. Idempotent —
a concurrent create loses harmlessly (the loser sees 'already exists').
@@ -132,22 +189,59 @@ class DockerGateway(Gateway):
)
def ensure_running(self) -> None:
if self.is_running():
# Recreate when the running container's image is stale (a rebuild),
# so source changes to the gateway's flat daemons take effect — not
# just when the container is absent.
if self.is_running() and self._running_image_is_current():
return
self._ensure_network()
# Clear any stale (stopped) container holding the fixed name, then
# start fresh. `rm --force` on an absent name is a tolerated no-op.
# Clear any stale (stopped OR outdated-image) container holding the
# fixed name, then start fresh. `rm --force` on an absent name is a
# tolerated no-op.
run_docker(["docker", "rm", "--force", self.name])
proc = run_docker([
argv = [
"docker", "run", "--detach",
"--name", self.name,
"--label", GATEWAY_LABEL,
"--network", self.network,
self.image_ref,
])
# Persist the self-generated CA so it survives restarts (agents
# trust it) — see GATEWAY_CA_VOLUME.
"--volume", f"{GATEWAY_CA_VOLUME}:{MITMPROXY_HOME}",
# Share the one host DB: the supervise daemon queues proposals
# into the same file the orchestrator (and the operator, over
# HTTP) reads — no second, disconnected DB in the container.
"--volume", f"{_host_db_dir()}:{_SUPERVISE_DB_DIR_IN_CONTAINER}",
"--env", f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
]
for port in self._host_port_bindings:
argv += ["--publish", f"0.0.0.0:{port}:{port}"]
if self._orchestrator_url:
# Makes the gateway's egress / git / supervise daemons multi-tenant:
# each request resolves source-IP -> policy against the control plane.
argv += ["--env", f"BOT_BOTTLE_ORCHESTRATOR_URL={self._orchestrator_url}"]
argv.append(self.image_ref)
proc = run_docker(argv)
if proc.returncode != 0:
raise GatewayError(f"gateway failed to start: {proc.stderr.strip()}")
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
"""The gateway's CA certificate (PEM) that agents install to trust its
TLS interception. mitmproxy generates it a moment after the container
starts, so this **polls** for it (up to `timeout`) rather than assuming
it's already there on a fresh gateway — raising only if it never
appears."""
deadline = time.monotonic() + timeout
while True:
proc = run_docker(["docker", "exec", self.name, "cat", GATEWAY_CA_CERT])
if proc.returncode == 0 and proc.stdout.strip():
return proc.stdout
if time.monotonic() >= deadline:
raise GatewayError(
f"gateway CA cert not available after {timeout:g}s: "
f"{proc.stderr.strip() or 'empty'}"
)
time.sleep(_CA_POLL_SECONDS)
def stop(self) -> None:
proc = run_docker(["docker", "rm", "--force", self.name])
if proc.returncode != 0 and "No such container" not in proc.stderr:
@@ -157,4 +251,5 @@ class DockerGateway(Gateway):
__all__ = [
"Gateway", "DockerGateway", "GatewayError",
"GATEWAY_NAME", "GATEWAY_LABEL", "GATEWAY_IMAGE", "GATEWAY_NETWORK",
"GATEWAY_CA_VOLUME", "GATEWAY_CA_CERT",
]
+203 -83
View File
@@ -1,91 +1,240 @@
"""Orchestrator process lifecycle (PRD 0070, docker slice).
"""Orchestrator + gateway lifecycle (PRD 0070, docker slice).
Before the CLI can register or launch bottles against the consolidated
model, exactly one orchestrator control plane and the single per-host
gateway it manages must be running. This starts the orchestrator
dev-harness (`python -m bot_bottle.orchestrator`) as a background host
process and health-checks it.
Runs the orchestrator control plane **as a container** on the shared gateway
network, alongside the gateway container. This is the PRD's "virtualize the
orchestrator": container↔container between the gateway and the orchestrator
avoids the host firewall (which drops containerhost traffic), and the gateway
reaches the control plane by container name over docker DNS. The host CLI
reaches it via a published loopback port.
It is an **idempotent singleton**: `ensure_running` returns immediately if a
healthy control plane already answers on the port, and otherwise spawns one
and waits for it to come up. The control-plane port is the singleton key
a second orchestrator can't bind it, so a stray double-start fails fast
rather than forking a rival.
Host-process (not container) on purpose: the PRD sequences the orchestrator
as a plain-process dev-harness first (fast iteration, and it already has the
host user's docker access to broker launches), while the data-plane
*gateway* it manages runs as a container. Wrapping the orchestrator itself
in a backend-native unit is a later step.
The orchestrator runs with the **register-only broker** the *backend*
launches agent containers (compose), so the orchestrator needs no docker
socket. That keeps this control-plane container unprivileged; the host manages
both containers. `ensure_running` is an idempotent singleton (fixed container
names + the published port).
"""
from __future__ import annotations
import subprocess
import sys
import hashlib
import os
import time
import urllib.error
import urllib.request
from pathlib import Path
from .. import log
from ..docker_cmd import run_docker
from ..paths import bot_bottle_root
from .gateway import GATEWAY_IMAGE, GATEWAY_NETWORK, DockerGateway, GatewayError
DEFAULT_PORT = 8099
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
ORCHESTRATOR_LABEL = "bot-bottle-orchestrator=1"
# The control-plane's own runtime image — lean (python + the stdlib-only
# `bot_bottle` package, bind-mounted at run time), distinct from the heavy
# gateway data-plane image it used to borrow (#384). Env override for
# operators pinning a published build.
ORCHESTRATOR_IMAGE = os.environ.get(
"BOT_BOTTLE_ORCHESTRATOR_IMAGE", "bot-bottle-orchestrator:latest"
)
ORCHESTRATOR_DOCKERFILE = "Dockerfile.orchestrator"
# Baked onto the container as a label so `ensure_running` can tell whether the
# running process is executing the *current* bind-mounted source — see
# `source_hash`.
ORCHESTRATOR_SOURCE_HASH_LABEL = "bot-bottle-orchestrator-source-hash"
# The repo root is bind-mounted into the control-plane container so
# `python -m bot_bottle.orchestrator` resolves the package (the orchestrator
# is stdlib-only, so the lean orchestrator image's python is enough).
_REPO_ROOT = Path(__file__).resolve().parents[2]
_APP_DIR = "/app"
_ROOT_IN_CONTAINER = "/bot-bottle-root"
DEFAULT_HOST = "127.0.0.1"
DEFAULT_PORT = 8080
# Poll cadence + default ceiling while waiting for a freshly-spawned control
# plane to answer /health (the first start also builds/boots the gateway).
_HEALTH_POLL_SECONDS = 0.25
DEFAULT_STARTUP_TIMEOUT_SECONDS = 45.0
_HEALTH_REQUEST_TIMEOUT_SECONDS = 1.0
class OrchestratorStartError(RuntimeError):
"""The orchestrator process did not become healthy within the timeout."""
"""The orchestrator container did not become healthy within the timeout."""
class OrchestratorProcess:
"""Manages the local orchestrator control-plane process for the docker
backend. Backend-neutral callers only need `ensure_running()` + `url`."""
def source_hash(repo_root: Path) -> str:
"""Content hash of the orchestrator's bind-mounted Python source (the
`bot_bottle` package the control-plane process imports). This only
changes when the code that would actually run inside the container
changes `ensure_running` recreates the container on a mismatch and
otherwise leaves a healthy one alone, so a bottle launch that isn't
accompanied by a code change doesn't restart the process and drop every
*other* active bottle's in-memory egress tokens (`Orchestrator._tokens`
in `service.py`, never persisted to disk by design)."""
h = hashlib.sha256()
for path in sorted((repo_root / "bot_bottle").rglob("*.py")):
h.update(str(path.relative_to(repo_root)).encode())
h.update(path.read_bytes())
return h.hexdigest()
class OrchestratorService:
"""Manages the orchestrator control-plane container + the shared gateway.
Callers only need `ensure_running()` + `url`.
`orchestrator_name` / `orchestrator_label` let backends run independent
orchestrators on the same host without name collisions (e.g. the
Firecracker backend uses `bot-bottle-fc-orchestrator` alongside the Docker
backend's `bot-bottle-orchestrator`). Subclass and override `_gateway()`
to supply a backend-specific gateway variant."""
def __init__(
self,
host: str = DEFAULT_HOST,
port: int = DEFAULT_PORT,
*,
broker: str = "docker",
gateway: bool = True,
port: int = DEFAULT_PORT,
network: str = GATEWAY_NETWORK,
image: str = ORCHESTRATOR_IMAGE,
gateway_image: str = GATEWAY_IMAGE,
repo_root: Path = _REPO_ROOT,
host_root: Path | None = None,
orchestrator_name: str = ORCHESTRATOR_NAME,
orchestrator_label: str = ORCHESTRATOR_LABEL,
) -> None:
self.host = host
self.port = port
self._broker = broker
self._gateway = gateway
self.network = network
# Two distinct images (#384): `image` is the lean control-plane
# runtime this container runs; `_gateway_image` is the heavy egress /
# git-gate / supervise data plane the gateway container runs. They
# were one conflated image before the split.
self.image = image
self._gateway_image = gateway_image
self._repo_root = repo_root
self._host_root = host_root or bot_bottle_root()
self._orchestrator_name = orchestrator_name
self._orchestrator_label = orchestrator_label
@property
def url(self) -> str:
"""The control-plane base URL — also what the data plane's
BOT_BOTTLE_ORCHESTRATOR_URL points at."""
return f"http://{self.host}:{self.port}"
"""Host-side control-plane URL (published loopback port)."""
return f"http://127.0.0.1:{self.port}"
@property
def internal_url(self) -> str:
"""Control-plane URL as the gateway container reaches it — by name over
docker DNS on the shared network. This is the gateway's
BOT_BOTTLE_ORCHESTRATOR_URL."""
return f"http://{self._orchestrator_name}:{self.port}"
def is_healthy(self, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS) -> bool:
"""True iff a control plane answers `GET /health` with 200 — the
singleton liveness check."""
try:
with urllib.request.urlopen(f"{self.url}/health", timeout=timeout) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
def _container_running(self, name: str) -> bool:
proc = run_docker(["docker", "ps", "--filter", f"name=^/{name}$", "--format", "{{.Names}}"])
return name in proc.stdout.split()
def _run_orchestrator_container(self, current_hash: str) -> None:
"""Start the control-plane container (idempotent: clears a stale
fixed-name container first). Register-only broker no docker socket.
Labels the container with `current_hash` so a later `ensure_running`
can detect a real code change (see `source_hash`)."""
run_docker(["docker", "rm", "--force", self._orchestrator_name])
proc = run_docker([
"docker", "run", "--detach",
"--name", self._orchestrator_name,
"--label", self._orchestrator_label,
"--label", f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={current_hash}",
"--network", self.network,
# Host CLI reaches the control plane here; bound to loopback so it
# is not exposed on the host's external interfaces.
"--publish", f"127.0.0.1:{self.port}:{self.port}",
"--volume", f"{self._repo_root}:{_APP_DIR}:ro",
"--workdir", _APP_DIR,
# Persist the registry DB on the host (sole-owner: only the
# orchestrator opens bot-bottle.db).
"--volume", f"{self._host_root}:{_ROOT_IN_CONTAINER}",
"--env", f"BOT_BOTTLE_ROOT={_ROOT_IN_CONTAINER}",
"--entrypoint", "python3",
self.image,
"-m", "bot_bottle.orchestrator",
"--host", "0.0.0.0", "--port", str(self.port), "--broker", "stub",
])
if proc.returncode != 0:
raise OrchestratorStartError(
f"orchestrator container failed to start: {proc.stderr.strip()}"
)
def _gateway(self) -> DockerGateway:
return DockerGateway(
self._gateway_image, network=self.network, orchestrator_url=self.internal_url
)
def _ensure_orchestrator_image(self) -> None:
"""Build the lean control-plane image from `Dockerfile.orchestrator`
when it's missing (#384). Cheap — a `FROM python:*-slim` base with no
deps to install, so the layer cache makes rebuilds a no-op. Unlike the
gateway image this is build-if-missing, not build-every-time: the
control plane bind-mounts its source, so a code change is caught by the
source-hash recreate (below), not by an image rebuild."""
if run_docker(["docker", "image", "inspect", self.image]).returncode == 0:
return
argv = ["docker", "build", "-t", self.image,
"-f", str(self._repo_root / ORCHESTRATOR_DOCKERFILE),
str(self._repo_root)]
if os.environ.get("BOT_BOTTLE_NO_CACHE"):
argv.insert(2, "--no-cache")
proc = run_docker(argv)
if proc.returncode != 0:
raise GatewayError(
f"orchestrator image build failed: {proc.stderr.strip()}"
)
def _orchestrator_source_current(self, current_hash: str) -> bool:
"""True iff the running orchestrator container was created from the
*current* bind-mounted source. Mirrors `DockerGateway`'s
image-staleness check, but by content hash rather than image id since
the orchestrator runs bind-mounted source, not a built image."""
if not self._container_running(self._orchestrator_name):
return False
proc = run_docker([
"docker", "inspect", "--format",
"{{ index .Config.Labels \"" + ORCHESTRATOR_SOURCE_HASH_LABEL + "\" }}",
self._orchestrator_name,
])
if proc.returncode != 0:
return True # can't compare -> don't churn a working container
return proc.stdout.strip() == current_hash
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Return the control-plane URL, starting the orchestrator first if it
isn't already healthy. Idempotent — a healthy control plane is left
untouched. Raises `OrchestratorStartError` if a freshly-spawned one
doesn't answer within `startup_timeout`."""
if self.is_healthy():
"""Ensure the control plane + shared gateway are up; return the host
control-plane URL. Idempotent a healthy control plane running
current code and a running gateway are left untouched. Raises
`OrchestratorStartError` on timeout."""
gateway = self._gateway()
gateway.ensure_built() # rebuild the bundle image on a source change
gateway.ensure_running() # creates the shared network + (re)starts gateway
# Recreate the orchestrator container only when its bind-mounted
# source has actually changed since it started — its Python process
# loaded that code at startup and won't reload, so a stale container
# would keep running OLD control-plane code. Recreating on *every*
# launch (the prior behaviour) would drop every other active
# bottle's in-memory egress tokens each time a new bottle starts,
# since the orchestrator process holds them only in memory (#381).
current_hash = source_hash(self._repo_root)
if self.is_healthy() and self._orchestrator_source_current(current_hash):
return self.url
log.info("starting orchestrator", context={"url": self.url, "broker": self._broker})
self._spawn()
self._ensure_orchestrator_image()
log.info(
"starting orchestrator container",
context={"name": self._orchestrator_name},
)
self._run_orchestrator_container(current_hash)
deadline = time.monotonic() + startup_timeout
while time.monotonic() < deadline:
if self.is_healthy():
@@ -93,49 +242,20 @@ class OrchestratorProcess:
return self.url
time.sleep(_HEALTH_POLL_SECONDS)
raise OrchestratorStartError(
f"orchestrator at {self.url} did not become healthy within "
f"{startup_timeout:g}s"
f"orchestrator at {self.url} did not become healthy within {startup_timeout:g}s"
)
def _argv(self) -> list[str]:
"""`python -m bot_bottle.orchestrator ...` — static flags only."""
argv = [
sys.executable, "-m", "bot_bottle.orchestrator",
"--host", self.host, "--port", str(self.port),
"--broker", self._broker,
]
if self._gateway:
argv.append("--gateway")
return argv
def _log_path(self) -> str:
"""Where the detached orchestrator's stdout/stderr goes so a failed
start is diagnosable after the CLI has moved on."""
return str(bot_bottle_root() / "orchestrator.log")
def _spawn(self) -> None:
"""Launch the orchestrator detached so it outlives this CLI process,
with its output tee'd to a log file under the bot-bottle root."""
root = bot_bottle_root()
root.mkdir(parents=True, exist_ok=True)
logfile = open(self._log_path(), "a", encoding="utf-8") # noqa: SIM115 # pylint: disable=consider-using-with
try:
subprocess.Popen( # noqa: S603 # pylint: disable=consider-using-with
self._argv(),
stdout=logfile,
stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL,
start_new_session=True,
)
finally:
# The child inherits its own dup'd fd; this handle is ours to drop.
logfile.close()
def stop(self) -> None:
"""Remove the orchestrator + gateway containers (idempotent)."""
run_docker(["docker", "rm", "--force", self._orchestrator_name])
self._gateway().stop()
__all__ = [
"OrchestratorProcess",
"OrchestratorService",
"OrchestratorStartError",
"DEFAULT_HOST",
"ORCHESTRATOR_NAME",
"ORCHESTRATOR_IMAGE",
"DEFAULT_PORT",
"DEFAULT_STARTUP_TIMEOUT_SECONDS",
]
+1 -1
View File
@@ -5,7 +5,7 @@ registry: turns a prepared bottle's egress plan into the backend-neutral
inputs `Orchestrator.launch_bottle` takes the egress **policy** blob and
launch **metadata**.
The policy blob is the exact routes YAML the per-bottle egress sidecar used
The policy blob is the exact routes YAML the per-bottle egress daemon used
to read from a file; in the consolidated model the multi-tenant gateway's
`PolicyResolver` fetches it from the registry per request (keyed by source
IP) instead. Same render, so consolidated and single-tenant egress apply
+14 -1
View File
@@ -149,7 +149,15 @@ class RegistryStore(DbStore):
policy: str = "",
) -> BottleRecord:
"""Register (or replace) a bottle. Mints a bottle_id / identity token
when not supplied. Live this is the control-plane reload path."""
when not supplied. Live this is the control-plane reload path.
Supersedes any *other* active bottle at the same source IP first:
`by_source_ip` fail-closes on ambiguity (>1 active row None), so a
leftover row at a reused IP from an untorn-down bottle, crash
recovery, or a persisted DB would otherwise *brick* the new bottle's
attribution. The new registration is authoritative for its IP; the
stale rows go. INSERT OR REPLACE handles a same-`bottle_id` reload in
place (its own row is exempt from the sweep)."""
rec = BottleRecord(
bottle_id=bottle_id or secrets.token_hex(8),
source_ip=source_ip,
@@ -160,6 +168,11 @@ class RegistryStore(DbStore):
policy=policy,
)
with self._connect() as conn:
conn.execute(
"DELETE FROM orchestrator_bottles "
"WHERE source_ip = ? AND state = 'active' AND bottle_id != ?",
(rec.source_ip, rec.bottle_id),
)
conn.execute(
"INSERT OR REPLACE INTO orchestrator_bottles "
"(bottle_id, source_ip, identity_token, state, created_at, metadata, policy) "
+170 -13
View File
@@ -17,9 +17,37 @@ Launch lifecycle:
from __future__ import annotations
import json
from datetime import datetime, timezone
from .broker import LaunchBroker, LaunchRequest, sign_request
from .registry import BottleRecord, RegistryStore
from .gateway import Gateway
from ..supervise import (
AuditEntry,
COMPONENT_FOR_TOOL,
Response,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
list_all_pending_proposals,
read_proposal,
render_diff,
write_audit_entry,
write_response,
)
# Operator decision → Response.status. The apply half (egress tools) runs
# for approve/modify only.
_RESPOND_STATUS = {
"approve": STATUS_APPROVED,
"modify": STATUS_MODIFIED,
"reject": STATUS_REJECTED,
}
_APPLY_TOOLS = (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK)
class Orchestrator:
@@ -38,6 +66,12 @@ class Orchestrator:
self._broker = broker
self._secret = sign_secret
self._gateway = gateway
# Per-bottle egress auth tokens (env_name -> value), keyed by bottle_id.
# Held **in memory only** — never written to the registry DB — so the
# gateway can inject each bottle's upstream credential without secrets
# at rest. Lost on restart (re-launch re-registers them); the future
# SecretProvider (#355) replaces this with per-request minting.
self._tokens: dict[str, dict[str, str]] = {}
def launch_bottle(
self,
@@ -47,11 +81,14 @@ class Orchestrator:
slot: int | None = None,
metadata: str = "",
policy: str = "",
tokens: dict[str, str] | None = None,
) -> BottleRecord:
"""Register a bottle (with its gateway policy) and broker its launch.
Rolls the registry entry back if the launch doesn't take, so a
failure leaves no orphan."""
"""Register a bottle (with its gateway policy + in-memory egress auth
tokens) and broker its launch. Rolls the registry entry back if the
launch doesn't take, so a failure leaves no orphan."""
rec = self.registry.register(source_ip, metadata=metadata, policy=policy)
if tokens:
self._tokens[rec.bottle_id] = dict(tokens)
req = LaunchRequest(
op="launch",
bottle_id=rec.bottle_id,
@@ -66,6 +103,7 @@ class Orchestrator:
finally:
if not launched:
self.registry.deregister(rec.bottle_id)
self._tokens.pop(rec.bottle_id, None)
return rec
def teardown_bottle(self, bottle_id: str) -> bool:
@@ -76,28 +114,147 @@ class Orchestrator:
req = LaunchRequest(op="teardown", bottle_id=bottle_id, source_ip=rec.source_ip)
self._broker.submit(sign_request(req, self._secret))
self.registry.deregister(bottle_id)
self._tokens.pop(bottle_id, None)
return True
def tokens_for(self, bottle_id: str) -> dict[str, str]:
"""The bottle's in-memory egress auth tokens (env_name -> value), or
empty. The gateway injects these per request; they are never
persisted."""
return dict(self._tokens.get(bottle_id, {}))
def attribute(self, source_ip: str, identity_token: str) -> BottleRecord | None:
"""Fail-closed attribution (delegates to the registry)."""
return self.registry.attribute(source_ip, identity_token)
def resolve(self, source_ip: str, identity_token: str = "") -> BottleRecord | None:
"""Resolve the bottle behind a request — the source-IP-keyed lookup
the multi-tenant gateway makes per request; the returned record
carries its `policy`. With a token, full attribution (source IP +
token); without, network-layer attribution by source IP alone
(valid where the IP is unspoofable and the control plane is
gateway-only)."""
if identity_token:
return self.registry.attribute(source_ip, identity_token)
return self.registry.by_source_ip(source_ip)
def resolve(self, source_ip: str, identity_token: str) -> BottleRecord | None:
"""Resolve the bottle behind a request — the per-request lookup the
multi-tenant gateway makes; the returned record carries its `policy`.
**Mandatory pair**: requires a matching `(source_ip, identity_token)`
(constant-time). There is no source-IP-only fallback the app-layer
token is delivered on every attributed data plane (egress proxy
credentials, git-gate/supervise headers), so a missing or mismatched
token fail-closes. This keeps a spoofed source IP (which the /31 TAP
alone does not prevent) from selecting another bottle's policy/tokens
without also holding that bottle's unguessable token."""
return self.registry.attribute(source_ip, identity_token)
def set_policy(self, bottle_id: str, policy: str) -> bool:
"""Update a bottle's gateway policy in place (live reload). False if
the bottle is unknown."""
return self.registry.set_policy(bottle_id, policy)
# --- supervise queue (operator approvals) ------------------------------
#
# The orchestrator owns the single DB *and* the live policy, so operator
# decisions are applied here, server-side, and reached over HTTP by the
# host TUI (no direct-DB access, one path for every backend).
def supervise_pending(self) -> list[dict[str, object]]:
"""All pending proposals across bottles, FIFO, as JSON dicts
(`Proposal.to_dict`, round-trippable via `Proposal.from_dict`).
Each dict carries an extra `bottle_label`: the bottle's human slug
resolved from the registry (the proposal itself is keyed by the
orchestrator-assigned bottle_id, which is opaque to an operator). The
CLI renders the label but still responds against `bottle_slug`."""
out: list[dict[str, object]] = []
for p in list_all_pending_proposals():
d = p.to_dict()
d["bottle_label"] = self._label_for(p.bottle_slug)
out.append(d)
return out
def _label_for(self, bottle_slug: str) -> str:
"""The human slug recorded in registry metadata for a proposal's
bottle, or the bottle_slug unchanged when the bottle is gone or has no
recorded slug so the label is always non-empty."""
rec = self.registry.get(bottle_slug)
if rec is None:
return bottle_slug
try:
meta = json.loads(rec.metadata) if rec.metadata else {}
except ValueError:
meta = {}
slug = meta.get("slug") if isinstance(meta, dict) else None
return slug if isinstance(slug, str) and slug else bottle_slug
def _record_for_slug(self, slug: str) -> BottleRecord | None:
"""The live registry record for a proposal's bottle, or None (e.g. the
bottle was torn down before the operator responded).
In consolidated mode the supervise server attributes each proposal to
the orchestrator-assigned bottle_id and stores that as the proposal's
`bottle_slug` (see supervise_server `_attributed_config`), so the fast
path is a direct bottle_id lookup. The metadata-slug scan is the
fallback for legacy single-tenant proposals keyed by the human slug."""
rec = self.registry.get(slug)
if rec is not None:
return rec
for rec in self.registry.all():
try:
meta = json.loads(rec.metadata) if rec.metadata else {}
except ValueError:
meta = {}
if isinstance(meta, dict) and meta.get("slug") == slug:
return rec
return None
def supervise_respond(
self,
proposal_id: str,
*,
bottle_slug: str,
decision: str,
notes: str = "",
final_file: str | None = None,
) -> tuple[bool, str]:
"""Record an operator decision on a queued proposal, applying it
server-side. `decision` is approve/modify/reject.
Approve/modify on an egress tool rewrites the bottle's policy so the
gateway serves the new routes on its next `/resolve` (the live apply);
then the queued Response is written (unblocking the agent's MCP call)
and an audit entry recorded all against the one DB. Returns
(ok, error): ok=False with a message when the proposal or decision is
unknown, or the bottle is gone so an approval can't be applied."""
status = _RESPOND_STATUS.get(decision)
if status is None:
return False, f"unknown decision {decision!r}"
try:
proposal = read_proposal(bottle_slug, proposal_id)
except FileNotFoundError:
return False, "no such proposal"
diff_before, diff_after = "", ""
if status in (STATUS_APPROVED, STATUS_MODIFIED) and proposal.tool in _APPLY_TOOLS:
new_policy = final_file if final_file is not None else proposal.proposed_file
rec = self._record_for_slug(bottle_slug)
if rec is None:
return False, (
f"bottle {bottle_slug!r} is no longer registered; "
"cannot apply the route change"
)
diff_before, diff_after = rec.policy, new_policy
self.set_policy(rec.bottle_id, new_policy)
write_response(bottle_slug, Response(
proposal_id=proposal_id, status=status, notes=notes, final_file=final_file,
))
component = COMPONENT_FOR_TOOL.get(proposal.tool)
if component is not None:
write_audit_entry(AuditEntry(
timestamp=datetime.now(timezone.utc).isoformat(),
bottle_slug=bottle_slug,
component=component,
operator_action=status,
operator_notes=notes,
justification=proposal.justification,
diff=render_diff(diff_before, diff_after, label=component),
))
return True, ""
# --- consolidated gateway ----------------------------------------------
def ensure_gateway(self) -> None:
+12 -3
View File
@@ -10,7 +10,7 @@ suite points it at a throwaway dir instead of monkey-patching the function
override covers them all), and operators can relocate the root if needed.
This module has no bot-bottle imports, so it is safe to import from any
layer (and to COPY flat into the sidecar bundle).
layer (and to COPY flat into the gateway).
"""
from __future__ import annotations
@@ -35,9 +35,18 @@ def host_db_path() -> Path:
Kept in its own `db/` subdirectory (not directly under the root) so a
backend that can only bind-mount *directories* can share this one file
with a sidecar without exposing the root's other contents (git-gate
with a gateway without exposing the root's other contents (git-gate
keys, per-bottle state, ...)."""
return bot_bottle_root() / "db" / HOST_DB_FILENAME
__all__ = ["HOST_DB_FILENAME", "bot_bottle_root", "host_db_path"]
def host_db_dir() -> Path:
"""The directory holding the shared host state DB, created if missing.
Backends bind-mount this into their gateway so the supervise daemon writes
to the one DB the orchestrator (and the operator over HTTP) reads."""
db_dir = host_db_path().parent
db_dir.mkdir(parents=True, exist_ok=True)
return db_dir
__all__ = ["HOST_DB_FILENAME", "bot_bottle_root", "host_db_path", "host_db_dir"]
+14 -9
View File
@@ -23,7 +23,7 @@ closed too rather than silently serving stale or empty policy.
The resolved value is the policy blob the orchestrator stores verbatim; the
consumer parses it (e.g. the egress addon's `load_config`). This module is
stdlib-only and free of bot-bottle imports so it can be COPYed flat into
the sidecar bundle.
the gateway.
"""
from __future__ import annotations
@@ -102,21 +102,26 @@ class PolicyResolver:
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = "",
) -> tuple[str | None, str | None]:
"""Both the policy blob and the bottle id in a single `/resolve` — so
a caller that needs each (the egress addon: policy for routing, bottle
id to key the calling bottle's supervise queue + safelist) makes one
round-trip, not two. Returns `(None, None)` when unattributed (a clean
`403`). Raises `PolicyResolveError` if the orchestrator can't be
reached, so the caller still fails closed."""
) -> tuple[str | None, str | None, dict[str, str]]:
"""The policy blob, bottle id, **and per-bottle egress auth tokens** in
a single `/resolve` so the egress addon gets everything it needs
(policy for routing, bottle id for the supervise queue/safelist, tokens
to inject upstream auth) in one round-trip. Returns `(None, None, {})`
when unattributed (a clean `403`). Raises `PolicyResolveError` if the
orchestrator can't be reached, so the caller still fails closed."""
payload = self._post_resolve(source_ip, identity_token)
if payload is None:
return None, None
return None, None, {}
policy = payload.get("policy")
bottle_id = payload.get("bottle_id")
raw = payload.get("tokens")
tokens = {
k: v for k, v in raw.items() if isinstance(k, str) and isinstance(v, str)
} if isinstance(raw, dict) else {}
return (
policy if isinstance(policy, str) else "",
bottle_id if isinstance(bottle_id, str) and bottle_id else None,
tokens,
)
+1 -1
View File
@@ -26,7 +26,7 @@ class QueueStore(DbStore):
if db_path is not None:
resolved = db_path
else:
# In the sidecar container SUPERVISE_DB_PATH points at the
# In the gateway container SUPERVISE_DB_PATH points at the
# bind-mounted host DB. On the host this env var is never set,
# so we always fall through to host_db_path().
env_path = os.environ.get("SUPERVISE_DB_PATH", "").strip()
+14 -14
View File
@@ -1,25 +1,25 @@
"""Per-bottle supervise plane (PRD 0013).
The supervise plane is the per-bottle MCP sidecar plus its host-side
queue/audit support. The sidecar (bot_bottle.supervise_server)
The supervise plane is the per-bottle MCP daemon plus its host-side
queue/audit support. The daemon (bot_bottle.supervise_server)
sits on the bottle's internal network and exposes MCP tools the agent
calls when it needs an operator-reviewed egress change:
* egress-block / allow agent proposes a new routes.yaml
Each tool call: the agent passes the full proposed file plus a
justification text. The sidecar validates the proposal syntactically,
justification text. The gateway validates the proposal syntactically,
writes it to the host SQLite queue table, and holds the tool-call
connection open. The operator's supervise TUI
(bot_bottle.cli.supervise) sees the proposal, accepts
approve / modify / reject, and writes a response row. The sidecar sees
approve / modify / reject, and writes a response row. The gateway sees
the response and returns `{status, notes}` to the agent.
This module defines the host-side library: dataclasses for the queue
record shapes, queue read/write helpers, the audit log writer, and the
diff renderer. The in-container sidecar lives in
diff renderer. The in-gateway daemon lives in
bot_bottle/supervise_server.py; the supervise daemon's container
lifecycle is owned by the sidecar bundle (PRD 0024).
lifecycle is owned by the gateway (PRD 0024).
For 0013 the supervisor's approval handlers are deliberately no-ops:
on approval the audit log is written and the response file is
@@ -75,18 +75,18 @@ except ImportError:
try:
from .paths import bot_bottle_root
except ImportError: # flat imports inside the sidecar bundle
except ImportError: # flat imports inside the gateway
from paths import bot_bottle_root # type: ignore[import-not-found,no-redef] # pylint: disable=import-error,no-name-in-module
SUPERVISE_HOSTNAME = "supervise"
SUPERVISE_PORT = 9100
# The supervise sidecar uses these to query egress's
# The supervise daemon uses these to query egress's
# introspection endpoint for the `list-egress-routes` MCP
# tool. The hostname + port match egress's docker network
# listen port (see backend.docker.egress.EGRESS_PORT). The supervise
# daemon runs inside the sidecar bundle alongside egress, so loopback
# daemon runs inside the gateway alongside egress, so loopback
# is the stable address across docker, firecracker, and Apple
# Container backends.
EGRESS_FORWARD_PROXY = "http://127.0.0.1:9099"
@@ -117,7 +117,7 @@ try:
from .audit_store import AuditStore
from .store_manager import StoreManager
except ImportError:
# Sidecar bundle: files are flat-copied under /app, not a package.
# Gateway: files are flat-copied under /app, not a package.
from queue_store import QueueStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from audit_store import AuditStore # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
from store_manager import StoreManager # type: ignore[import-not-found] # pylint: disable=import-error,no-name-in-module
@@ -222,14 +222,14 @@ def sha256_hex(content: str) -> str:
return hashlib.sha256(content.encode("utf-8")).hexdigest()
# --- Sidecar plan + abstract lifecycle -------------------------------------
# --- Gateway plan + abstract lifecycle -------------------------------------
@dataclass(frozen=True)
class SupervisePlan:
"""Output of Supervise.prepare; consumed by .start.
`db_path` is the host database bind-mounted into the sidecar at
`db_path` is the host database bind-mounted into the gateway at
/run/supervise/bot-bottle.db. `internal_network` is empty at
prepare time; the backend's launch step fills it via
dataclasses.replace before calling .start."""
@@ -240,8 +240,8 @@ class SupervisePlan:
class Supervise(ABC):
"""Per-bottle supervise sidecar. Encapsulates host-side database
staging; the sidecar's start/stop lifecycle is backend-specific."""
"""Per-bottle supervise daemon. Encapsulates host-side database
staging; the gateway's start/stop lifecycle is backend-specific."""
def prepare(
self,
+49 -5
View File
@@ -1,4 +1,4 @@
"""Supervise sidecar HTTP server (PRD 0013).
"""Supervise daemon HTTP server (PRD 0013).
Per-bottle MCP server exposing tools the agent calls to propose egress
config changes when stuck. The tools are `egress-allow`,
@@ -50,13 +50,17 @@ from dataclasses import dataclass, replace
try:
# Same-directory imports inside the bundle container; these files are
# COPYed flat under /app by Dockerfile.sidecars.
from egress_addon_core import LOG_OFF, load_config
# COPYed flat under /app by Dockerfile.gateway.
from egress_addon_core import (
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
)
from policy_resolver import PolicyResolveError, PolicyResolver
import supervise as _sv
except ModuleNotFoundError:
# Package imports for host-side tests and tooling.
from .egress_addon_core import LOG_OFF, load_config
from .egress_addon_core import (
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
)
from .policy_resolver import PolicyResolveError, PolicyResolver
from . import supervise as _sv
@@ -65,6 +69,8 @@ except ModuleNotFoundError:
MCP_PROTOCOL_VERSION = "2024-11-05"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
IDENTITY_HEADER = "x-bot-bottle-identity"
SERVER_NAME = "bot-bottle-supervise"
SERVER_VERSION = "0.1.0"
@@ -525,6 +531,17 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
if method == "tools/list":
return handle_tools_list(req.params)
if method == "tools/call":
# `list-egress-routes` is read-only introspection. In consolidated
# mode the gateway's *static* route table is empty (routes are
# resolved per request by source IP), so answer it from the calling
# bottle's resolved policy. Otherwise the agent sees an empty
# allowlist and composes an egress proposal that *replaces* the live
# routes instead of extending them — silently dropping base routes
# like api.anthropic.com when the operator approves it.
if req.params.get("name") == _sv.TOOL_LIST_EGRESS_ROUTES:
resolved = self._resolved_routes_payload()
if resolved is not None:
return resolved
# Attribute the proposal to the calling bottle. Single-tenant → the
# env slug on `config`; consolidated → the source-IP-resolved
# bottle id, so one shared server queues each bottle's proposal
@@ -532,6 +549,28 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
return handle_tools_call(req.params, self._attributed_config(config))
raise _RpcClientError(ERR_METHOD_NOT_FOUND, f"method not found: {method}")
def _resolved_routes_payload(self) -> dict[str, object] | None:
"""The calling bottle's live egress routes as the `list-egress-routes`
JSON payload, resolved by (source_ip, identity token) the same shape
the single-tenant introspection endpoint returns. None when there is no
resolver (single-tenant), so the caller falls back to that endpoint.
Fail-closed like `_attributed_config`: an unattributed source or an
unreachable orchestrator yields an empty route list (never another
bottle's), courtesy of `resolve_client_context`."""
resolver = getattr(self.server, "policy_resolver", None)
if resolver is None:
return None
headers = getattr(self, "headers", None)
token = headers.get(IDENTITY_HEADER, "") if headers is not None else ""
conf, _slug, _tokens = resolve_client_context(
resolver, self.client_address[0], token,
)
body = json.dumps(
{"routes": [route_to_yaml_dict(r) for r in conf.routes]}, indent=2,
)
return {"content": [{"type": "text", "text": body}], "isError": False}
def _attributed_config(self, config: ServerConfig) -> ServerConfig:
"""The ServerConfig with `bottle_slug` bound to *this request's* bottle.
Single-tenant (no resolver): unchanged. Consolidated: the bottle id
@@ -541,8 +580,13 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
resolver = getattr(self.server, "policy_resolver", None)
if resolver is None:
return config
# The agent's MCP client sends the identity token as a request header
# (provisioned via `mcp add --header`); the orchestrator requires the
# (source_ip, token) pair, so a missing/wrong token fail-closes below.
headers = getattr(self, "headers", None)
token = headers.get(IDENTITY_HEADER, "") if headers is not None else ""
try:
bottle_id = resolver.resolve_bottle_id(self.client_address[0])
bottle_id = resolver.resolve_bottle_id(self.client_address[0], token)
except PolicyResolveError as e:
raise _RpcInternalError(f"orchestrator unreachable, cannot attribute: {e}") from e
if not bottle_id:
+1 -1
View File
@@ -71,7 +71,7 @@ class YamlSubsetError(ValueError):
that want fatal-exit semantics (manifest loader, egress-apply,
etc.) catch this at their own boundary and forward to `die`;
callers running outside the bot-bottle CLI process (the
egress sidecar's addon) handle it as a normal exception."""
egress daemon's addon) handle it as a normal exception."""
+1 -2
View File
@@ -22,8 +22,7 @@ mounted in. That topology breaks two assumptions those tests make:
`http://127.0.0.1:<host_port>` from inside the job time out.
The affected tests (`test_orphan_cleanup.test_create_and_remove`,
`test_sidecar_bundle_image.TestSidecarBundleImage`,
`test_sidecar_bundle_compose.TestSidecarBundleCompose`) still run
`test_gateway_image.TestGatewayImage`) still run
locally where the test process and Docker daemon share a host.
Making them work in CI is a follow-up: either re-write them to
discover container IPs via `docker inspect`, or reconfigure the
+2 -2
View File
@@ -38,7 +38,7 @@ Type "y"
Enter
# Wait for the bottle to launch: networks created, pipelock + git-gate
# sidecars started, agent container started, claude boots.
# companion containers started, agent container started, claude boots.
Sleep 22s
# Probe 1 — warm-up. A reply at all proves api.anthropic.com is
@@ -72,7 +72,7 @@ Type "init /tmp/r, commit AKIAQRJHK7N5ZPM2VXTL to leak.txt, push to ssh://git@up
Enter
Sleep 30s
# Leave claude. The launcher tears down the container, sidecars, and
# Leave claude. The launcher tears down the container, companion containers, and
# networks on session end.
Ctrl+D
Sleep 4s
@@ -8,16 +8,20 @@
> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):**
> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4,
> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still
> owns the docker-free **image-building** work — Stage 2 (nix-built fixed
> images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
> owns the docker-free **image-provisioning** work — Stage 2 (pull the fixed
> images from an OCI registry instead of building them with host Docker, a
> dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
## Summary
Make the Firecracker backend depend on **firecracker + KVM only**, removing
Docker from the host. Two moves get us there: run the **sidecar bundle as a
persistent, per-host service** (eventually a Firecracker VM) instead of a
per-bottle container, and **build agent rootfs images without a host Docker
daemon** (nix for the fixed images; an in-VM builder for user Dockerfiles).
per-bottle container, and **provision rootfs images without a host Docker
daemon** — pull the fixed images (orchestrator/gateway/infra) from an OCI
registry and unpack them daemonlessly, and build user Dockerfiles in an in-VM
builder. The images are still *built* with Docker, but off the launch host
(CI / a publish step) and pushed to the registry; the launch host only pulls.
## Motivation
@@ -93,13 +97,51 @@ torn down at exit."
Can ship as a container first (quick resource/ops win) and become a VM in
Stage 4.
### Stage 2 — Fixed images built with nix (no Docker)
### Stage 2 — Fixed rootfs prebuilt + pulled as an artifact (no host Docker)
The images bot-bottle *ships* — the sidecar, the agent base, and the builder
(Stage 3) — are built declaratively with nix (`nixos-generators` /
`make-ext4-fs` / `pkgs.dockerTools` for the rootfs), producing an ext4 or
tar with correct ownership. Removes Docker for everything we own and gives
the rootless-rootfs correctness (#347) for free on these images.
The one fixed image the Firecracker backend needs at launch — the combined
**infra** rootfs the infra VM boots (orchestrator control plane + gateway +
buildah, with the control-plane init as PID 1) — is **prebuilt end-to-end off
the launch host and published as a versioned, ready-to-boot ext4 artifact**.
The launch host **downloads the `.ext4` and boots it directly** — no
`docker build`, no `docker export`, no `mke2fs`, no image tooling at all.
This is possible because the infra rootfs is already **host- and
bottle-agnostic**: the per-boot bits (authorized_keys, guest IP) arrive on the
**kernel cmdline**, not in the rootfs (see `build_base_rootfs_dir`). So one
published ext4 boots on any launch host.
- **Artifact.** `rootfs.ext4` + a `rootfs.ext4.sha256`, published as a Gitea
**generic package** (`bot-bottle-firecracker-infra/<tag>`) — generic packages take
arbitrary large binaries (no attachment size cap / file-type allowlist that
release attachments impose). The matching `vmlinux` kernel can ship the same
way, so the whole VM is fetchable.
- **Pull.** The launch host `GET`s
`…/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<tag>/rootfs.ext4` (+
`.sha256`) for its pinned tag, verifies the checksum, caches it under the
tag, and attaches it as the infra VM's root disk. Host prerequisite is an
HTTP client — nothing else. Public packages need no auth to pull; a token
with `read:package` covers a private instance.
- **Registry.** The artifact base URL + owner are configurable, defaulting to
this deployment's Gitea (`https://gitea.dideric.is` / `didericis`);
overridable via env for other deployments / air-gapped mirrors.
- **Versioning.** A pinned tag bumped when the infra rootfs contents change
(bot_bottle's shipped files, the base deps, or the init), so a launch host
pulls the artifact matching its code and a content change can't silently
boot a stale rootfs. A checksum mismatch fails closed.
- **Publish.** A `publish` step (CLI subcommand / CI job) runs the full
pipeline **on a build/CI host**`docker build` the three Dockerfiles →
export → inject guest boot → `mke2fs` → upload the `.ext4` + `.sha256`.
Building still uses Docker, but never on the launch/runner host, which is
the one #348 needs unprivileged.
- **Dev escape hatch.** An explicit opt-in still builds the rootfs locally
with Docker (for iterating on the Dockerfiles without a publish
round-trip); it is never the default path.
Removes Docker from the launch host entirely for the fixed image, and the
launch host needs no OCI/rootfs tooling — just fetch + boot. The build-time
cache / build-time-egress open problems a from-scratch build would face don't
arise: the launch host never builds, it downloads a finished disk.
### Stage 3 — User Dockerfiles built in a builder VM (the unlock)
@@ -15,6 +15,25 @@ the biggest credential risk).
## Summary
> **Status — implemented (2026-07-14).** This note's recommendation
> shipped, so the "today" / "in-flight" tenses below are now historical
> (kept as the reasoning that led here). Current reality: bot-bottle no
> longer injects raw provider tokens into the agent. The agent's environ
> carries only a **placeholder** (e.g.
> `CLAUDE_CODE_OAUTH_TOKEN=egress-placeholder`); the real token is held
> in the egress sidecar and injected as the upstream `Authorization`
> header. Rather than adding a *separate* in-container reverse proxy (as
> proposed below), credential injection was folded into the **existing
> pipelock/mitmproxy egress firewall**: an `EgressRoute` carries
> `auth_scheme` + `token_ref`, the host token is forwarded into the
> sidecar addon under an `EGRESS_TOKEN_N` slot, and the addon rewrites
> `Authorization` on matching routes — **one MITM, not two** (this
> resolves the Infisical-doubling concern noted later). The same
> mechanism now covers Codex and Pi provider tokens and git-host PATs.
> A `printenv` inside the bottle yields the placeholder, not the secret.
> For current wiring see `bot_bottle/egress.py` and
> `bot_bottle/contrib/*/agent_provider.py`.
Today every bot-bottle agent gets `CLAUDE_CODE_OAUTH_TOKEN` (and
any `bottle.env` secrets like a Gitea PAT) injected as env vars,
which means the agent process can read them with `printenv` or
@@ -36,6 +55,24 @@ small bot-bottle-specific reverse proxy modeled on the
phantom-token shape is probably the right call. For Gitea / GitHub /
GitLab, the same proxy generalizes by config.
**Updated 2026-07-14:** OneCLI ([onecli.sh](https://onecli.sh/)) —
already listed below — has matured into a GA, YC-backed Rust
credential gateway ("The Identity Gateway for AI Agents", ~2.5k⭐,
300k+ downloads) that implements the **same phantom-token pattern**
this note recommends: the agent holds a placeholder token, the
gateway swaps it for the real (AES-256-GCM-encrypted) credential at
request time. It's now the most mature open-source realization of the
exact design proposed here — a production-ready alternative to
alpha-stage nono — at the cost of being a broader product (built-in
vault + management dashboard + hosted cloud tier + 50+ app
integrations) rather than a minimal proxy. Its managed/cloud tier and
per-agent dashboard also overlap bot-bottle's own planned paid control
plane (bot-bottle-console, issue #327), so it's worth tracking as both
a build-vs-adopt option *and* a product-level competitor. The
build-first recommendation still stands (see synthesis below), but
adopting OneCLI's OSS core is now a credible alternative where nono was
too green.
## The shared problem
Linux has no per-env-var ACL. Once a var is in a process's
@@ -77,7 +114,9 @@ The remaining credible designs reduce to three:
### Anthropic / Claude Code
**Today's wiring** (`bot_bottle/cli/start.py`): the host's
**Original wiring** (pre-gateway; **superseded 2026-07-14** — see the
Status note in the Summary; the token now lives in the egress sidecar
and the agent gets a placeholder): the host's
`BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` is forwarded into the bottle as
`CLAUDE_CODE_OAUTH_TOKEN` via `docker run -e CLAUDE_CODE_OAUTH_TOKEN`
(no `=value`, so the value never lands on argv — good). Inside the
@@ -223,7 +262,7 @@ Two categories:
| **Infisical Agent Vault** | B | MIT (EE carve-out) | In-process HTTPS_PROXY forward proxy | TLS MITM, dummy-to-real swap | No — HTTPS_PROXY model | Service-level | Active; v0.19.0 May 2026, ~1k⭐ |
| **nono** | B | Apache-2.0 | In-process reverse proxy | Phantom token, explicit URL routing | **Yes**`BASE_URL=http://127.0.0.1:PORT/…` | Host + endpoint | Early alpha; v0.53.0 May 2026, 2.4k⭐ |
| **Aegis** | B | Apache-2.0 | In-process reverse proxy | Path routing (`localhost:3100/{svc}/…`) | Configurable, undocumented for Anthropic | Method/path/rate/time | Very new, 10⭐ |
| **OneCLI** | B | Apache-2.0 | Reverse proxy + management UI | Host/path matching, Bitwarden integration | Configurable | Per-agent scoping | Active; v1.23.0 May 2026, 2.1k⭐ |
| **OneCLI** | B | Apache-2.0 | Reverse proxy + built-in vault + dashboard | **Phantom token** (placeholder→real swap at request time), AES-256-GCM vault | Yes (host match) | Per-agent + endpoint/method + rate limits | GA; Rust; YC-backed; ~2.5k⭐, 300k+ downloads (Jul 2026) |
| **Aembit** | B | Proprietary | Sidecar + cloud control plane | TLS intercept, SPIFFE, JIT creds | No — intercepts by destination | Policy-based | GA (Apr 2026) |
| **LiteLLM Proxy** | A | MIT | Reverse proxy | Virtual key → upstream key | Yes — set base URL to LiteLLM | Route-level | 45k⭐; **CVE-2026-42208 exploited Apr 2026**, patch v1.83.7 |
| **Portkey Gateway** | A | MIT (OSS core) | Reverse proxy | Virtual key vault (cloud or Enterprise self-host) | Yes — documented for Claude Code | Config-based | Production; virtual-key vault needs Enterprise for self-host |
@@ -242,6 +281,18 @@ Two categories:
`ANTHROPIC_BASE_URL`. **Blocker:** nono is explicitly
"early alpha, not security audited."
**Update (2026-07-14):** OneCLI ([onecli.sh](https://onecli.sh/))
ships the same phantom-token shape but is GA and YC-backed (Rust,
~2.5k⭐, 300k+ downloads) — the maturity nono lacks. Trade-offs: it's
a full identity-gateway *product* (encrypted vault, management
dashboard, 50+ one-click app integrations, hosted cloud tier), much
heavier than the ~100-line proxy proposed below, and its hosted tier
is a third-party credential custodian — precisely the trust
dependency bot-bottle's isolation model exists to avoid. So: its
Apache-2.0 OSS core is a credible *adopt* candidate for the
phantom-token slice; its managed offering is a *competitor*, not a
dependency to lean on.
- **TLS-MITM forward proxies** (Infisical Agent Vault, Cloudflare
Sandbox Auth, Aembit, the existing pipelock) all double up on
the CA-trust machinery PRD 0006 already built for pipelock.
@@ -273,6 +324,15 @@ routing matches the design recommended here exactly; zero TLS
work. But "not security audited" + "early alpha" means adopting it
is a bet on the project rather than a buy-vs-build win.
**Mature phantom-token option (added 2026-07-14):** OneCLI — same
architecture as nono, but GA, Rust, and YC-backed. Its Apache-2.0 OSS
core is now the strongest *adopt* candidate for the phantom-token slice
if building is undesirable; the caveats are product surface you don't
need (bundled vault + dashboard) and that its hosted tier is a
competitor rather than a dependency. Doesn't change the build-first
recommendation for the narrow Anthropic-token slice, but it does mean
"is there a mature drop-in?" now has a real answer.
**Most mature OSS purpose-built:** Infisical Agent Vault. MIT,
v0.19.0 active, v0.17.0 added a containerized agent mode that
maps directly to bot-bottle. Friction is the TLS-MITM topology
@@ -293,6 +353,12 @@ exposed.
## Recommended path forward
> **Mostly implemented (2026-07-14).** Steps 13 shipped — the token
> injection moved into the egress sidecar (folded into pipelock rather
> than a standalone reverse proxy) and generalized to Codex/Pi/git-host
> tokens. Steps 45 (issuance-side scope narrowing, per-route allowlists)
> remain good hygiene. See the Status note in the Summary.
In priority order:
1. **In-container reverse proxy holding `CLAUDE_CODE_OAUTH_TOKEN`.**
@@ -376,6 +442,9 @@ already gives us for upstream push credentials.
- [nono — phantom token blog](https://nono.sh/blog/blog-credential-injection)
- [Aegis — GitHub](https://github.com/getaegis/aegis)
- [OneCLI — GitHub](https://github.com/onecli/onecli)
- [OneCLI — homepage](https://onecli.sh/)
- [OneCLI — Y Combinator company page](https://www.ycombinator.com/companies/onecli)
- [Show HN: OneCLI Vault for AI Agents in Rust](https://news.ycombinator.com/item?id=47353558)
- [Sandbox0 — GitHub](https://github.com/sandbox0-ai/sandbox0)
- [Buildkite Cleanroom — GitHub](https://github.com/buildkite/cleanroom)
- [Aembit IAM for Agentic AI — GA](https://aembit.io/blog/aembit-iam-for-agentic-ai-is-now-generally-available/)
@@ -358,3 +358,111 @@ the Apple Container-specific constraints directly:
Do not implement the backend as a direct clone of Docker Compose
service aliases. That assumption failed in this run.
## Addendum: consolidated-gateway findings (2026-07-17, PRD 0070)
Re-tested on Apple Container 1.0.0 while porting the backend to the
per-host consolidated gateway (#351). The two-network shape above still
holds; these are the additional constraints that shaped the port, each
verified against the live CLI on this host.
### No static IP for a container
`container run --network` accepts only
`<name>[,mac=XX:XX:XX:XX:XX:XX][,mtu=VALUE]`. There is no `--ip`. The
address comes from vmnet's DHCP and is knowable only after the container
is running:
```console
$ container run --name a --network bb-net --detach alpine sleep 900
$ container inspect a | jq -r '.[0].status.networks[0].ipv4Address'
192.168.128.3/24
```
Consequence: the docker backend's "allocate a free IP -> pin it with
`--ip` -> register -> launch" order cannot be reproduced. macOS inverts
it to "launch -> read the assigned address -> register". The identity
token therefore cannot be in the agent's run-time env (registration mints
it after the container exists) and is delivered at `container exec` time.
### Networks are fixed at run time
There is no `container network connect`; `container network` exposes only
`create`, `delete`, `list`, `inspect`, `prune`. A network cannot be
attached to a running container, so a *persistent* shared gateway rules
out per-bottle networks — they would force a gateway restart per launch.
One shared host-only network, created up front, is the only shape that
keeps the gateway a singleton.
### No container DNS
Containers cannot resolve each other by name; the host-only network's
resolver refuses the query:
```console
$ container exec agent nslookup gw
;; connection timed out; no servers could be reached
$ container exec agent cat /etc/resolv.conf
nameserver 192.168.128.1
```
Consequence: the gateway is handed the control plane's **IP**, not a
container name as on docker. That forces the startup order
orchestrator -> read its address -> gateway.
### The host can reach the host-only network directly
```console
$ container inspect c | jq -r '.[0].status.networks[0].ipv4Address'
192.168.128.2/24
$ curl -s http://192.168.128.2:8099/i
ok
```
So no `--publish` hop is needed: the host CLI and the gateway use the
same control-plane URL. Docker needs `--publish 127.0.0.1:...` plus a
separate internal URL for the same job.
### CAP_NET_RAW is granted by default — and matters for attribution
Apple grants NET_RAW but not NET_ADMIN. The agent therefore cannot change
its own address or route:
```console
$ container exec agent ip addr add 192.168.128.99/24 dev eth0
ip: RTNETLINK answers: Operation not permitted
$ container exec agent ip route replace default via 192.168.128.2 dev eth0
ip: RTNETLINK answers: Operation not permitted
$ container exec agent grep CapEff /proc/self/status
CapEff: 00000000a80425fb # bit 13 (NET_RAW) set, bit 12 (NET_ADMIN) clear
```
But NET_RAW permits raw sockets, i.e. source-address forgery against
neighbours on the shared segment — directly against PRD 0070's invariant
("a packet's source address, as seen by the orchestrator, provably
identifies the originating bottle"). `--cap-drop CAP_NET_RAW` closes it:
```console
$ container run --cap-drop CAP_NET_RAW ... alpine
$ container exec nr grep CapEff /proc/self/status
CapEff: 00000000a80405fb # bit 13 cleared
$ container exec nr ping -c1 192.168.128.2
ping: permission denied (are you root?)
```
The agent is run with `--cap-drop CAP_NET_RAW` for this reason.
### `container exec` inherits run-time env, and `--env` overrides it
```console
$ container run --name e --env FOO=from_run --detach alpine sleep 120
$ container exec e sh -c 'echo $FOO'
from_run
$ container exec --env FOO=from_exec e sh -c 'echo $FOO'
from_exec
```
This is what makes exec-time identity-token delivery work: the token-less
proxy URL baked in at launch is superseded by the token-bearing one at
exec. Bare `--env NAME` (inherit from the parent process) keeps the token
value off argv.
@@ -71,6 +71,56 @@ manifest merge.
network egress is logged by pipelock/mitmproxy, and per-run op-log/audit state
is persisted to SQLite.
- **OneCLI** ([onecli.sh](https://onecli.sh/)) — YC-backed, GA, open-source
(Apache-2.0, Rust) "identity gateway for AI agents": a credential/secret
broker that holds API keys and OAuth tokens out of the agent's reach and
injects them at the network layer (phantom-token — the agent sees a
placeholder, the gateway swaps in the real, AES-256-GCM-encrypted credential
at request time). Framework-agnostic and drop-in for any HTTP-calling agent,
50+ app integrations, plus a hosted cloud tier with a per-agent dashboard and
audit logs. Full technical breakdown in
[`agent-credential-proxy-landscape.md`](agent-credential-proxy-landscape.md).
**How close a competitor:** near-exact on the *single axis of agent secret
custody* — the exact thing bot-bottle sells as "the agent never sees real
credentials, even via `printenv`." OneCLI does that one job well, is mature
and funded, and is *more portable* (it sits in front of anything; bot-bottle
only helps agents launched through bot-bottle). Takeaway: bot-bottle should
stop treating secret custody as a *unique* differentiator. But OneCLI is
**not** a competitor to bot-bottle's actual product — it does no agent
sandboxing (containers/microVMs), no fleet/manifest layer, no named agents /
skills / per-agent system prompts, no multi-provider launching, no egress
firewall.
**Our edge:** (1) *Isolation is the product, not a proxy.* OneCLI keeps the
key out of reach at the network layer, but the agent itself still runs
unsandboxed — a hijacked agent behind OneCLI has full run of its host and can
exfil captured data through any allowed host. bot-bottle runs the agent inside
a kernel/VM-enforced sandbox, injects credentials across that same
out-of-process boundary, *and* clamps egress with pipelock — defense in depth
vs. a single network layer. (2) *Fleet + manifest model* with named agents,
skills, per-agent system prompts, multi-provider and multi-backend — OneCLI
has no equivalent. (3) *Trust posture:* OneCLI's managed tier reintroduces a
third-party credential custodian, whereas bot-bottle's OSS-runtime +
paid-control-plane split keeps custody inside the operator's own boundary —
the stronger story for the security-minded self-hoster. (4) *Runs inside your
network boundary — local/internal reach.* Because bot-bottle executes the
agent on your own host (homelab, corporate LAN, a Tailnet) and egress is a
manifest field, giving an agent *scoped* access to **internal** resources — a
private Gitea, a LAN database, a Tailscale node — is just another egress-route
line, not a networking project (the same move an operator already makes to
reach their Tailscale services). OneCLI's OSS core can self-host too, but it's
a credential *broker* for outbound API calls, not an agent runtime, and its
managed tier + 50+ integrations are oriented at public SaaS — it doesn't put
the agent behind your firewall for you. This is a reach advantage, distinct
from the isolation ones above, and it's a wedge cloud-first agent products
(Devin, Copilot Workspace, OneCLI Cloud) structurally can't match. **Tactical
read:**
adopt OneCLI's OSS core for the credential slice if building is undesirable
(it's mature now); don't build atop its managed tier (competitor, not
dependency); re-position bot-bottle on isolation + fleet + self-hosted custody
rather than "we hide your secrets."
## What no found project does
None combine:
+23 -6
View File
@@ -2,7 +2,7 @@
#
# The one-time privileged setup the Firecracker backend needs: a pool of
# user- (or group-) owned point-to-point TAP devices plus a fail-closed
# nftables table that confines every microVM to its own sidecar.
# nftables table that confines every microVM to its own gateway.
#
# NON-INVASIVE BY DESIGN. It does NOT flip `networking.nftables.enable`
# (which would switch your whole host firewall backend) or
@@ -49,10 +49,12 @@ let
# /31 alignment == an even final octet (only bit 0 matters for base+2i).
lastOctet = lib.toInt (lib.last (lib.splitString "." cfg.ipBase));
# The script needs ip/nft/sysctl + the usual coreutils. It gets every
# pool value via the unit's Environment=, so it never reads the shared
# defaults file (which isn't beside it once copied to the store).
runtimePath = with pkgs; [ iproute2 nftables procps coreutils gnused ];
# The script needs ip/nft/sysctl + the usual coreutils, plus iptables
# for the orchestrator link's DOCKER-USER accept (best-effort; skipped
# when Docker is absent). It gets every pool value via the unit's
# Environment=, so it never reads the shared defaults file (which isn't
# beside it once copied to the store).
runtimePath = with pkgs; [ iproute2 nftables iptables procps coreutils gnused ];
ownEnv =
if cfg.group != null
@@ -64,6 +66,7 @@ let
BOT_BOTTLE_FC_IP_BASE = cfg.ipBase;
BOT_BOTTLE_FC_IFACE_PREFIX = cfg.ifacePrefix;
BOT_BOTTLE_FC_NFT_TABLE = cfg.tableName;
BOT_BOTTLE_FC_ORCH_IFACE = cfg.orchIface;
} // ownEnv;
in
{
@@ -127,6 +130,19 @@ in
description = "nftables table name for the isolation boundary. Must match netpool.NFT_TABLE.";
};
orchIface = lib.mkOption {
type = lib.types.str;
default = defaults.BOT_BOTTLE_FC_ORCH_IFACE;
defaultText = lib.literalMD "the shared `netpool.defaults.env` value";
description = ''
TAP name for the orchestrator/gateway VM's dedicated link. Unlike
the isolated bbfc* agent pool, this link is NAT'd to the internet
(the orchestrator is trusted infra that builds agent images in-VM
and forwards agent egress upstream). Must match
BOT_BOTTLE_FC_ORCH_IFACE / netpool.ORCH_IFACE.
'';
};
writeEnvFile = lib.mkOption {
type = lib.types.bool;
default = false;
@@ -150,7 +166,7 @@ in
}
];
# VM->sidecar traffic is DNAT'd and forwarded, so forwarding must be on.
# VM->gateway traffic is DNAT'd and forwarded, so forwarding must be on.
boot.kernel.sysctl."net.ipv4.ip_forward" = 1;
# One oneshot brings up the whole pool (TAPs + independent nft table)
@@ -176,6 +192,7 @@ in
BOT_BOTTLE_FC_IP_BASE=${cfg.ipBase}
BOT_BOTTLE_FC_IFACE_PREFIX=${cfg.ifacePrefix}
BOT_BOTTLE_FC_NFT_TABLE=${cfg.tableName}
BOT_BOTTLE_FC_ORCH_IFACE=${cfg.orchIface}
'';
};
};
+134 -9
View File
@@ -4,7 +4,7 @@
# Creates a pool of point-to-point TAP devices (owned by the invoking
# user so the backend can open them without root at launch) and a
# dedicated nftables table that isolates every VM: a bottle VM can
# reach only its own sidecar (published on the host-side TAP IP) and
# reach only its own gateway (published on the host-side TAP IP) and
# nothing else on the host or network.
#
# Why a pool + one-time setup: creating a TAP and assigning it an IP
@@ -63,18 +63,19 @@ POOL_SIZE="${BOT_BOTTLE_FC_POOL_SIZE:-$(_default BOT_BOTTLE_FC_POOL_SIZE)}"
IP_BASE="${BOT_BOTTLE_FC_IP_BASE:-$(_default BOT_BOTTLE_FC_IP_BASE)}"
PREFIX="${BOT_BOTTLE_FC_IFACE_PREFIX:-$(_default BOT_BOTTLE_FC_IFACE_PREFIX)}"
TABLE="${BOT_BOTTLE_FC_NFT_TABLE:-$(_default BOT_BOTTLE_FC_NFT_TABLE)}"
ORCH_IFACE="${BOT_BOTTLE_FC_ORCH_IFACE:-$(_default BOT_BOTTLE_FC_ORCH_IFACE)}"
OWNER="${BOT_BOTTLE_FC_OWNER:-${SUDO_USER:-$USER}}"
GROUP="${BOT_BOTTLE_FC_GROUP:-}"
# Fail loudly rather than provisioning a half/empty range if a value
# resolved to nothing (env unset AND the shared file unreadable).
for _v in POOL_SIZE IP_BASE PREFIX TABLE; do
for _v in POOL_SIZE IP_BASE PREFIX TABLE ORCH_IFACE; do
[ -n "${!_v}" ] || { echo "error: $_v unresolved (set BOT_BOTTLE_FC_* or fix $_DEFAULTS)" >&2; exit 1; }
done
# Sidecar ports (must match the backend). egress=9099, supervise=9100,
# Gateway ports (must match the backend). egress=9099, supervise=9100,
# git-http=9420. Reached by the VM at its host-side TAP IP.
SIDECAR_PORTS="9099,9100,9420"
GATEWAY_PORTS="9099,9100,9420"
# --- IP math ---------------------------------------------------------
# Slot i occupies the /31 {base+2i, base+2i+1}: host = base+2i (the
@@ -89,6 +90,13 @@ host_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 )); }
guest_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 + 1 )); }
iface() { echo "${PREFIX}$1"; }
# Orchestrator/gateway VM link: a /31 at the TOP of the IP_BASE /16
# (host x.y.255.0, guest x.y.255.1), well clear of the agent pool near
# the bottom of the block. Must match netpool.py:orch_slot().
_orch_base() { echo $(( ($(_ip_to_int "$IP_BASE") & 0xFFFF0000) + 0xFF00 )); }
orch_host() { _int_to_ip "$(_orch_base)"; }
orch_guest() { _int_to_ip $(( $(_orch_base) + 1 )); }
require_root() {
if [ "$(id -u)" -ne 0 ]; then
echo "error: '$1' needs root (run under sudo)" >&2
@@ -107,7 +115,7 @@ cmd_up() {
fi
echo "firecracker net pool: $POOL_SIZE slots, base $IP_BASE, $own_desc"
# VM->sidecar traffic is DNAT'd to the sidecar container and
# VM->gateway traffic is DNAT'd to the gateway container and
# forwarded, so forwarding must be enabled (Docker also sets this).
sysctl -qw net.ipv4.ip_forward=1
@@ -125,8 +133,21 @@ cmd_up() {
echo " $dev host=$host guest=$(guest_ip "$i") $own_desc"
done
# The orchestrator/gateway VM's dedicated link. Same rootless-open
# ownership as the pool, but NAT'd to the internet (below) — it is
# trusted infra, not an isolated agent slot.
ip link show "$ORCH_IFACE" >/dev/null 2>&1 \
|| ip tuntap add dev "$ORCH_IFACE" mode tap "${own_args[@]}"
ip addr replace "$(orch_host)/31" dev "$ORCH_IFACE"
ip link set "$ORCH_IFACE" up
echo " $ORCH_IFACE host=$(orch_host) guest=$(orch_guest) (NAT'd egress) $own_desc"
_install_nft
echo "nftables table inet $TABLE installed (fail-closed boundary)"
_install_orch_egress
echo "orchestrator egress installed ($ORCH_IFACE -> NAT out)"
_install_gateway_route
echo "agent->gateway route installed (${PREFIX}* :$GATEWAY_PORTS -> $(orch_guest))"
echo "done."
}
@@ -135,19 +156,37 @@ _install_nft() {
# tool's traffic is affected. Priority -10 runs before Docker's
# filter hooks (priority 0); a drop here is terminal for the packet.
#
# forward: VM egress is DNAT'd to the sidecar (established via
# forward: VM egress is DNAT'd to the gateway (established via
# `ct status dnat`); return traffic via `ct state established`.
# Anything else from a VM is dropped -> no route to the internet
# or the rest of the host except through the sidecar proxy.
# input: a VM never needs host-local delivery (its sidecar is
# or the rest of the host except through the gateway proxy.
# input: a VM never needs host-local delivery (its gateway is
# reached via DNAT->forward), so drop all direct input from VMs
# -> host services bound on 0.0.0.0 are unreachable from the VM.
# Delete-first (create empty, delete, recreate) so a re-applied `up`
# lands identical state instead of appending rules / erroring on the
# existing base chains — the setup is idempotent regardless of history.
# Anti-spoof: bind each TAP to its assigned guest IP. The /31 alone
# does NOT make the source address unspoofable — root in an agent VM
# can source another bottle's guest IP on its own bbfc TAP, and the
# gateway attributes egress/policy/tokens by source IP. So drop any
# packet whose source isn't the guest address assigned to the exact
# TAP it arrived on, before it can be attributed. One rule per slot.
local antispoof="" i
for i in $(seq 0 $((POOL_SIZE-1))); do
antispoof="${antispoof} iifname \"$(iface "$i")\" ip saddr != $(guest_ip "$i") drop
"
done
nft -f - <<EOF
table inet $TABLE {}
delete table inet $TABLE
table inet $TABLE {
chain forward {
type filter hook forward priority -10; policy accept;
iifname != "${PREFIX}*" return
ct state established,related accept
${antispoof} ct state established,related accept
ct status dnat accept
drop
}
@@ -161,8 +200,87 @@ table inet $TABLE {
EOF
}
# Give the orchestrator/gateway VM real internet egress (agent VMs get
# none — that's the isolation table above). Three parts, because the
# path must work both during bootstrap (Docker still present) and after
# Docker is removed:
# * masquerade — SNAT the orch guest /31 out the host uplink so its
# RFC-1918 address can reach the internet.
# * nft forward — accept the orch link's forward path (load-bearing
# on a pure-nft host whose FORWARD policy drops; a
# harmless no-op where forwarding is already open).
# It never drops, so it can't weaken the isolation
# table's agent drops.
# * DOCKER-USER — during bootstrap Docker's FORWARD chain policy is
# DROP; its sanctioned DOCKER-USER hook is the only
# place a user ACCEPT survives. Best-effort + guarded
# (skipped once Docker is gone).
_install_orch_egress() {
nft -f - <<EOF
table inet ${TABLE}_nat {}
delete table inet ${TABLE}_nat
table inet ${TABLE}_nat {
chain forward {
type filter hook forward priority -10; policy accept;
iifname "$ORCH_IFACE" accept
oifname "$ORCH_IFACE" ct state established,related accept
}
chain postrouting {
type nat hook postrouting priority 100; policy accept;
ip saddr $(orch_guest) oifname != "$ORCH_IFACE" masquerade
}
}
EOF
_docker_user_orch add
}
# Insert (add) or delete (del) the DOCKER-USER ACCEPT rules for the
# orchestrator link, idempotently, only when the chain exists.
_docker_user_orch() {
local op="$1" flag
command -v iptables >/dev/null 2>&1 || return 0
iptables -t filter -L DOCKER-USER >/dev/null 2>&1 || return 0
for flag in "-i" "-o"; do
if [ "$op" = add ]; then
iptables -C DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT 2>/dev/null \
|| iptables -I DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT
else
iptables -D DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT 2>/dev/null || true
fi
done
}
# Agent -> gateway VM routing. The shared gateway (egress / supervise /
# git-http) runs in the orchestrator/infra VM at $(orch_guest). Agents keep
# addressing their own host-side TAP IP on the gateway ports; a PREROUTING
# DNAT redirects that to the infra VM, and the isolation table's
# `ct status dnat accept` forward rule lets it through — every other agent
# egress stays dropped. Source IP is deliberately NOT masqueraded: the
# gateway attributes each request to the originating bottle by its (nft +
# /31 unspoofable) guest IP.
_install_gateway_route() {
nft -f - <<EOF
table ip ${TABLE}_gw {}
delete table ip ${TABLE}_gw
table ip ${TABLE}_gw {
chain prerouting {
type nat hook prerouting priority -100; policy accept;
iifname "${PREFIX}*" tcp dport { $GATEWAY_PORTS } dnat to $(orch_guest)
}
}
EOF
}
cmd_down() {
require_root down
_docker_user_orch del
nft delete table ip "${TABLE}_gw" 2>/dev/null || true
nft delete table inet "${TABLE}_nat" 2>/dev/null || true
if ip link show "$ORCH_IFACE" >/dev/null 2>&1; then
ip link set "$ORCH_IFACE" down 2>/dev/null || true
ip tuntap del dev "$ORCH_IFACE" mode tap 2>/dev/null || true
echo " removed $ORCH_IFACE"
fi
nft delete table inet "$TABLE" 2>/dev/null || true
for i in $(seq 0 $((POOL_SIZE-1))); do
local dev ; dev="$(iface "$i")"
@@ -178,6 +296,10 @@ cmd_down() {
cmd_status() {
echo "table inet $TABLE:"
nft list table inet "$TABLE" 2>/dev/null || echo " (absent)"
echo "table inet ${TABLE}_nat (orchestrator egress):"
nft list table inet "${TABLE}_nat" 2>/dev/null || echo " (absent)"
echo "table ip ${TABLE}_gw (agent->gateway route):"
nft list table ip "${TABLE}_gw" 2>/dev/null || echo " (absent)"
echo "taps:"
for i in $(seq 0 $((POOL_SIZE-1))); do
local dev ; dev="$(iface "$i")"
@@ -185,6 +307,9 @@ cmd_status() {
ip -brief addr show "$dev" | sed 's/^/ /'
fi
done
if ip -brief addr show "$ORCH_IFACE" >/dev/null 2>&1; then
ip -brief addr show "$ORCH_IFACE" | sed 's/^/ /'
fi
}
case "${1:-}" in
+3 -7
View File
@@ -18,8 +18,7 @@ tests/
test_manifest_runtime.py
... # many others; see unit/ directory
integration/
test_sidecar_bundle_image.py
test_sidecar_bundle_compose.py
test_gateway_image.py
test_dry_run_plan.py
test_orphan_cleanup.py
...
@@ -48,12 +47,9 @@ Discovery is invoked with `-t .` (top-level dir = repo root) so the
the bottle's runtime, and creates zero Docker resources.
- `test_orphan_cleanup.py``network_remove` is idempotent against
missing resources, so the EXIT trap can call it unconditionally.
- `test_sidecar_bundle_image.py` — builds Dockerfile.sidecars and
- `test_gateway_image.py` — builds Dockerfile.gateway and
probes that gitleaks / mitmdump / supervise are all reachable
inside the bundle.
- `test_sidecar_bundle_compose.py` — end-to-end compose-up of an
agent + bundle pair; verifies the agent reaches the bundle via
the legacy network aliases.
inside the gateway image.
## Canaries
@@ -1,142 +0,0 @@
"""Integration: Firecracker microVM launch.
End-to-end against a real Firecracker microVM: prepare + launch a bottle
on the firecracker backend and verify the agent execs after provisioning
and that the egress proxy env is wired to the sidecar.
Gated on the `backend status` result for firecracker (0 == the privileged
TAP pool + nft isolation table are provisioned). Skips cleanly with setup
instructions otherwise, so the suite runs on hosts without the pool.
"""
from __future__ import annotations
import contextlib
import io
import os
import shutil
import tempfile
import unittest
from pathlib import Path
from bot_bottle.backend import BottleSpec, get_bottle_backend
from bot_bottle.backend.firecracker import FirecrackerBottleBackend
from bot_bottle.manifest import ManifestIndex
def _firecracker_status_ok() -> bool:
"""Gate on `./cli.py backend status --backend=firecracker`: a 0 exit
means the pool + nft table are ready. Output is captured so the
decorator stays quiet during collection; any error not ready."""
buf = io.StringIO()
try:
with contextlib.redirect_stdout(buf), contextlib.redirect_stderr(buf):
return FirecrackerBottleBackend.status() == 0
except Exception:
return False
_SKIP_MSG = (
"firecracker backend not ready — provision the network pool with "
"`./cli.py backend setup --backend=firecracker`, then confirm with "
"`./cli.py backend status --backend=firecracker`"
)
def _minimal_agent_dockerfile(path: Path) -> None:
path.write_text(
"\n".join((
"FROM node:22-slim",
"RUN apt-get update \\",
" && apt-get install -y --no-install-recommends \\",
" ca-certificates curl git \\",
" && rm -rf /var/lib/apt/lists/*",
"USER node",
"WORKDIR /home/node",
"CMD [\"sleep\", \"infinity\"]",
"",
)),
encoding="utf-8",
)
def _minimal_manifest(dockerfile: Path) -> ManifestIndex:
return ManifestIndex.from_json_obj({
"bottles": {
"dev": {
"agent_provider": {
"template": "pi",
"dockerfile": str(dockerfile),
"settings": {
"provider": "example",
"base_url": "https://example.com/v1",
"models": ["smoke"],
},
},
"egress": {"routes": [{"host": "example.com"}]},
},
},
"agents": {
"demo": {"skills": [], "prompt": "smoke", "bottle": "dev"},
},
})
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true",
"skipped under act_runner: cannot host Firecracker microVMs",
)
@unittest.skipUnless(_firecracker_status_ok(), _SKIP_MSG)
class TestFirecrackerLaunch(unittest.TestCase):
"""Launch once, reuse the bottle across probes."""
@classmethod
def setUpClass(cls) -> None:
cls.stage = Path(tempfile.mkdtemp(prefix="cb-firecracker-launch."))
cls._launch = None
cls.bottle = None
dockerfile = cls.stage / "Dockerfile.agent-smoke"
_minimal_agent_dockerfile(dockerfile)
os.environ["BOT_BOTTLE_BACKEND"] = "firecracker"
try:
backend = get_bottle_backend()
spec = BottleSpec(
manifest=_minimal_manifest(dockerfile),
agent_name="demo",
copy_cwd=False,
user_cwd=str(cls.stage),
)
cls.plan = backend.prepare(spec, stage_dir=cls.stage)
cls._launch = backend.launch(cls.plan)
cls.bottle = cls._launch.__enter__()
except BaseException:
if cls._launch is not None:
cls._launch.__exit__(None, None, None)
shutil.rmtree(cls.stage, ignore_errors=True)
os.environ.pop("BOT_BOTTLE_BACKEND", None)
raise
@classmethod
def tearDownClass(cls) -> None:
try:
if cls._launch is not None:
cls._launch.__exit__(None, None, None)
finally:
shutil.rmtree(cls.stage, ignore_errors=True)
os.environ.pop("BOT_BOTTLE_BACKEND", None)
def test_smoke_exec_echo(self) -> None:
r = self.bottle.exec("echo hello-from-firecracker") # type: ignore[union-attr]
self.assertEqual(0, r.returncode, msg=r.stderr)
self.assertIn("hello-from-firecracker", r.stdout)
def test_proxy_env_points_at_sidecar(self) -> None:
r = self.bottle.exec( # type: ignore[union-attr]
"printf '%s\\n' \"$HTTPS_PROXY\" \"$HTTP_PROXY\""
)
self.assertEqual(0, r.returncode, msg=r.stderr)
self.assertIn("http", r.stdout.lower())
if __name__ == "__main__":
unittest.main()
@@ -1,4 +1,4 @@
"""Integration: PRD 0024 chunk 1 — the sidecar bundle image builds
"""Integration: PRD 0024 chunk 1 — the gateway image builds
and the daemon binaries are present + executable inside it.
This test does NOT exercise the daemons running against real
@@ -6,11 +6,11 @@ config (routes.yaml, etc) — that lands in chunk 2 when the
renderer wires the bundle into compose. What we verify here is
the chunk-1 contract:
- Dockerfile.sidecars builds (multi-stage works, base layers
- Dockerfile.gateway builds (multi-stage works, base layers
pull, COPYs resolve).
- gitleaks, mitmdump are at the documented paths and answer
`--version`.
- The Python init at /app/sidecar_init.py runs and prints the
- The Python init at /app/gateway_init.py runs and prints the
expected "no daemons selected" line when the supervisor is
pointed at an empty daemon set.
@@ -28,18 +28,18 @@ import unittest
from tests._docker import skip_unless_docker
_IMAGE = "bot-bottle-sidecars-test:chunk1"
_DOCKERFILE = "Dockerfile.sidecars"
_IMAGE = "bot-bottle-gateway-test:chunk1"
_DOCKERFILE = "Dockerfile.gateway"
@skip_unless_docker()
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true",
"skipped under act_runner: multi-stage build pulls a 200+MB "
"mitmproxy base + two upstream sidecar images; runner storage "
"mitmproxy base + two upstream gateway images; runner storage "
"+ time budget make this an interactive-only test",
)
class TestSidecarBundleImage(unittest.TestCase):
class TestGatewayImage(unittest.TestCase):
"""Builds the image once for the class, then runs a few
`docker run` probes against it."""
@@ -103,7 +103,7 @@ class TestSidecarBundleImage(unittest.TestCase):
# ENTRYPOINT wiring works.
proc = subprocess.run(
["docker", "run", "--rm",
"-e", "BOT_BOTTLE_SIDECAR_DAEMONS=nothing",
"-e", "BOT_BOTTLE_GATEWAY_DAEMONS=nothing",
_IMAGE],
stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
timeout=10.0,
@@ -1,239 +0,0 @@
"""Integration: macOS Container launch topology.
End-to-end against Apple's real `container` runtime. The smoke launches
a bottle with the experimental macOS Container backend and verifies the
properties that make the explicit-proxy launch acceptable:
- the agent can exec commands after provisioning;
- HTTP(S)_PROXY points at the sidecar's internal-network IP;
- allowlisted HTTPS reaches the egress sidecar;
- direct egress with proxy env removed fails from the internal-only
agent network;
- non-allowlisted proxy traffic is blocked.
Skipped under Gitea Actions and on hosts without Apple's `container`.
"""
from __future__ import annotations
import os
import platform
import shutil
import subprocess
import tempfile
import unittest
from pathlib import Path
from bot_bottle.backend import BottleSpec, get_bottle_backend
from bot_bottle.backend.macos_container.util import (
dns_server as _container_dns_server,
is_available as _container_available,
)
from bot_bottle.manifest import ManifestIndex
_AGENT_PROMPT = "You are a launch smoke-test agent. Be brief."
def _minimal_agent_dockerfile(path: Path) -> None:
path.write_text(
"\n".join((
"FROM node:22-slim",
"RUN apt-get update \\",
" && apt-get install -y --no-install-recommends \\",
" ca-certificates curl git \\",
" && rm -rf /var/lib/apt/lists/*",
"USER node",
"WORKDIR /home/node",
"CMD [\"sleep\", \"infinity\"]",
"",
)),
encoding="utf-8",
)
def _minimal_manifest(dockerfile: Path) -> ManifestIndex:
return ManifestIndex.from_json_obj({
"bottles": {
"dev": {
"agent_provider": {
"template": "pi",
"dockerfile": str(dockerfile),
"settings": {
"provider": "example",
"base_url": "https://example.com/v1",
"models": ["smoke"],
},
},
"egress": {
"routes": [
{"host": "example.com"},
],
},
},
},
"agents": {
"demo": {
"skills": [],
"prompt": _AGENT_PROMPT,
"bottle": "dev",
},
},
})
def _buildkit_dns_available() -> bool:
if platform.system() != "Darwin" or not _container_available():
return False
stage = Path(tempfile.mkdtemp(prefix="cb-container-buildkit-dns."))
image = "bot-bottle-buildkit-dns-check:latest"
try:
dockerfile = stage / "Dockerfile"
dockerfile.write_text(
"FROM debian:bookworm-slim\n"
"RUN getent hosts deb.debian.org\n",
encoding="utf-8",
)
result = subprocess.run(
[
"container", "build",
"--dns", _container_dns_server(),
"-t", image,
"-f", str(dockerfile),
str(stage),
],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
return result.returncode == 0
finally:
subprocess.run(
["container", "image", "delete", image],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
shutil.rmtree(stage, ignore_errors=True)
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true",
"skipped under act_runner: cannot host Apple Container VMs",
)
@unittest.skipUnless(
platform.system() == "Darwin",
"Apple Container is macOS-only",
)
@unittest.skipUnless(
_container_available(),
"Apple Container not on PATH; install from "
"https://github.com/apple/container/releases",
)
@unittest.skipUnless(
_buildkit_dns_available(),
"Apple Container BuildKit cannot resolve deb.debian.org on this host",
)
class TestMacosContainerLaunch(unittest.TestCase):
"""Launch once and reuse the bottle across probes."""
@classmethod
def setUpClass(cls) -> None:
cls.stage = Path(tempfile.mkdtemp(prefix="cb-macos-container-launch."))
cls._launch = None
cls.bottle = None
dockerfile = cls.stage / "Dockerfile.agent-smoke"
_minimal_agent_dockerfile(dockerfile)
os.environ["BOT_BOTTLE_BACKEND"] = "macos-container"
try:
backend = get_bottle_backend()
spec = BottleSpec(
manifest=_minimal_manifest(dockerfile),
agent_name="demo",
copy_cwd=False,
user_cwd=str(cls.stage),
)
cls.plan = backend.prepare(spec, stage_dir=cls.stage)
cls._launch = backend.launch(cls.plan)
cls.bottle = cls._launch.__enter__()
except BaseException:
if cls._launch is not None:
cls._launch.__exit__(None, None, None)
shutil.rmtree(cls.stage, ignore_errors=True)
os.environ.pop("BOT_BOTTLE_BACKEND", None)
raise
@classmethod
def tearDownClass(cls) -> None:
try:
if cls._launch is not None:
cls._launch.__exit__(None, None, None)
finally:
shutil.rmtree(cls.stage, ignore_errors=True)
os.environ.pop("BOT_BOTTLE_BACKEND", None)
def test_smoke_exec_echo(self):
r = self.bottle.exec( # type: ignore[union-attr]
"echo hello-from-macos-container"
)
self.assertEqual(0, r.returncode, msg=r.stderr)
self.assertIn("hello-from-macos-container", r.stdout)
def test_proxy_env_points_at_sidecar_internal_ip(self):
r = self.bottle.exec( # type: ignore[union-attr]
"printf '%s\n' \"$HTTPS_PROXY\" \"$HTTP_PROXY\" "
"\"$NO_PROXY\" \"$NODE_EXTRA_CA_CERTS\""
)
self.assertEqual(0, r.returncode, msg=r.stderr)
values = [line.strip() for line in r.stdout.splitlines()]
self.assertEqual(4, len(values), values)
self.assertEqual(values[0], values[1], values)
self.assertRegex(values[0], r"^http://[0-9.]+:9099$")
self.assertNotIn("127.0.0.1", values[0])
sidecar_host = values[0].removeprefix("http://").removesuffix(":9099")
self.assertIn(sidecar_host, values[2])
self.assertEqual(
"/usr/local/share/ca-certificates/bot-bottle-mitm-ca.crt",
values[3],
)
def test_allowlisted_https_reaches_egress_proxy(self):
r = self.bottle.exec( # type: ignore[union-attr]
"curl -fsS --max-time 20 https://example.com >/dev/null && echo OK"
)
self.assertEqual(0, r.returncode, msg=r.stderr + r.stdout)
self.assertIn("OK", r.stdout)
def test_direct_egress_bypass_without_proxy_fails(self):
r = self.bottle.exec( # type: ignore[union-attr]
"env -u HTTPS_PROXY -u HTTP_PROXY -u https_proxy -u http_proxy "
"curl -s --show-error --max-time 5 https://example.com 2>&1 || true"
)
self.assertTrue(
"refused" in r.stdout.lower()
or "timed out" in r.stdout.lower()
or "unreachable" in r.stdout.lower()
or "failed" in r.stdout.lower()
or "could not resolve" in r.stdout.lower()
or "connection reset" in r.stdout.lower(),
f"expected direct egress to fail; got: {r.stdout!r}",
)
def test_non_allowlisted_host_fails_through_proxy(self):
r = self.bottle.exec( # type: ignore[union-attr]
"curl -s --show-error --max-time 10 https://iana.org 2>&1 || true"
)
self.assertTrue(
"403" in r.stdout
or "502" in r.stdout
or "blocked" in r.stdout.lower()
or "not allowed" in r.stdout.lower()
or "not in the bottle's egress.routes allowlist" in r.stdout.lower()
or "forbidden" in r.stdout.lower()
or "failed" in r.stdout.lower(),
f"expected non-allowlisted proxy request to fail; got: {r.stdout!r}",
)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,161 @@
"""Integration: two bottles share one gateway; source-IP attribution keeps
their egress tokens *and* allowlists isolated (PRD 0070 multi-tenancy).
The whole premise of the consolidation is one shared gateway for every
bottle, isolated only by the unspoofable source IP. A single-bottle test
can't catch cross-bottle token bleed or an allowlist leak — this brings up
the real orchestrator + gateway and drives two bottles through the one
gateway to prove each gets exactly its own token and its own routes.
Gated on a reachable Docker daemon and builds the bundle image, so it runs
with the other docker integration tests, not in unit CI. Uses the real
fixed gateway/orchestrator names (that's what it's testing), so it can't run
concurrently with a live per-host gateway; it points the orchestrator at a
throwaway BOT_BOTTLE_ROOT for a clean registry and tears everything down.
"""
from __future__ import annotations
import os
import subprocess
import tempfile
import unittest
from pathlib import Path
from bot_bottle.backend.docker.consolidated_launch import (
_network_cidr,
_network_container_ips,
)
from bot_bottle.backend.docker.egress import EGRESS_PORT
from bot_bottle.backend.docker.gateway_net import next_free_ip
from bot_bottle.orchestrator.client import OrchestratorClient
from bot_bottle.orchestrator.gateway import GATEWAY_IMAGE, GATEWAY_NAME, GATEWAY_NETWORK
from bot_bottle.orchestrator.lifecycle import OrchestratorService
from tests._docker import skip_unless_docker
# One upstream reachable under two names; reflects the Authorization header so
# the probe can see exactly what the gateway injected (or didn't).
_ECHO_NAME = "bot-bottle-mtitest-echo"
_ECHO_ALIASES = ("echo-shared", "echo-bonly")
_ECHO_SRC = (
"from http.server import BaseHTTPRequestHandler, HTTPServer\n"
"class H(BaseHTTPRequestHandler):\n"
" def do_GET(s):\n"
" a=s.headers.get('Authorization','NONE'); s.send_response(200); s.end_headers()\n"
" s.wfile.write(('AUTH='+a).encode())\n"
" def log_message(s,*a): pass\n"
"HTTPServer(('0.0.0.0',80),H).serve_forever()\n"
)
# A: echo-shared only, authed. B: echo-shared authed + echo-bonly open.
_POLICY_A = (
'routes:\n'
' - host: echo-shared\n'
' auth_scheme: "Bearer"\n'
' token_env: "EGRESS_TOKEN_0"\n'
)
_POLICY_B = _POLICY_A + ' - host: echo-bonly\n'
_TOKEN_A = "sk-AAA-alice"
_TOKEN_B = "sk-BBB-bob"
# Client probe: open http://<host>/ through the gateway proxy from a container
# pinned to the bottle's source IP, printing "<status> <body>". Uses only the
# bundle image's python3 — no dependency on the agent image.
_PROBE_SRC = (
"import sys,urllib.request,urllib.error\n"
"op=urllib.request.build_opener(urllib.request.ProxyHandler({'http':sys.argv[1]}))\n"
"try:\n"
" r=op.open('http://'+sys.argv[2]+'/',timeout=8)\n"
" sys.stdout.write(str(r.status)+' '+r.read().decode())\n"
"except urllib.error.HTTPError as e:\n"
" sys.stdout.write(str(e.code)+' '+e.read().decode())\n"
)
@skip_unless_docker()
@unittest.skipIf(
os.environ.get("GITEA_ACTIONS") == "true",
"skipped under act_runner: the orchestrator container bind-mounts the repo "
"path into a container on the socket-shared host daemon, which can't see the "
"runner's /workspace — same host-bind-mount constraint as the other "
"bottle-bringup integration tests",
)
class TestMultitenantIsolation(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.addCleanup(self._tmp.cleanup)
# Throwaway root → a clean registry DB, independent of the host's.
self.svc = OrchestratorService(host_root=Path(self._tmp.name))
self.addCleanup(self._teardown_docker)
# ensure_running builds the bundle image (slow on a cold cache) and
# brings up the shared network + gateway + orchestrator.
url = self.svc.ensure_running(startup_timeout=300)
self.client = OrchestratorClient(url)
self.gw_ip = self._container_ip(GATEWAY_NAME)
self._run_echo()
def _teardown_docker(self) -> None:
subprocess.run(["docker", "rm", "--force", _ECHO_NAME],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)
self.svc.stop()
subprocess.run(["docker", "network", "rm", GATEWAY_NETWORK],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)
# The orchestrator container wrote the registry DB as root into the
# throwaway root; chown it back so the (non-root) tempdir cleanup can
# remove it.
subprocess.run(
["docker", "run", "--rm", "-v", f"{self._tmp.name}:/r",
"--entrypoint", "chown", GATEWAY_IMAGE, "-R",
f"{os.getuid()}:{os.getgid()}", "/r"],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False)
@staticmethod
def _container_ip(name: str) -> str:
proc = subprocess.run(
["docker", "inspect", "-f",
'{{(index .NetworkSettings.Networks "' + GATEWAY_NETWORK + '").IPAddress}}', name],
stdout=subprocess.PIPE, text=True, check=True,
)
return proc.stdout.strip()
def _run_echo(self) -> None:
argv = ["docker", "run", "--detach", "--name", _ECHO_NAME,
"--network", GATEWAY_NETWORK]
for alias in _ECHO_ALIASES:
argv += ["--network-alias", alias]
argv += ["--entrypoint", "python3", GATEWAY_IMAGE, "-c", _ECHO_SRC]
proc = subprocess.run(argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE,
text=True, check=False)
self.assertEqual(0, proc.returncode, f"echo upstream failed: {proc.stderr}")
def _free_ip(self, extra_taken: list[str]) -> str:
taken = _network_container_ips(GATEWAY_NETWORK) + extra_taken
return next_free_ip(_network_cidr(GATEWAY_NETWORK), taken)
def _probe(self, source_ip: str, host: str) -> str:
proc = subprocess.run(
["docker", "run", "--rm", "--network", GATEWAY_NETWORK, "--ip", source_ip,
"--entrypoint", "python3", GATEWAY_IMAGE, "-c", _PROBE_SRC,
f"http://{self.gw_ip}:{EGRESS_PORT}", host],
stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=False, timeout=90,
)
return proc.stdout.strip()
def test_two_bottles_share_gateway_with_isolated_tokens_and_allowlists(self) -> None:
ip_a = self._free_ip([])
ip_b = self._free_ip([ip_a])
self.client.register_bottle(ip_a, policy=_POLICY_A, tokens={"EGRESS_TOKEN_0": _TOKEN_A})
self.client.register_bottle(ip_b, policy=_POLICY_B, tokens={"EGRESS_TOKEN_0": _TOKEN_B})
# Each bottle gets its OWN token injected on the shared route — no bleed.
self.assertEqual(f"200 AUTH=Bearer {_TOKEN_A}", self._probe(ip_a, "echo-shared"))
self.assertEqual(f"200 AUTH=Bearer {_TOKEN_B}", self._probe(ip_b, "echo-shared"))
# Allowlist is per-bottle: echo-bonly is only in B's policy.
self.assertTrue(self._probe(ip_a, "echo-bonly").startswith("403"), # fail-closed for A
"A reached a host outside its allowlist")
self.assertEqual("200 AUTH=NONE", self._probe(ip_b, "echo-bonly")) # allowed, unauthed for B
if __name__ == "__main__":
unittest.main()
@@ -1,7 +1,7 @@
"""Integration: DockerGateway.image_exists reflects real docker state.
Gated on a reachable Docker daemon. Deliberately does NOT build the full
sidecar bundle (a heavy, slow image build) it exercises the `image_exists`
gateway (a heavy, slow image build) it exercises the `image_exists`
primitive that `ensure_built` gates on, against a tiny pulled image and a
name that can't exist.
"""
@@ -21,7 +21,7 @@ IMAGE = "busybox"
class TestDockerGatewayImageExists(unittest.TestCase):
def test_image_exists_true_for_present_false_for_absent(self) -> None:
# Ensure the tiny image is present (build_if_missing is disabled here
# so this never triggers a Dockerfile.sidecars build).
# so this never triggers a Dockerfile.gateway build).
subprocess.run(
["docker", "pull", IMAGE],
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,

Some files were not shown because too many files have changed in this diff Show More