Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 0a1f949937 | |||
| 64ca39ddd4 | |||
| 3c426f45f5 | |||
| 1d3e9e859c |
+18
-28
@@ -34,45 +34,35 @@
|
||||
# 9420 git-gate smart HTTP (VM-backend agent-facing transport)
|
||||
# 9100 supervise (MCP HTTP)
|
||||
|
||||
# Based on `python:3.12-slim` (Debian trixie) rather than the
|
||||
# `mitmproxy/mitmproxy` image (Debian bookworm) so the whole stack —
|
||||
# gateway here, and the firecracker infra image that builds FROM this —
|
||||
# lands on trixie, whose buildah (1.39) can build agent Dockerfiles that
|
||||
# use heredocs. mitmproxy is pip-installed to the same effect as the
|
||||
# upstream image. (bookworm's buildah is 1.28, which can't parse
|
||||
# `RUN ... <<EOF`; see the infra image + PR discussion.)
|
||||
FROM python:3.12-slim
|
||||
# Stage 1: gitleaks binary. The upstream gitleaks image is alpine
|
||||
# with the binary at /usr/bin/gitleaks. Pinned by digest in lockstep
|
||||
# with Dockerfile.git-gate's prior base (now deleted at chunk 3).
|
||||
FROM zricethezav/gitleaks@sha256:c00b6bd0aeb3071cbcb79009cb16a60dd9e0a7c60e2be9ab65d25e6bc8abbb7f AS gitleaks-src
|
||||
|
||||
# Stage 2: assembly. mitmproxy/mitmproxy is debian-slim-based with
|
||||
# Python + mitmdump pre-installed — heavier than the others, so
|
||||
# this stage starts there and pulls the standalone binaries in.
|
||||
FROM mitmproxy/mitmproxy:11.1.3
|
||||
|
||||
# Run as root inside the bundle. The bundle is the isolation
|
||||
# boundary; per-daemon user separation inside it is not load-bearing
|
||||
# and complicates the supervisor's spawn path.
|
||||
USER root
|
||||
|
||||
# Runtime system deps:
|
||||
# git supplies the `git daemon` subcommand (no separate package)
|
||||
# plus the core `git` binary the pre-receive hook invokes.
|
||||
# openssh-client supplies the upstream SSH transport the
|
||||
# pre-receive hook uses to forward accepted refs.
|
||||
# ca-certificates is needed for mitmdump upstream TLS.
|
||||
# ca-certificates is needed for mitmdump upstream TLS (the
|
||||
# base image already has it; listed for explicitness).
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
git openssh-client ca-certificates \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# mitmdump (the egress data plane). The upstream mitmproxy image baked
|
||||
# this in; on the plain python base we pip-install the same pinned
|
||||
# version. Its CA dir is set explicitly via `--set confdir=` in
|
||||
# egress-entrypoint.sh, so it doesn't depend on a `mitmproxy` home user.
|
||||
RUN pip install --no-cache-dir mitmproxy==11.1.3
|
||||
|
||||
# gitleaks (the pre-receive hook's secret scanner). Installed from its
|
||||
# official release, pinned by version + SHA256 and verified — rather than
|
||||
# using a third-party image as a build stage (supply-chain surface, and it
|
||||
# would pin us to that image's cadence). python (already present) does the
|
||||
# download so we add no curl/wget. trixie apt also ships gitleaks, but an
|
||||
# older 8.16; the pinned download keeps the verified 8.30.1.
|
||||
ARG GITLEAKS_VERSION=8.30.1
|
||||
ARG GITLEAKS_SHA256=551f6fc83ea457d62a0d98237cbad105af8d557003051f41f3e7ca7b3f2470eb
|
||||
RUN url="https://github.com/gitleaks/gitleaks/releases/download/v${GITLEAKS_VERSION}/gitleaks_${GITLEAKS_VERSION}_linux_x64.tar.gz" \
|
||||
&& python3 -c "import sys,urllib.request; urllib.request.urlretrieve(sys.argv[1], '/tmp/gitleaks.tar.gz')" "$url" \
|
||||
&& echo "${GITLEAKS_SHA256} /tmp/gitleaks.tar.gz" | sha256sum -c - \
|
||||
&& tar -xzf /tmp/gitleaks.tar.gz -C /usr/bin gitleaks \
|
||||
&& rm /tmp/gitleaks.tar.gz
|
||||
# Pull the standalone binaries into the final image.
|
||||
COPY --from=gitleaks-src /usr/bin/gitleaks /usr/bin/gitleaks
|
||||
|
||||
# Project Python: addon + server modules + the init supervisor.
|
||||
# Kept flat under /app/ so mitmdump's loader resolves them as
|
||||
|
||||
@@ -1,45 +0,0 @@
|
||||
# Firecracker single infra-VM image (PRD 0070 Stage B).
|
||||
#
|
||||
# The per-host infra VM runs the orchestrator control plane, the gateway
|
||||
# data plane, AND builds agent images (buildah) — all in one microVM (see
|
||||
# backend/firecracker/infra_vm.py). It composes:
|
||||
# * FROM the gateway image (mitmproxy / git / gitleaks / supervise + the
|
||||
# flat daemon modules) — now trixie-based, so buildah 1.39 is available;
|
||||
# * `COPY --from` the orchestrator image's content (the single definition
|
||||
# of the control-plane payload — see Dockerfile.orchestrator), so this
|
||||
# VM and the docker backend share one orchestrator definition; and
|
||||
# * buildah, installed HERE only (the docker orchestrator/gateway images
|
||||
# never carry it).
|
||||
#
|
||||
# multi-`FROM` can't union two bases (that's multi-stage, not multiple
|
||||
# inheritance), so the orchestrator content is pulled in via `COPY --from`
|
||||
# rather than a second base. Both images share the trixie `python:3.12-slim`
|
||||
# base, so the copy is clean (same python; future installed deps copy too).
|
||||
#
|
||||
# The docker backend keeps orchestrator + gateway as separate images; this
|
||||
# combined image exists only for the Firecracker single-VM cut. Splitting a
|
||||
# service back into its own VM later is a routing change, not a repackaging
|
||||
# (PRD 0070's "secret concentration"; a disposable builder can boot from
|
||||
# this same image on its own TAP).
|
||||
FROM bot-bottle-gateway:latest
|
||||
|
||||
# --- in-VM agent-image builder (PRD 0069 Stage 3) -------------------
|
||||
# The Firecracker backend builds users' agent Dockerfiles *inside this VM*
|
||||
# with buildah (rootless, daemonless) instead of on the host — no host
|
||||
# Docker daemon, no root-equivalent `docker` group. `crun` is the OCI
|
||||
# runtime; `netavark` + `aardvark-dns` are the network backend for `FROM`
|
||||
# pulls + `RUN` egress. Requires the trixie base (buildah 1.39: bookworm's
|
||||
# 1.28 can't parse Dockerfile heredocs that agent images use).
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends \
|
||||
buildah crun netavark aardvark-dns \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
# vfs + chroot: buildah works as root in the bare microVM (no
|
||||
# fuse-overlayfs / overlay module / subuid maps). Matches image_builder.
|
||||
ENV STORAGE_DRIVER=vfs \
|
||||
BUILDAH_ISOLATION=chroot
|
||||
|
||||
# The orchestrator content, pulled from its single definition. The gateway
|
||||
# image already has the flat daemon modules under /app; this adds the full
|
||||
# `bot_bottle` package so `python3 -m bot_bottle.orchestrator` resolves.
|
||||
COPY --from=bot-bottle-orchestrator:latest /app/bot_bottle /app/bot_bottle
|
||||
+16
-25
@@ -1,36 +1,27 @@
|
||||
# Orchestrator control-plane image (PRD 0070, #384).
|
||||
#
|
||||
# This is the **single definition of the orchestrator's content** — the
|
||||
# `bot_bottle` package baked onto a Python runtime — referenced by BOTH:
|
||||
# * the docker backend, which runs this image directly as the lean
|
||||
# control-plane container; and
|
||||
# * the firecracker infra image (Dockerfile.infra), which `COPY --from`s
|
||||
# this image's `/app/bot_bottle` so the single infra VM runs the same
|
||||
# control plane. Keeping it in one place means future orchestrator deps
|
||||
# (e.g. iroh) are added here once, not duplicated per backend.
|
||||
# The per-host orchestrator runs `python3 -m bot_bottle.orchestrator`.
|
||||
# The `bot_bottle` package is **stdlib-only** by design, so the control
|
||||
# plane needs nothing but a Python runtime — none of the gateway's
|
||||
# mitmproxy / git / gitleaks payload (that is the separate
|
||||
# `bot-bottle-gateway` image, Dockerfile.gateway). Splitting them keeps
|
||||
# the secret-dense control plane (it concentrates every bottle's egress
|
||||
# tokens — see PRD 0070's "secret concentration") on a minimal
|
||||
# dependency surface.
|
||||
#
|
||||
# It stays deliberately lean: the control plane is **stdlib-only** today, so
|
||||
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
|
||||
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
|
||||
# builder, and lives only in Dockerfile.infra). Keeping the secret-dense
|
||||
# control plane on a minimal dependency surface is the point (PRD 0070's
|
||||
# "secret concentration").
|
||||
#
|
||||
# Shares the trixie `python:3.12-slim` base with the gateway image, so when
|
||||
# the orchestrator grows real deps they can be `COPY --from`'d into the
|
||||
# infra image cleanly (same base/python — installed packages copy safely).
|
||||
# The repo is bind-mounted read-only into the container at run time (see
|
||||
# `orchestrator/lifecycle.py`), so the source is NOT copied in here: the
|
||||
# image is just the runtime. `ensure_running` recreates the container
|
||||
# only when the bind-mounted source hash changes (#381), which is why
|
||||
# the code stays a mount rather than a baked layer.
|
||||
|
||||
FROM python:3.12-slim
|
||||
|
||||
# No third-party deps to install — stdlib only. Kept as an explicit,
|
||||
# self-documenting stage so a future confinement step (baking the
|
||||
# package, dropping the bind mount) has an obvious home.
|
||||
WORKDIR /app
|
||||
|
||||
# The orchestrator content. Baked so the image is self-contained (runs from
|
||||
# a built image, no runtime bind-mount); the docker backend may still
|
||||
# bind-mount /app for dev live-reload, which simply overlays this copy.
|
||||
# `.dockerignore` keeps .git/docs/*.md out of the context. (Future deps like
|
||||
# iroh go here too — a shared requirements installed on this same base.)
|
||||
COPY bot_bottle /app/bot_bottle
|
||||
|
||||
# Documentation only; lifecycle.py overrides the entrypoint to
|
||||
# `python3 -m bot_bottle.orchestrator` with the runtime flags.
|
||||
ENTRYPOINT ["python3", "-m", "bot_bottle.orchestrator"]
|
||||
|
||||
@@ -5,7 +5,7 @@
|
||||
# bot-bottle
|
||||
|
||||
[](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
|
||||
[](https://coverage.readthedocs.io/)
|
||||
[](https://coverage.readthedocs.io/)
|
||||
[](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
|
||||
|
||||
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
|
||||
|
||||
@@ -266,7 +266,6 @@ class AgentProvider(ABC):
|
||||
gate_scheme = getattr(plan, "git_gate_insteadof_scheme", "git")
|
||||
content = git_gate_render_gitconfig(
|
||||
manifest_bottle.git, gate_host, scheme=gate_scheme,
|
||||
identity_token=getattr(plan, "identity_token", ""),
|
||||
)
|
||||
guest_gitconfig = f"{plan.guest_home}/.gitconfig"
|
||||
with tempfile.NamedTemporaryFile(
|
||||
|
||||
@@ -511,18 +511,6 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
|
||||
del plan
|
||||
return ""
|
||||
|
||||
def ensure_orchestrator(self) -> str:
|
||||
"""Bring up this backend's per-host orchestrator + shared gateway
|
||||
(idempotent) and return the host-reachable control-plane URL.
|
||||
|
||||
This is the backend-agnostic bring-up entry point: `launch` calls
|
||||
it as part of starting a bottle, and operator tools (`supervise`)
|
||||
call it to start the control plane on demand when none is running
|
||||
yet. Docker starts the orchestrator + gateway containers;
|
||||
firecracker boots the infra VM. Backends with no orchestrator
|
||||
(macos-container) die with a pointer — the default here."""
|
||||
die(f"backend {self.name!r} has no orchestrator control plane")
|
||||
|
||||
@abstractmethod
|
||||
def prepare_cleanup(self) -> CleanupT:
|
||||
"""Enumerate orphaned resources from previous bottles. No side
|
||||
|
||||
@@ -105,10 +105,6 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
|
||||
with _launch.launch(plan, provision=self.provision) as bottle:
|
||||
yield bottle
|
||||
|
||||
def ensure_orchestrator(self) -> str:
|
||||
from ...orchestrator.lifecycle import OrchestratorService
|
||||
return OrchestratorService().ensure_running()
|
||||
|
||||
def supervise_mcp_url(self, plan: DockerBottlePlan) -> str:
|
||||
"""Docker bottles reach the supervise daemon via the
|
||||
compose-network alias `supervise:9100`. No per-bottle URL
|
||||
|
||||
@@ -35,10 +35,6 @@ class DockerBottlePlan(BottlePlan):
|
||||
# Likewise the supervise MCP endpoint at the gateway (`http://<gw>:9100/`);
|
||||
# empty → the single-tenant `supervise` alias.
|
||||
agent_supervise_url: str = ""
|
||||
# Per-bottle identity token the agent presents on every attributed request
|
||||
# (egress proxy credentials, git-gate/supervise headers); set by launch
|
||||
# from the orchestrator registration. Empty pre-registration.
|
||||
identity_token: str = ""
|
||||
|
||||
@property
|
||||
def container_name(self) -> str:
|
||||
|
||||
@@ -31,13 +31,7 @@ def consolidated_agent_compose(
|
||||
) -> dict[str, Any]:
|
||||
"""A compose spec with only the agent service, on the external gateway
|
||||
network at `source_ip`, proxying egress through `gateway_ip`."""
|
||||
# Deliver the identity token as egress proxy credentials — the gateway
|
||||
# reads Proxy-Authorization, validates the (source_ip, token) pair, and
|
||||
# strips it before upstream. git-http/supervise get it via their own
|
||||
# headers (git config extraHeader / MCP header).
|
||||
token = getattr(plan, "identity_token", "")
|
||||
cred = f"bottle:{token}@" if token else ""
|
||||
proxy_url = f"http://{cred}{gateway_ip}:{EGRESS_PORT}"
|
||||
proxy_url = f"http://{gateway_ip}:{EGRESS_PORT}"
|
||||
# git-http + supervise live on the gateway too and must NOT go through the
|
||||
# egress proxy — the agent reaches them directly by the gateway address.
|
||||
no_proxy = f"localhost,127.0.0.1,{gateway_ip}"
|
||||
|
||||
@@ -29,11 +29,7 @@ from ...orchestrator.gateway import GATEWAY_NAME, GATEWAY_NETWORK
|
||||
from ...orchestrator.lifecycle import OrchestratorService
|
||||
from ...orchestrator.registration import registration_inputs
|
||||
from .gateway_net import next_free_ip
|
||||
from .gateway_provision import (
|
||||
DockerGatewayTransport,
|
||||
deprovision_git_gate,
|
||||
provision_git_gate,
|
||||
)
|
||||
from .gateway_provision import deprovision_git_gate, provision_git_gate
|
||||
|
||||
|
||||
class ConsolidatedLaunchError(RuntimeError):
|
||||
@@ -125,8 +121,7 @@ def launch_consolidated(
|
||||
metadata=inputs.metadata, tokens=tokens,
|
||||
)
|
||||
try:
|
||||
provision_git_gate(
|
||||
DockerGatewayTransport(gateway_name), reg.bottle_id, git_gate_plan)
|
||||
provision_git_gate(gateway_name, reg.bottle_id, git_gate_plan)
|
||||
except Exception:
|
||||
# Roll the registration back so a provisioning failure leaves no orphan.
|
||||
client.teardown_bottle(reg.bottle_id)
|
||||
@@ -147,7 +142,7 @@ def teardown_consolidated(
|
||||
"""Deregister the bottle and remove its git-gate state from the gateway.
|
||||
Both steps are idempotent so this is safe from a cleanup trap."""
|
||||
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
|
||||
deprovision_git_gate(DockerGatewayTransport(gateway_name), bottle_id)
|
||||
deprovision_git_gate(gateway_name, bottle_id)
|
||||
|
||||
|
||||
__all__ = [
|
||||
|
||||
@@ -15,15 +15,14 @@ bottle's push credentials out of another's repos on the shared gateway.
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from typing import Protocol
|
||||
|
||||
from ...docker_cmd import run_docker
|
||||
from ...git_gate import GitGatePlan, git_gate_render_provision
|
||||
|
||||
# bottle ids index the gateway's per-bottle repo + creds dirs; they land in
|
||||
# exec/cp path arguments, so validate before any path is built (a traversal
|
||||
# id like "../etc" must never reach the gateway). Registry ids are token_hex —
|
||||
# this is defense in depth at the transport boundary.
|
||||
# `docker cp`/`rm` path arguments, so validate before any path is built (a
|
||||
# traversal id like "../etc" must never reach the container). Registry ids are
|
||||
# token_hex — this is defense in depth at the docker boundary.
|
||||
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
|
||||
|
||||
|
||||
@@ -31,42 +30,6 @@ class GatewayProvisionError(RuntimeError):
|
||||
"""A git-gate provisioning step against the running gateway failed."""
|
||||
|
||||
|
||||
class GatewayTransport(Protocol):
|
||||
"""How the launcher stages files + runs commands in the running gateway.
|
||||
Backend-neutral so the same provisioning logic serves the docker gateway
|
||||
(exec/cp over the docker socket) and the firecracker gateway VM (over
|
||||
SSH)."""
|
||||
|
||||
def exec(self, argv: list[str]) -> None:
|
||||
"""Run `argv` in the gateway, raising `GatewayProvisionError` on
|
||||
failure."""
|
||||
|
||||
def cp_into(self, src: str, dest: str) -> None:
|
||||
"""Copy host file `src` to `dest` in the gateway, raising on
|
||||
failure."""
|
||||
|
||||
|
||||
class DockerGatewayTransport:
|
||||
"""`GatewayTransport` for the docker gateway container (exec/cp)."""
|
||||
|
||||
def __init__(self, gateway: str) -> None:
|
||||
self.gateway = gateway
|
||||
|
||||
def exec(self, argv: list[str]) -> None:
|
||||
proc = run_docker(["docker", "exec", self.gateway, *argv])
|
||||
if proc.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
|
||||
)
|
||||
|
||||
def cp_into(self, src: str, dest: str) -> None:
|
||||
proc = run_docker(["docker", "cp", src, f"{self.gateway}:{dest}"])
|
||||
if proc.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
|
||||
)
|
||||
|
||||
|
||||
def _require_safe(bottle_id: str) -> None:
|
||||
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
|
||||
raise GatewayProvisionError(f"unsafe bottle id {bottle_id!r}")
|
||||
@@ -76,11 +39,27 @@ def _creds_dir(bottle_id: str) -> str:
|
||||
return f"/git-gate/creds/{bottle_id}"
|
||||
|
||||
|
||||
def provision_git_gate(
|
||||
transport: GatewayTransport, bottle_id: str, plan: GitGatePlan,
|
||||
) -> None:
|
||||
"""Place `bottle_id`'s git-gate credentials into the running gateway and
|
||||
init its bare repos under `/git/<bottle_id>/`.
|
||||
def _exec(gateway: str, argv: list[str]) -> None:
|
||||
"""`docker exec` a command in the gateway, raising on non-zero exit."""
|
||||
proc = run_docker(["docker", "exec", gateway, *argv])
|
||||
if proc.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"gateway exec {argv!r} failed: {proc.stderr.strip()}"
|
||||
)
|
||||
|
||||
|
||||
def _cp_into(gateway: str, src: str, dest: str) -> None:
|
||||
"""`docker cp` a host file into the gateway, raising on non-zero exit."""
|
||||
proc = run_docker(["docker", "cp", src, f"{gateway}:{dest}"])
|
||||
if proc.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"gateway cp {src} -> {dest} failed: {proc.stderr.strip()}"
|
||||
)
|
||||
|
||||
|
||||
def provision_git_gate(gateway: str, bottle_id: str, plan: GitGatePlan) -> None:
|
||||
"""Place `bottle_id`'s git-gate credentials into the running `gateway`
|
||||
container and init its bare repos under `/git/<bottle_id>/`.
|
||||
|
||||
Copies each upstream's identity key (and known_hosts, when present) into
|
||||
`/git-gate/creds/<bottle_id>/`, then runs the namespaced provisioning
|
||||
@@ -91,36 +70,31 @@ def provision_git_gate(
|
||||
# The pre-receive + access hooks are bottle-agnostic and shared by every
|
||||
# bottle's repos; install them into the gateway (idempotent — same content
|
||||
# each time). The per-bottle model cp'd these into each bundle at start.
|
||||
transport.exec(["mkdir", "-p", "/etc/git-gate"])
|
||||
transport.cp_into(str(plan.hook_script), "/etc/git-gate/pre-receive")
|
||||
transport.cp_into(str(plan.access_hook_script), "/etc/git-gate/access-hook")
|
||||
_exec(gateway, ["mkdir", "-p", "/etc/git-gate"])
|
||||
_cp_into(gateway, str(plan.hook_script), "/etc/git-gate/pre-receive")
|
||||
_cp_into(gateway, str(plan.access_hook_script), "/etc/git-gate/access-hook")
|
||||
creds = _creds_dir(bottle_id)
|
||||
transport.exec(["mkdir", "-p", creds])
|
||||
_exec(gateway, ["mkdir", "-p", creds])
|
||||
for u in plan.upstreams:
|
||||
if u.identity_file:
|
||||
transport.cp_into(u.identity_file, f"{creds}/{u.name}-key")
|
||||
_cp_into(gateway, u.identity_file, f"{creds}/{u.name}-key")
|
||||
known_hosts = str(u.known_hosts_file)
|
||||
if known_hosts and known_hosts != ".":
|
||||
transport.cp_into(known_hosts, f"{creds}/{u.name}-known_hosts")
|
||||
_cp_into(gateway, known_hosts, f"{creds}/{u.name}-known_hosts")
|
||||
# Init the bare repos + per-repo credential config for this namespace.
|
||||
script = git_gate_render_provision(bottle_id, plan.upstreams)
|
||||
transport.exec(["sh", "-c", script])
|
||||
_exec(gateway, ["sh", "-c", script])
|
||||
|
||||
|
||||
def deprovision_git_gate(transport: GatewayTransport, bottle_id: str) -> None:
|
||||
def deprovision_git_gate(gateway: str, bottle_id: str) -> None:
|
||||
"""Remove a bottle's repos + creds from the gateway on teardown. Idempotent
|
||||
— an already-absent namespace is a clean no-op (best effort; a stray dir
|
||||
can't leak, since attribution is by source IP and the bottle is gone)."""
|
||||
_require_safe(bottle_id)
|
||||
try:
|
||||
transport.exec([
|
||||
"rm", "-rf", f"/git/{bottle_id}", _creds_dir(bottle_id),
|
||||
])
|
||||
except GatewayProvisionError:
|
||||
pass # best-effort teardown; absent namespace is success
|
||||
run_docker([
|
||||
"docker", "exec", gateway, "rm", "-rf",
|
||||
f"/git/{bottle_id}", _creds_dir(bottle_id),
|
||||
])
|
||||
|
||||
|
||||
__all__ = [
|
||||
"provision_git_gate", "deprovision_git_gate",
|
||||
"GatewayProvisionError", "GatewayTransport", "DockerGatewayTransport",
|
||||
]
|
||||
__all__ = ["provision_git_gate", "deprovision_git_gate", "GatewayProvisionError"]
|
||||
|
||||
@@ -167,7 +167,6 @@ def launch(
|
||||
egress_plan=egress_plan,
|
||||
agent_git_gate_url=git_gate_url,
|
||||
agent_supervise_url=supervise_url,
|
||||
identity_token=ctx.identity_token,
|
||||
)
|
||||
|
||||
# Step 5: render + up the agent-only compose, pinned on the shared
|
||||
|
||||
@@ -110,7 +110,3 @@ class FirecrackerBottleBackend(
|
||||
|
||||
def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
|
||||
return plan.agent_supervise_url
|
||||
|
||||
def ensure_orchestrator(self) -> str:
|
||||
from . import infra_vm
|
||||
return infra_vm.ensure_running().control_plane_url
|
||||
|
||||
@@ -105,9 +105,10 @@ class FirecrackerBottle(Bottle):
|
||||
# root-owned and unreadable by node, which breaks Node's
|
||||
# process.cwd(), the shell-snapshot machinery, and `/doctor`.
|
||||
# Use `env --chdir` rather than a `sh -c 'cd … && exec "$@"'`
|
||||
# wrapper: it keeps the guest command a flat argv that `agent_argv`
|
||||
# can quote token-by-token for the ssh→guest-shell round trip,
|
||||
# avoiding a fragile nested-quoting `"$@"` script.
|
||||
# wrapper: ssh space-joins everything after the host into one
|
||||
# string for the guest shell, so a quoted script + $@ would be
|
||||
# re-split and mangled (exec'ing the $0 placeholder). All-simple
|
||||
# words survive that join.
|
||||
workdir = self.agent_workdir or _HOME_FOR["node"]
|
||||
remote = ["runuser", "-u", "node", "--",
|
||||
"env", f"--chdir={workdir}",
|
||||
@@ -116,15 +117,7 @@ class FirecrackerBottle(Bottle):
|
||||
return remote
|
||||
|
||||
def agent_argv(self, argv: list[str], *, tty: bool = True) -> list[str]:
|
||||
# ssh space-joins everything after the host into one line the guest
|
||||
# shell re-parses, so pre-quote each remote token for that shell.
|
||||
# Simple words are unchanged (existing behaviour); an arg containing
|
||||
# spaces — e.g. codex's `read_prompt_file` positional "Read and follow
|
||||
# the instructions in <path>." — is quoted so it survives as ONE
|
||||
# argument instead of being re-split (which made codex parse "and" as
|
||||
# a subcommand).
|
||||
remote = self._agent_remote_argv(argv)
|
||||
return [*self._ssh(tty=tty), "--", *(shlex.quote(t) for t in remote)]
|
||||
return [*self._ssh(tty=tty), "--", *self._agent_remote_argv(argv)]
|
||||
|
||||
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
|
||||
agent_argv = self.agent_argv(argv, tty=tty)
|
||||
|
||||
@@ -18,10 +18,6 @@ class FirecrackerBottlePlan(BottlePlan):
|
||||
agent_proxy_url: str = ""
|
||||
agent_git_gate_url: str = ""
|
||||
agent_supervise_url: str = ""
|
||||
# Per-bottle identity token the agent presents on every attributed request
|
||||
# (egress proxy credentials, git-gate/supervise headers); set by launch
|
||||
# from the orchestrator registration. Empty pre-registration.
|
||||
identity_token: str = ""
|
||||
|
||||
@property
|
||||
def container_name(self) -> str:
|
||||
|
||||
@@ -1,23 +1,27 @@
|
||||
"""Consolidated bottle launch sequence for the Firecracker backend
|
||||
(PRD 0070, Stage B).
|
||||
"""Consolidated bottle launch sequence for the Firecracker backend (PRD 0070).
|
||||
|
||||
The shared gateway + orchestrator control plane run in a single persistent
|
||||
per-host **infra VM** (`infra_vm.py`), not Docker containers. Agent VMs reach
|
||||
the gateway's egress / supervise / git-http ports at the infra VM via a
|
||||
PREROUTING DNAT on their own host-side TAP IP (see
|
||||
`scripts/firecracker-netpool.sh`), and the host CLI reaches the control plane
|
||||
over HTTP at the infra VM's guest IP.
|
||||
Mirrors bot_bottle.backend.docker.consolidated_launch but wired for
|
||||
Firecracker's TAP-based network topology instead of a shared Docker bridge.
|
||||
|
||||
Attribution is by the agent VM's guest IP, unspoofable by construction: the
|
||||
/31 point-to-point TAP + the `bot_bottle_fc` nft table ensure only the
|
||||
expected VM can source-IP that address.
|
||||
The per-bottle sidecar bundle (one `docker run` per bottle, published on the
|
||||
slot's host-side TAP IP) is replaced by a single persistent gateway that
|
||||
every Firecracker VM shares. The gateway runs as a Docker container in the
|
||||
dev-harness (a Firecracker VM is stage B per PRD 0070), with its ports
|
||||
published on the host (`0.0.0.0:PORT`). VMs reach it at their slot's
|
||||
host-side TAP IP because Docker's iptables PREROUTING DNAT redirects
|
||||
port 9099/9100/9420 traffic to the gateway container — a path the nft
|
||||
isolation table already allows via `ct status dnat accept` in the forward
|
||||
chain.
|
||||
|
||||
Attribution is by the VM's guest IP, which is unspoofable by construction:
|
||||
the /31 point-to-point TAP topology + the `bot_bottle_fc` nft table ensure
|
||||
that only the expected VM can source-IP that address.
|
||||
|
||||
Sequence:
|
||||
1. ensure the infra VM (control plane + gateway) is up (a singleton — a
|
||||
prior launcher may already have booted it);
|
||||
1. ensure the Firecracker-flavoured orchestrator + gateway are up;
|
||||
2. register the bottle by its guest IP (attribution key) → bottle id +
|
||||
identity token;
|
||||
3. provision its git-gate repos/creds into the gateway VM (over SSH);
|
||||
3. provision its git-gate repos/creds into the running gateway;
|
||||
4. fetch the shared gateway CA for the provisioner to install in the rootfs.
|
||||
|
||||
The TAP slot allocation, rootfs build, and VM boot are the caller's job.
|
||||
@@ -30,12 +34,30 @@ from dataclasses import dataclass
|
||||
from ...egress import EgressPlan
|
||||
from ...git_gate import GitGatePlan
|
||||
from ...orchestrator.client import OrchestratorClient
|
||||
from ...orchestrator.gateway import (
|
||||
GATEWAY_NETWORK,
|
||||
DockerGateway,
|
||||
)
|
||||
from ...orchestrator.lifecycle import (
|
||||
OrchestratorService,
|
||||
OrchestratorStartError, # re-exported so callers can catch it
|
||||
)
|
||||
from ...orchestrator.registration import registration_inputs
|
||||
from ..docker.egress import EGRESS_PORT
|
||||
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
|
||||
from . import infra_vm
|
||||
from ...supervise import SUPERVISE_PORT
|
||||
|
||||
_GIT_HTTP_PORT = 9420
|
||||
|
||||
# Separate names from the Docker gateway so both backends can coexist on one
|
||||
# host (and for clarity in `docker ps` output).
|
||||
_FC_GATEWAY_NAME = "bot-bottle-fc-gateway"
|
||||
_FC_ORCHESTRATOR_NAME = "bot-bottle-fc-orchestrator"
|
||||
_FC_ORCHESTRATOR_LABEL = "bot-bottle-fc-orchestrator=1"
|
||||
# Ports the gateway publishes on the host so Firecracker VMs can reach it
|
||||
# via their TAP link. Docker's PREROUTING DNAT + nft's `ct status dnat
|
||||
# accept` in the forward chain route the traffic.
|
||||
_FC_GATEWAY_HOST_PORTS = (EGRESS_PORT, SUPERVISE_PORT, _GIT_HTTP_PORT)
|
||||
|
||||
|
||||
class ConsolidatedLaunchError(RuntimeError):
|
||||
@@ -53,6 +75,33 @@ class LaunchContext:
|
||||
orchestrator_url: str
|
||||
|
||||
|
||||
class _FirecrackerOrchestratorService(OrchestratorService):
|
||||
"""Dev-harness orchestrator for the Firecracker backend.
|
||||
|
||||
Uses a gateway that publishes its ports on the host so Firecracker VMs can
|
||||
reach it via their TAP link. The gateway and orchestrator containers use
|
||||
`*-fc-*` names so both backends can run independently on the same host.
|
||||
"""
|
||||
|
||||
def __init__(self, **kwargs: object) -> None:
|
||||
super().__init__(
|
||||
orchestrator_name=_FC_ORCHESTRATOR_NAME,
|
||||
orchestrator_label=_FC_ORCHESTRATOR_LABEL,
|
||||
**kwargs, # type: ignore[arg-type]
|
||||
)
|
||||
|
||||
def _gateway(self) -> DockerGateway:
|
||||
# The heavy data-plane image (#384 split it from the lean control-plane
|
||||
# `image` this service's orchestrator container runs).
|
||||
return DockerGateway(
|
||||
self._gateway_image,
|
||||
name=_FC_GATEWAY_NAME,
|
||||
network=self.network,
|
||||
orchestrator_url=self.internal_url,
|
||||
host_port_bindings=_FC_GATEWAY_HOST_PORTS,
|
||||
)
|
||||
|
||||
|
||||
def launch_consolidated(
|
||||
egress_plan: EgressPlan,
|
||||
git_gate_plan: GitGatePlan,
|
||||
@@ -60,12 +109,15 @@ def launch_consolidated(
|
||||
guest_ip: str,
|
||||
image_ref: str = "",
|
||||
tokens: dict[str, str] | None = None,
|
||||
service: OrchestratorService | None = None,
|
||||
gateway_name: str = _FC_GATEWAY_NAME,
|
||||
) -> LaunchContext:
|
||||
"""Ensure the infra VM is up, register the bottle by its guest IP, and
|
||||
provision its git-gate state into the gateway VM. Returns the context the
|
||||
agent-VM launch needs. Raises on failure — the caller tears down."""
|
||||
infra = infra_vm.ensure_running()
|
||||
url = infra.control_plane_url
|
||||
"""Ensure the orchestrator + Firecracker gateway are up, register the
|
||||
bottle by its guest IP, and provision its git-gate state. Returns the
|
||||
context the VM launch needs. Raises `ConsolidatedLaunchError` (or the
|
||||
primitives' own errors) on failure — the caller tears down on failure."""
|
||||
service = service or _FirecrackerOrchestratorService()
|
||||
url = service.ensure_running()
|
||||
client = OrchestratorClient(url)
|
||||
|
||||
inputs = registration_inputs(egress_plan)
|
||||
@@ -74,30 +126,33 @@ def launch_consolidated(
|
||||
metadata=inputs.metadata, tokens=tokens,
|
||||
)
|
||||
try:
|
||||
provision_git_gate(
|
||||
infra_vm.gateway_transport(), reg.bottle_id, git_gate_plan)
|
||||
provision_git_gate(gateway_name, reg.bottle_id, git_gate_plan)
|
||||
except Exception:
|
||||
client.teardown_bottle(reg.bottle_id)
|
||||
raise
|
||||
|
||||
# The shared gateway CA every agent on this host trusts for TLS
|
||||
# interception — fetched from the infra VM over SSH.
|
||||
# Fetch the shared gateway CA here so the caller can install it in the
|
||||
# rootfs (the same CA every agent on this host trusts for TLS interception).
|
||||
gateway_ca_pem = DockerGateway(
|
||||
name=gateway_name, network=GATEWAY_NETWORK,
|
||||
).ca_cert_pem()
|
||||
|
||||
return LaunchContext(
|
||||
bottle_id=reg.bottle_id,
|
||||
identity_token=reg.identity_token,
|
||||
source_ip=guest_ip,
|
||||
gateway_ca_pem=infra.gateway_ca_pem(),
|
||||
gateway_ca_pem=gateway_ca_pem,
|
||||
orchestrator_url=url,
|
||||
)
|
||||
|
||||
|
||||
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
|
||||
"""Deregister the bottle and remove its git-gate state from the gateway
|
||||
VM. Both steps are idempotent so this is safe from a cleanup trap. Does
|
||||
NOT stop the infra VM — it's a persistent per-host singleton shared by
|
||||
every bottle."""
|
||||
def teardown_consolidated(
|
||||
bottle_id: str, *, orchestrator_url: str, gateway_name: str = _FC_GATEWAY_NAME,
|
||||
) -> None:
|
||||
"""Deregister the bottle and remove its git-gate state from the gateway.
|
||||
Both steps are idempotent so this is safe from a cleanup trap."""
|
||||
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
|
||||
deprovision_git_gate(infra_vm.gateway_transport(), bottle_id)
|
||||
deprovision_git_gate(gateway_name, bottle_id)
|
||||
|
||||
|
||||
__all__ = [
|
||||
|
||||
@@ -78,32 +78,20 @@ def _config(
|
||||
vcpus: int,
|
||||
mem_mib: int,
|
||||
guest_mac: str,
|
||||
data_drive: Path | None = None,
|
||||
) -> dict[str, object]:
|
||||
drives: list[dict[str, object]] = [
|
||||
{
|
||||
"drive_id": "rootfs",
|
||||
"path_on_host": str(rootfs),
|
||||
"is_root_device": True,
|
||||
"is_read_only": False,
|
||||
}
|
||||
]
|
||||
# A second virtio-block device (guest /dev/vdb) — the infra VM's
|
||||
# persistent registry "volume", a host-side ext4 file that outlives the
|
||||
# ephemeral rootfs across VM restarts.
|
||||
if data_drive is not None:
|
||||
drives.append({
|
||||
"drive_id": "data",
|
||||
"path_on_host": str(data_drive),
|
||||
"is_root_device": False,
|
||||
"is_read_only": False,
|
||||
})
|
||||
return {
|
||||
"boot-source": {
|
||||
"kernel_image_path": str(util.kernel_path()),
|
||||
"boot_args": _boot_args(guest_ip, host_ip, pubkey),
|
||||
},
|
||||
"drives": drives,
|
||||
"drives": [
|
||||
{
|
||||
"drive_id": "rootfs",
|
||||
"path_on_host": str(rootfs),
|
||||
"is_root_device": True,
|
||||
"is_read_only": False,
|
||||
}
|
||||
],
|
||||
"network-interfaces": [
|
||||
{
|
||||
"iface_id": "eth0",
|
||||
@@ -130,16 +118,9 @@ def boot(
|
||||
vcpus: int = 2,
|
||||
mem_mib: int = 2048,
|
||||
guest_mac: str = "06:00:AC:10:00:02",
|
||||
detached: bool = False,
|
||||
data_drive: Path | None = None,
|
||||
) -> VmHandle:
|
||||
"""Write the config and launch the VMM. Returns once the process is
|
||||
spawned; callers wait for SSH readiness separately.
|
||||
|
||||
`detached` starts the VMM in its own session (`start_new_session`) so it
|
||||
survives the launcher exiting — used for the persistent per-host infra
|
||||
VM, which must outlive the short-lived `start` process (agent VMs stay
|
||||
attached and are torn down with the launcher)."""
|
||||
spawned; callers wait for SSH readiness separately."""
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
config_path = run_dir / "config.json"
|
||||
console_log = run_dir / "console.log"
|
||||
@@ -147,7 +128,6 @@ def boot(
|
||||
_config(
|
||||
rootfs=rootfs, tap=tap, guest_ip=guest_ip, host_ip=host_ip,
|
||||
pubkey=pubkey, vcpus=vcpus, mem_mib=mem_mib, guest_mac=guest_mac,
|
||||
data_drive=data_drive,
|
||||
),
|
||||
indent=2,
|
||||
))
|
||||
@@ -157,7 +137,6 @@ def boot(
|
||||
process = subprocess.Popen(
|
||||
["firecracker", "--no-api", "--config-file", str(config_path)],
|
||||
stdout=log_fh, stderr=subprocess.STDOUT, stdin=subprocess.DEVNULL,
|
||||
start_new_session=detached,
|
||||
)
|
||||
return VmHandle(process=process, guest_ip=guest_ip, console_log=console_log)
|
||||
|
||||
|
||||
@@ -1,12 +1,9 @@
|
||||
"""FirecrackerFreezer — snapshot a running microVM to a rootfs tar.
|
||||
"""FirecrackerFreezer — snapshot a running microVM to a Docker image.
|
||||
|
||||
The VM is live and can't be block-copied safely, so — like the macOS
|
||||
backend — we stream the guest root filesystem out over the control
|
||||
channel (SSH here). Unlike the other backends this needs no Docker: the
|
||||
tar *is* the resumable artifact. `resume` extracts it and rebuilds a
|
||||
fresh per-bottle ext4 with `mke2fs -d` (see `util.build_committed_rootfs_dir`
|
||||
and `launch._build_agent_base`). The bottle keeps running after the
|
||||
snapshot.
|
||||
channel (SSH here) and rebuild an image from it. The bottle keeps
|
||||
running after the snapshot.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -14,9 +11,9 @@ from __future__ import annotations
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
|
||||
from ...bottle_state import committed_rootfs_path
|
||||
from ...log import die, info
|
||||
from .. import ActiveAgent
|
||||
from ..freeze import Freezer
|
||||
@@ -33,13 +30,14 @@ class FirecrackerFreezer(Freezer):
|
||||
if not private_key.is_file() or not guest_ip:
|
||||
die(f"cannot freeze {agent.slug}: run dir {run_dir} is missing the "
|
||||
f"SSH key or VM config (is the bottle still running?)")
|
||||
tar_path = committed_rootfs_path(agent.slug)
|
||||
_commit_rootfs_via_ssh(private_key, guest_ip, tar_path)
|
||||
info(f"committed {agent.slug} -> {tar_path}")
|
||||
return str(tar_path)
|
||||
image_tag = f"bot-bottle-committed-{agent.slug}:latest"
|
||||
_commit_via_ssh(private_key, guest_ip, image_tag)
|
||||
info(f"committed {agent.slug} -> {image_tag!r}")
|
||||
return image_tag
|
||||
|
||||
def _export_hint(self, slug: str, image_ref: str) -> None:
|
||||
info(f"to export for migration: cp {image_ref} {slug}.tar")
|
||||
info(f"to export for migration: docker image save {image_ref} "
|
||||
f"-o {slug}.tar")
|
||||
|
||||
|
||||
def _guest_ip_from_config(config_path: Path) -> str:
|
||||
@@ -55,36 +53,24 @@ def _guest_ip_from_config(config_path: Path) -> str:
|
||||
return ""
|
||||
|
||||
|
||||
def _commit_rootfs_via_ssh(private_key: Path, guest_ip: str, tar_path: Path) -> None:
|
||||
"""Stream the guest rootfs out over SSH into `tar_path`. Excludes the
|
||||
virtual/live mounts (proc/sys/dev/run) — resume recreates those empty
|
||||
mount points. Written to a `.partial` sibling and renamed on success so
|
||||
a failed freeze never leaves a truncated artifact in its place."""
|
||||
tar_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
partial = tar_path.with_name(tar_path.name + ".partial")
|
||||
ssh = util.ssh_base_argv(private_key, guest_ip)
|
||||
# The snapshot can contain the bottle's private workspace, so keep it
|
||||
# owner-only (0600) for the whole stream. The `os.open` mode only applies
|
||||
# on *creation*, so unlink any leftover partial (a prior interrupted run
|
||||
# could have left it world-readable, or something could swap in a symlink
|
||||
# at this predictable name) and exclusively recreate it — O_EXCL|O_NOFOLLOW
|
||||
# — then fchmod immediately so umask can't loosen it. Re-assert after the
|
||||
# rename too (os.replace carries the source mode, but be explicit).
|
||||
partial.unlink(missing_ok=True)
|
||||
fd = os.open(
|
||||
partial, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o600
|
||||
)
|
||||
os.fchmod(fd, 0o600)
|
||||
with os.fdopen(fd, "wb") as tar_out:
|
||||
result = subprocess.run(
|
||||
[*ssh, "--", "tar", "--create", "--one-file-system",
|
||||
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
|
||||
"--exclude=./run", "--file=-", "--directory=/", "."],
|
||||
stdout=tar_out, stderr=subprocess.PIPE, check=False,
|
||||
def _commit_via_ssh(private_key: Path, guest_ip: str, image_tag: str) -> None:
|
||||
with tempfile.TemporaryDirectory(prefix="bot-bottle-fc-commit.") as tmp:
|
||||
rootfs_tar = os.path.join(tmp, "rootfs.tar")
|
||||
ssh = util.ssh_base_argv(private_key, guest_ip)
|
||||
with open(rootfs_tar, "wb") as tar_out:
|
||||
result = subprocess.run(
|
||||
[*ssh, "--", "tar", "--create", "--one-file-system",
|
||||
"--exclude=./proc", "--exclude=./sys", "--exclude=./dev",
|
||||
"--exclude=./run", "--file=-", "--directory=/", "."],
|
||||
stdout=tar_out, stderr=subprocess.PIPE, check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
die(f"ssh tar for {guest_ip} failed: "
|
||||
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
|
||||
with open(os.path.join(tmp, "Dockerfile"), "w", encoding="utf-8") as f:
|
||||
f.write("FROM scratch\nADD rootfs.tar /\nUSER node\nWORKDIR /home/node\n")
|
||||
build = subprocess.run(
|
||||
["docker", "build", "-t", image_tag, tmp], check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
partial.unlink(missing_ok=True)
|
||||
die(f"ssh tar for {guest_ip} failed: "
|
||||
f"{(result.stderr or b'').decode().strip() or '<no stderr>'}")
|
||||
os.replace(partial, tar_path)
|
||||
os.chmod(tar_path, 0o600)
|
||||
if build.returncode != 0:
|
||||
die(f"docker build for {image_tag!r} failed")
|
||||
|
||||
@@ -1,224 +0,0 @@
|
||||
"""Docker-free agent-image builds for the Firecracker backend (PRD 0069 Stage 3).
|
||||
|
||||
Agent Dockerfiles build **inside the persistent per-host infra VM**
|
||||
(`infra_vm.py`), which carries buildah (rootless, daemonless): no host Docker
|
||||
daemon, no root-equivalent `docker` group. The build runs over SSH against the
|
||||
infra VM and its rootfs streams back to the host, where the existing
|
||||
`mke2fs -d` path (`util.build_rootfs_ext4`) turns it into a bootable ext4.
|
||||
|
||||
Building in the infra VM — rather than a throwaway builder VM — means there is
|
||||
one buildah image (`bot-bottle-infra`) and no contention for the orchestrator
|
||||
TAP. Tradeoff: an untrusted Dockerfile's `RUN` steps share the VM with the
|
||||
control plane + gateway (buildah `--isolation chroot` isn't a hard boundary) —
|
||||
the accepted single-VM blast-radius tradeoff, re-splittable into a disposable
|
||||
builder (booted from this same image on its own TAP) later.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
from contextlib import contextmanager
|
||||
from pathlib import Path
|
||||
from typing import Generator
|
||||
|
||||
from ...log import die, info
|
||||
from . import infra_vm, util
|
||||
|
||||
# vfs + chroot: buildah works as root in the microVM (no fuse-overlayfs /
|
||||
# overlay module / subuid maps). `--isolation` is a build/run-only flag;
|
||||
# `from`/`mount` take just the store.
|
||||
_BUILD_FLAGS = "--isolation chroot --storage-driver vfs"
|
||||
_STORE_FLAG = "--storage-driver vfs"
|
||||
|
||||
_BUILD_TIMEOUT_SECONDS = 900.0
|
||||
|
||||
|
||||
def _dockerfile_hash(dockerfile: Path) -> str:
|
||||
"""Cache key: the Dockerfile's content. The shipped agent Dockerfiles
|
||||
COPY nothing from the build context (see .dockerignore), so their content
|
||||
fully determines the image; a Dockerfile that adds COPY will want the
|
||||
context folded in here too."""
|
||||
return hashlib.sha256(dockerfile.read_bytes()).hexdigest()[:16]
|
||||
|
||||
|
||||
def build_agent_rootfs_dir(
|
||||
dockerfile: Path, *, image_tag: str, smoke_test: tuple[str, ...] = (),
|
||||
) -> Path:
|
||||
"""Build `dockerfile` in the infra VM (buildah, no host docker), export its
|
||||
rootfs, inject the guest boot bits, and return the cached base dir — the
|
||||
same shape `util.build_rootfs_ext4` consumes. Cached by Dockerfile content,
|
||||
so a repeat launch skips the rebuild.
|
||||
|
||||
`smoke_test` (the provider's declared argv, e.g. `("claude","--version")`)
|
||||
is run in the freshly built image before export, catching an npm
|
||||
silent-failure image at build time rather than at first agent use."""
|
||||
digest = _dockerfile_hash(dockerfile)
|
||||
base = util.cache_dir() / "rootfs" / f"agent-{digest}"
|
||||
if (base / ".bb-ready").is_file():
|
||||
info(f"using cached agent rootfs {base.name}")
|
||||
return base
|
||||
|
||||
# Serialize builds: the infra VM's buildah store + this cache dir are
|
||||
# shared, so concurrent `start`s must not build into them at once. The
|
||||
# lock covers the cache lookup + build + atomic publish; the ready
|
||||
# fast-path above takes no lock.
|
||||
with _build_lock():
|
||||
if (base / ".bb-ready").is_file(): # another build finished while we waited
|
||||
info(f"using cached agent rootfs {base.name}")
|
||||
return base
|
||||
# Build into a temp dir and publish by atomic rename, so a partial
|
||||
# build is never visible as `agent-<digest>`.
|
||||
staging = util.cache_dir() / "rootfs" / f".building-{digest}"
|
||||
shutil.rmtree(staging, ignore_errors=True)
|
||||
staging.mkdir(parents=True)
|
||||
info(f"building agent image {image_tag!r} in the infra VM")
|
||||
_build_in_infra(dockerfile, staging, smoke_test, digest)
|
||||
util.inject_guest_boot(staging)
|
||||
(staging / ".bb-ready").write_text("ok\n")
|
||||
shutil.rmtree(base, ignore_errors=True)
|
||||
os.rename(staging, base)
|
||||
return base
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _build_lock() -> Generator[None, None, None]:
|
||||
"""Host-level exclusive lock serializing agent-image builds (shared infra
|
||||
buildah store + cache dir). flock auto-releases on a crash."""
|
||||
lock_path = util.cache_dir() / "rootfs" / ".build.lock"
|
||||
lock_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
handle = open(lock_path, "w", encoding="utf-8")
|
||||
try:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX)
|
||||
yield
|
||||
finally:
|
||||
handle.close()
|
||||
|
||||
|
||||
def _build_in_infra(
|
||||
dockerfile: Path, base: Path, smoke_test: tuple[str, ...], digest: str,
|
||||
) -> None:
|
||||
"""Ensure the infra VM is up, `buildah build` the Dockerfile in it, smoke
|
||||
test the image, and stream its rootfs into `base`. The infra VM persists;
|
||||
only the per-build container/image/context are cleaned up."""
|
||||
infra = infra_vm.ensure_running()
|
||||
key, ip = infra.private_key, infra.guest_ip
|
||||
tag = f"bot-bottle-agent-build-{digest}"
|
||||
ctx = f"/tmp/agent-build-{digest}"
|
||||
smoke_ctr, export_ctr = f"{tag}-smoke", f"{tag}-export"
|
||||
|
||||
def _cleanup() -> None:
|
||||
# Remove only THIS build's working containers/image/context — never
|
||||
# `buildah rm -a`, which would nuke a concurrent build's container.
|
||||
_ssh(key, ip,
|
||||
f"buildah rm {smoke_ctr} {export_ctr} >/dev/null 2>&1; "
|
||||
f"buildah rmi {_STORE_FLAG} {tag} >/dev/null 2>&1; rm -rf {ctx}",
|
||||
timeout=60)
|
||||
|
||||
_cleanup() # clear leftovers from a crashed prior build of this digest
|
||||
try:
|
||||
prep = _ssh(key, ip, f"mkdir -p {ctx}/ctx")
|
||||
if prep.returncode != 0:
|
||||
die(f"preparing build dir in the infra VM failed: {prep.stderr.strip()}")
|
||||
_send_dockerfile(key, ip, dockerfile, ctx)
|
||||
_buildah_build(key, ip, ctx, tag)
|
||||
_smoke_test(key, ip, tag, smoke_ctr, smoke_test)
|
||||
_stream_rootfs(key, ip, tag, export_ctr, base)
|
||||
finally:
|
||||
_cleanup()
|
||||
|
||||
|
||||
def _ssh(private_key: Path, guest_ip: str, script: str,
|
||||
*, timeout: float = 60.0) -> subprocess.CompletedProcess[str]:
|
||||
return subprocess.run(
|
||||
util.ssh_base_argv(private_key, guest_ip) + [script],
|
||||
capture_output=True, text=True, timeout=timeout, check=False,
|
||||
)
|
||||
|
||||
|
||||
def _ssh_streamed(private_key: Path, guest_ip: str, script: str,
|
||||
*, timeout: float) -> int:
|
||||
"""Run an SSH command letting the remote's stdout/stderr flow straight to
|
||||
ours (no capture), for long chatty steps where live progress beats a
|
||||
silent wait. Returns the exit code."""
|
||||
proc = subprocess.run(
|
||||
util.ssh_base_argv(private_key, guest_ip) + [script],
|
||||
timeout=timeout, check=False,
|
||||
)
|
||||
return proc.returncode
|
||||
|
||||
|
||||
def _send_dockerfile(private_key: Path, guest_ip: str, dockerfile: Path, ctx: str) -> None:
|
||||
proc = subprocess.run(
|
||||
util.ssh_base_argv(private_key, guest_ip) + [f"cat > {ctx}/Dockerfile"],
|
||||
input=dockerfile.read_bytes(), capture_output=True, timeout=30, check=False,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
die(f"sending Dockerfile to the infra VM failed: "
|
||||
f"{proc.stderr.decode(errors='replace').strip()}")
|
||||
|
||||
|
||||
def _buildah_build(private_key: Path, guest_ip: str, ctx: str, tag: str) -> None:
|
||||
# Stream buildah's step-by-step output straight to our stderr (like the
|
||||
# docker backend's `docker build`), so a long first build (base pull +
|
||||
# apt/npm installs) shows live progress instead of a silent wait. The
|
||||
# remote stderr is where buildah writes its `STEP i/n` lines.
|
||||
info(f"buildah build {tag} in the infra VM (streaming output)")
|
||||
rc = _ssh_streamed(
|
||||
private_key, guest_ip,
|
||||
f"buildah build {_BUILD_FLAGS} -t {tag} -f {ctx}/Dockerfile {ctx}/ctx",
|
||||
timeout=_BUILD_TIMEOUT_SECONDS,
|
||||
)
|
||||
if rc != 0:
|
||||
die(f"buildah build in the infra VM failed (exit {rc}); "
|
||||
"see the build output above.")
|
||||
|
||||
|
||||
def _smoke_test(private_key: Path, guest_ip: str, tag: str, ctr: str,
|
||||
argv: tuple[str, ...]) -> None:
|
||||
"""Run the provider's smoke argv inside the freshly built image
|
||||
(`buildah run`, which uses the image's own PATH), failing the build
|
||||
loudly if the CLI is a broken stub. No-op without a declared test. Uses a
|
||||
named working container (`ctr`) so cleanup is scoped to this build."""
|
||||
if not argv:
|
||||
return
|
||||
cmd = (
|
||||
f"set -e; buildah from {_STORE_FLAG} --name {ctr} {tag} >/dev/null; "
|
||||
f"buildah run {_BUILD_FLAGS} {ctr} -- {' '.join(argv)}; rc=$?; "
|
||||
f"buildah rm {ctr} >/dev/null 2>&1 || true; exit $rc"
|
||||
)
|
||||
result = _ssh(private_key, guest_ip, cmd, timeout=120)
|
||||
if result.returncode != 0:
|
||||
detail = (result.stdout + result.stderr).strip().splitlines()[-10:]
|
||||
die(f"agent image failed its post-build smoke test "
|
||||
f"({' '.join(argv)}):\n" + "\n".join(detail))
|
||||
|
||||
|
||||
def _stream_rootfs(private_key: Path, guest_ip: str, tag: str, ctr: str, base: Path) -> None:
|
||||
"""`buildah mount` the built image in the infra VM and pipe its rootfs tar
|
||||
straight into `base` on the host (extracted as the non-root host user, so
|
||||
uid 0 isn't preserved — the guest init restores /root ownership). Uses a
|
||||
named working container so cleanup is scoped to this build."""
|
||||
export = (
|
||||
f"set -e; buildah from {_STORE_FLAG} --name {ctr} {tag} >/dev/null; "
|
||||
f"mnt=$(buildah mount {_STORE_FLAG} {ctr}); "
|
||||
f"tar -C \"$mnt\" -cf - ."
|
||||
)
|
||||
ssh_proc = subprocess.Popen(
|
||||
util.ssh_base_argv(private_key, guest_ip) + [export],
|
||||
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
|
||||
)
|
||||
assert ssh_proc.stdout is not None
|
||||
untar = subprocess.run(
|
||||
["tar", "-x", "-C", str(base)], stdin=ssh_proc.stdout, check=False,
|
||||
)
|
||||
ssh_proc.stdout.close()
|
||||
ssh_err = (ssh_proc.stderr.read().decode(errors="replace")
|
||||
if ssh_proc.stderr else "")
|
||||
rc = ssh_proc.wait()
|
||||
if rc != 0 or untar.returncode != 0:
|
||||
die(f"exporting the built rootfs from the infra VM failed: "
|
||||
f"{ssh_err.strip() or '<no stderr>'}")
|
||||
@@ -1,199 +0,0 @@
|
||||
"""Prebuilt infra-VM rootfs, pulled as an artifact (PRD 0069 Stage 2).
|
||||
|
||||
The Firecracker infra VM boots a fixed rootfs (orchestrator control plane +
|
||||
gateway + buildah, control-plane init as PID 1) that does not vary per launch —
|
||||
the per-boot bits (authorized_keys, guest IP) ride the kernel cmdline, so one
|
||||
rootfs boots on any host. Instead of building that rootfs on the launch host
|
||||
with Docker, we build it **off-host** and publish it as a versioned, ready-to-
|
||||
boot ext4 (gzip-compressed) to a Gitea **generic package**; the launch host
|
||||
downloads + verifies + boots it. No Docker, no image tooling on the launch
|
||||
host — just an HTTP fetch and gunzip.
|
||||
|
||||
publish (off-host, see publish_infra.py):
|
||||
docker build -> rootfs dir -> mke2fs -> gzip -> PUT generic package
|
||||
pull (this module, launch host):
|
||||
GET .../rootfs.ext4.gz (+ .sha256) -> verify -> gunzip -> boot
|
||||
|
||||
The artifact **version** is a content hash of everything baked into the rootfs
|
||||
(the shipped bot_bottle package, the three Dockerfiles, and the init), so a
|
||||
launch host always pulls the artifact matching its code and a content change
|
||||
can't silently boot a stale rootfs. A checksum mismatch fails closed.
|
||||
|
||||
Set `BOT_BOTTLE_INFRA_BUILD=local` to skip the pull and build the rootfs
|
||||
locally with Docker (dev iteration on the Dockerfiles) — see `infra_vm`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import hashlib
|
||||
import os
|
||||
import shutil
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from ...log import die, info
|
||||
from . import util
|
||||
|
||||
# Bump if the on-disk artifact *format* changes (compression, layout) so a new
|
||||
# scheme can't collide with a cached/published artifact of the old one.
|
||||
_ARTIFACT_FORMAT = "1"
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
_DOCKERFILES = ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra")
|
||||
|
||||
_DEFAULT_BASE = "https://gitea.dideric.is"
|
||||
_DEFAULT_OWNER = "didericis"
|
||||
_PACKAGE = "bot-bottle-firecracker-infra"
|
||||
|
||||
# Streaming copy chunk for the (hundreds-of-MB) download.
|
||||
_CHUNK = 1 << 20
|
||||
|
||||
|
||||
def local_build_requested() -> bool:
|
||||
"""True when the operator opted into the dev Docker-build path instead of
|
||||
pulling the published artifact (`BOT_BOTTLE_INFRA_BUILD=local`)."""
|
||||
return os.environ.get("BOT_BOTTLE_INFRA_BUILD", "").strip().lower() == "local"
|
||||
|
||||
|
||||
def infra_artifact_version(init_script: str, *, repo_root: Path = _REPO_ROOT) -> str:
|
||||
"""Content hash (16 hex) of everything baked into the infra rootfs: the
|
||||
whole shipped `bot_bottle` package, the three fixed Dockerfiles, and the
|
||||
guest init. Deterministic across the publish host and the launch host when
|
||||
both run the same checkout, so the tag the launch host pulls is exactly the
|
||||
tag publish produced.
|
||||
|
||||
The package is `COPY bot_bottle /app/bot_bottle`'d wholesale into the image,
|
||||
so hash *every* regular file under it — not just `*.py`. Non-Python inputs
|
||||
(e.g. `egress_entrypoint.sh`, `netpool.defaults.env`) are baked in too, and
|
||||
a change to one must bump the version or a launch host could boot a stale
|
||||
rootfs whose code differs from its checkout. `__pycache__`/`.pyc` are the
|
||||
only exclusions — build artifacts, never copied."""
|
||||
h = hashlib.sha256()
|
||||
h.update(f"format={_ARTIFACT_FORMAT}\n".encode())
|
||||
pkg = repo_root / "bot_bottle"
|
||||
for path in sorted(pkg.rglob("*")):
|
||||
if not path.is_file():
|
||||
continue
|
||||
if "__pycache__" in path.parts or path.suffix == ".pyc":
|
||||
continue
|
||||
h.update(str(path.relative_to(repo_root)).encode())
|
||||
h.update(b"\0")
|
||||
h.update(path.read_bytes())
|
||||
for name in _DOCKERFILES:
|
||||
h.update(name.encode())
|
||||
h.update(b"\0")
|
||||
h.update((repo_root / name).read_bytes())
|
||||
h.update(b"init\0")
|
||||
h.update(init_script.encode())
|
||||
return h.hexdigest()[:16]
|
||||
|
||||
|
||||
def _config() -> tuple[str, str, str]:
|
||||
"""(base_url, owner, token) for the generic-package endpoint. Base + owner
|
||||
are overridable for other deployments / mirrors; the token comes solely from
|
||||
`BOT_BOTTLE_INFRA_ARTIFACT_TOKEN` (a dedicated package-scoped token, kept
|
||||
separate from the general-purpose Gitea token) and is optional — a public
|
||||
package needs none to pull."""
|
||||
base = os.environ.get("BOT_BOTTLE_INFRA_ARTIFACT_BASE", _DEFAULT_BASE).rstrip("/")
|
||||
owner = os.environ.get("BOT_BOTTLE_INFRA_ARTIFACT_OWNER", _DEFAULT_OWNER)
|
||||
token = os.environ.get("BOT_BOTTLE_INFRA_ARTIFACT_TOKEN", "")
|
||||
return base, owner, token
|
||||
|
||||
|
||||
def artifact_url(version: str, filename: str) -> str:
|
||||
"""The generic-package download URL for one file of this version's
|
||||
artifact (`rootfs.ext4.gz` / `rootfs.ext4.gz.sha256`)."""
|
||||
base, owner, _ = _config()
|
||||
return f"{base}/api/packages/{owner}/generic/{_PACKAGE}/{version}/{filename}"
|
||||
|
||||
|
||||
_GZ_NAME = "rootfs.ext4.gz"
|
||||
_SHA_NAME = "rootfs.ext4.gz.sha256"
|
||||
|
||||
|
||||
def _cache_root(version: str) -> Path:
|
||||
return util.cache_dir() / "infra-artifact" / version
|
||||
|
||||
|
||||
def _open(url: str) -> urllib.request.Request:
|
||||
_, _, token = _config()
|
||||
req = urllib.request.Request(url)
|
||||
if token:
|
||||
req.add_header("Authorization", f"token {token}")
|
||||
return req
|
||||
|
||||
|
||||
def _download(url: str, dest: Path) -> None:
|
||||
"""Stream `url` to `dest` (atomic via a `.part` sibling)."""
|
||||
tmp = dest.with_suffix(dest.suffix + ".part")
|
||||
try:
|
||||
with urllib.request.urlopen(_open(url)) as resp, open(tmp, "wb") as out:
|
||||
shutil.copyfileobj(resp, out, _CHUNK)
|
||||
except urllib.error.HTTPError as e:
|
||||
tmp.unlink(missing_ok=True)
|
||||
if e.code == 404:
|
||||
die(
|
||||
f"infra artifact not published for this code version.\n"
|
||||
f" missing: {url}\n"
|
||||
f" publish it from a build host (Docker):\n"
|
||||
f" python3 -m bot_bottle.backend.firecracker.publish_infra\n"
|
||||
f" or build the rootfs locally: BOT_BOTTLE_INFRA_BUILD=local"
|
||||
)
|
||||
die(f"downloading infra artifact failed (HTTP {e.code}): {url}")
|
||||
except urllib.error.URLError as e:
|
||||
tmp.unlink(missing_ok=True)
|
||||
die(f"infra artifact registry unreachable: {url} ({e.reason})")
|
||||
tmp.replace(dest)
|
||||
|
||||
|
||||
def _sha256_file(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as fh:
|
||||
for chunk in iter(lambda: fh.read(_CHUNK), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def ensure_artifact_gz(version: str) -> Path:
|
||||
"""The verified, cached `rootfs.ext4.gz` for `version` — downloading it (and
|
||||
its `.sha256`) once, then reusing it. Fail-closed on a checksum mismatch:
|
||||
the partial is removed and we die rather than boot an unverified rootfs."""
|
||||
root = _cache_root(version)
|
||||
root.mkdir(parents=True, exist_ok=True)
|
||||
gz = root / _GZ_NAME
|
||||
ok = root / ".verified"
|
||||
if gz.is_file() and ok.is_file():
|
||||
return gz
|
||||
|
||||
info(f"pulling infra rootfs artifact {_PACKAGE}/{version}")
|
||||
_download(artifact_url(version, _GZ_NAME), gz)
|
||||
sha = root / _SHA_NAME
|
||||
_download(artifact_url(version, _SHA_NAME), sha)
|
||||
|
||||
expected = sha.read_text().split()[0].strip().lower()
|
||||
actual = _sha256_file(gz)
|
||||
if actual != expected:
|
||||
gz.unlink(missing_ok=True)
|
||||
sha.unlink(missing_ok=True)
|
||||
die(
|
||||
f"infra artifact checksum mismatch for {version}:\n"
|
||||
f" expected {expected}\n"
|
||||
f" actual {actual}\n"
|
||||
f" refusing to boot an unverified rootfs."
|
||||
)
|
||||
ok.write_text("ok\n")
|
||||
return gz
|
||||
|
||||
|
||||
def materialize_ext4(version: str, dest: Path) -> None:
|
||||
"""Ensure the verified artifact is cached, then gunzip it to `dest` — a
|
||||
fresh, writable per-boot rootfs (the VM mutates it; the cached `.gz` stays
|
||||
pristine). Atomic via a `.part` sibling."""
|
||||
gz = ensure_artifact_gz(version)
|
||||
tmp = dest.with_suffix(dest.suffix + ".part")
|
||||
info(f"expanding infra rootfs -> {dest}")
|
||||
with gzip.open(gz, "rb") as src, open(tmp, "wb") as out:
|
||||
shutil.copyfileobj(src, out, _CHUNK)
|
||||
tmp.replace(dest)
|
||||
@@ -1,440 +0,0 @@
|
||||
"""The per-host infra VM for the Firecracker backend (PRD 0070 Stage B).
|
||||
|
||||
A single persistent microVM that runs the orchestrator **control plane** (and,
|
||||
in a following step, the gateway **data plane**) — the trusted per-host service
|
||||
the docker backend runs as containers. It boots on the NAT'd orchestrator link
|
||||
(`netpool.orch_slot()`): the host CLI reaches its control plane over HTTP at the
|
||||
guest IP, and agent VMs reach its gateway ports over VM-to-VM routing.
|
||||
|
||||
Build-from-source (the default while the design churns): the rootfs is exported
|
||||
from the locally built orchestrator image, which bakes the stdlib-only
|
||||
control-plane source. A pull-from-registry mode (Gitea's OCI registry) becomes
|
||||
the default later.
|
||||
|
||||
SSH is left enabled for debugging; the control plane is the load-bearing
|
||||
surface.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import fcntl
|
||||
import hashlib
|
||||
import os
|
||||
import shlex
|
||||
import signal
|
||||
import stat
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from contextlib import contextmanager
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import Generator
|
||||
|
||||
from ...log import die, info
|
||||
from ..docker import util as docker_mod
|
||||
from ..docker.gateway_provision import GatewayProvisionError
|
||||
from . import firecracker_vm, infra_artifact, netpool, util
|
||||
|
||||
# The single infra-VM image: gateway data plane + baked control-plane source
|
||||
# (Dockerfile.infra FROM the gateway image). Built from source by default;
|
||||
# a pull-from-registry mode lands later.
|
||||
_INFRA_IMAGE = "bot-bottle-infra:latest"
|
||||
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
|
||||
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
CONTROL_PLANE_PORT = 8099
|
||||
# Gateway data-plane ports (agent-facing): egress proxy, supervise MCP,
|
||||
# git-http. Reached by agent VMs over VM-to-VM routing (added next).
|
||||
EGRESS_PORT = 9099
|
||||
SUPERVISE_PORT = 9100
|
||||
GIT_HTTP_PORT = 9420
|
||||
# mitmproxy writes its CA here a beat after start; agents install it to trust
|
||||
# the gateway's TLS interception.
|
||||
_GATEWAY_CA_PATH = "/home/mitmproxy/.mitmproxy/mitmproxy-ca-cert.pem"
|
||||
|
||||
# The infra VM makes direct upstream connections (gateway egress, and buildah
|
||||
# during builds), and the kernel `ip=` cmdline sets no resolver. Public for
|
||||
# now; routing DNS through a filtered path is a later refinement.
|
||||
_INFRA_RESOLVER = "1.1.1.1"
|
||||
|
||||
_HEALTH_TIMEOUT_SECONDS = 45.0
|
||||
_HEALTH_POLL_SECONDS = 0.5
|
||||
_CA_TIMEOUT_SECONDS = 30.0
|
||||
|
||||
|
||||
@dataclass
|
||||
class InfraVm:
|
||||
"""A handle to the per-host infra VM: its guest IP and the stable SSH key
|
||||
used to fetch the gateway CA / provision git-gate. `vm` is the live VMM
|
||||
handle when this process booted it, and None when adopting a singleton a
|
||||
prior launcher started (teardown then goes through the PID file)."""
|
||||
|
||||
guest_ip: str
|
||||
private_key: Path
|
||||
vm: firecracker_vm.VmHandle | None = None
|
||||
|
||||
@property
|
||||
def control_plane_url(self) -> str:
|
||||
return f"http://{self.guest_ip}:{CONTROL_PLANE_PORT}"
|
||||
|
||||
def terminate(self) -> None:
|
||||
"""Stop the infra VM — via the live handle if we booted it, else the
|
||||
PID file (adopting-process case)."""
|
||||
if self.vm is not None:
|
||||
self.vm.terminate()
|
||||
else:
|
||||
_kill_pidfile()
|
||||
_pid_file().unlink(missing_ok=True)
|
||||
|
||||
def gateway_ca_pem(self, *, timeout: float = _CA_TIMEOUT_SECONDS) -> str:
|
||||
"""The gateway's mitmproxy CA (PEM) that agents install to trust its
|
||||
TLS interception. Generated a moment after boot, so this polls over
|
||||
SSH until it appears (mirrors DockerGateway.ca_cert_pem)."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
proc = subprocess.run(
|
||||
util.ssh_base_argv(self.private_key, self.guest_ip)
|
||||
+ [f"cat {_GATEWAY_CA_PATH}"],
|
||||
capture_output=True, text=True, timeout=15, check=False,
|
||||
)
|
||||
if proc.returncode == 0 and "BEGIN CERTIFICATE" in proc.stdout:
|
||||
return proc.stdout
|
||||
if time.monotonic() >= deadline:
|
||||
die(f"gateway CA not available after {timeout:g}s: "
|
||||
f"{proc.stderr.strip() or 'empty'}")
|
||||
time.sleep(_HEALTH_POLL_SECONDS)
|
||||
|
||||
|
||||
def ensure_built() -> None:
|
||||
"""Ensure the infra rootfs is available before boot.
|
||||
|
||||
Default (docker-free, PRD 0069 Stage 2): download + verify the prebuilt
|
||||
rootfs artifact matching this code version (see `infra_artifact`); the
|
||||
launch host needs no Docker. `BOT_BOTTLE_INFRA_BUILD=local` instead builds
|
||||
the three fixed images from source with host Docker — the infra image
|
||||
`COPY --from`s the orchestrator image and is `FROM` the gateway image, so
|
||||
both must exist first — for iterating on the Dockerfiles."""
|
||||
if infra_artifact.local_build_requested():
|
||||
build_infra_images_with_docker()
|
||||
return
|
||||
infra_artifact.ensure_artifact_gz(
|
||||
infra_artifact.infra_artifact_version(_infra_init()))
|
||||
|
||||
|
||||
def build_infra_images_with_docker() -> None:
|
||||
"""Build the three fixed images from source with host Docker: orchestrator,
|
||||
gateway, then the combined infra image (`COPY --from` orchestrator, `FROM`
|
||||
gateway). The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local`
|
||||
mode; `publish_infra` uses it off-host to produce the published artifact."""
|
||||
docker_mod.build_image(
|
||||
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
|
||||
docker_mod.build_image(
|
||||
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
|
||||
docker_mod.build_image(
|
||||
_INFRA_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.infra")
|
||||
|
||||
|
||||
def build_infra_rootfs_dir() -> Path:
|
||||
"""The infra VM's base rootfs: the infra image prepared with the
|
||||
control-plane + gateway init as PID 1. The init's content is folded into
|
||||
the cache key so an init change rebuilds the rootfs (the base image digest
|
||||
alone wouldn't catch it)."""
|
||||
init = _infra_init()
|
||||
tag = hashlib.sha256(init.encode()).hexdigest()[:8]
|
||||
return util.build_base_rootfs_dir(
|
||||
_INFRA_IMAGE, variant=f"-infra-{tag}", init_script=init,
|
||||
)
|
||||
|
||||
|
||||
def ensure_running() -> InfraVm:
|
||||
"""Idempotent per-host singleton. Adopt the infra VM if its control plane
|
||||
is already healthy (a prior launcher booted it — it outlives short-lived
|
||||
`start` processes); otherwise clear any stale VM and boot a fresh one.
|
||||
Returns a handle usable for CA fetch / git-gate provisioning.
|
||||
|
||||
Concurrency-safe: the cold stop/build/boot path is serialized by a host
|
||||
flock, so two simultaneous first launches don't both boot on the same
|
||||
rootfs/PID. The healthy fast-path takes no lock."""
|
||||
slot = netpool.orch_slot()
|
||||
url = f"http://{slot.guest_ip}:{CONTROL_PLANE_PORT}"
|
||||
key = _infra_dir() / "id_ed25519"
|
||||
if key.exists() and _health_ok(url):
|
||||
info(f"adopting running infra VM at {url}")
|
||||
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
|
||||
|
||||
with _singleton_lock():
|
||||
# Re-check under the lock: another launcher may have booted it while
|
||||
# we waited for the lock (double-checked, so we adopt not re-boot).
|
||||
if key.exists() and _health_ok(url):
|
||||
info(f"adopting running infra VM at {url}")
|
||||
return InfraVm(guest_ip=slot.guest_ip, private_key=key)
|
||||
stop() # clear a stale/hung VM holding the link before booting fresh
|
||||
ensure_built()
|
||||
infra = boot()
|
||||
wait_for_health(infra)
|
||||
return infra
|
||||
|
||||
|
||||
@contextmanager
|
||||
def _singleton_lock() -> Generator[None, None, None]:
|
||||
"""Host-level exclusive lock serializing the infra VM's cold create path
|
||||
(`stop`/`ensure_built`/`boot`). flock auto-releases if the launcher
|
||||
crashes, so the lock is never leaked."""
|
||||
lock_path = _infra_dir() / "singleton.lock"
|
||||
handle = open(lock_path, "w", encoding="utf-8")
|
||||
try:
|
||||
fcntl.flock(handle, fcntl.LOCK_EX)
|
||||
yield
|
||||
finally:
|
||||
handle.close()
|
||||
|
||||
|
||||
def stop() -> None:
|
||||
"""Stop the infra VM singleton (idempotent — absent is success)."""
|
||||
_kill_pidfile()
|
||||
_pid_file().unlink(missing_ok=True)
|
||||
|
||||
|
||||
def boot() -> InfraVm:
|
||||
"""Boot the infra VM (detached, so it outlives the launcher) on the
|
||||
orchestrator link, recording its PID. Prefer `ensure_running`."""
|
||||
slot = netpool.orch_slot()
|
||||
if not netpool.tap_present(slot.iface):
|
||||
die(f"orchestrator link {slot.iface} not present.\n"
|
||||
f" ./cli.py backend setup --backend=firecracker")
|
||||
|
||||
run_dir = _infra_dir()
|
||||
rootfs = run_dir / "rootfs.ext4"
|
||||
if infra_artifact.local_build_requested():
|
||||
util.build_rootfs_ext4(build_infra_rootfs_dir(), rootfs, slack_mib=8192)
|
||||
else:
|
||||
# Prebuilt artifact already carries the buildah build slack; expand it
|
||||
# to a fresh writable rootfs for this boot.
|
||||
infra_artifact.materialize_ext4(
|
||||
infra_artifact.infra_artifact_version(_infra_init()), rootfs)
|
||||
private_key, pubkey = _stable_keypair()
|
||||
|
||||
info(f"booting infra VM on {slot.iface} (guest {slot.guest_ip})")
|
||||
vm = firecracker_vm.boot(
|
||||
name="bot-bottle-infra", rootfs=rootfs, tap=slot.iface,
|
||||
guest_ip=slot.guest_ip, host_ip=slot.host_ip, pubkey=pubkey,
|
||||
run_dir=run_dir, mem_mib=4096, detached=True,
|
||||
data_drive=_ensure_registry_volume(),
|
||||
)
|
||||
_pid_file().write_text(str(vm.process.pid))
|
||||
return InfraVm(guest_ip=slot.guest_ip, private_key=private_key, vm=vm)
|
||||
|
||||
|
||||
def _infra_dir() -> Path:
|
||||
d = util.cache_dir() / "infra"
|
||||
d.mkdir(parents=True, exist_ok=True)
|
||||
return d
|
||||
|
||||
|
||||
def _pid_file() -> Path:
|
||||
return _infra_dir() / "vm.pid"
|
||||
|
||||
|
||||
# The registry "volume": a host-side ext4 file attached to the infra VM as a
|
||||
# second virtio-block device (guest /dev/vdb), mounted at the control plane's
|
||||
# DB dir. It outlives the ephemeral rootfs, so the bottle registry survives an
|
||||
# infra-VM restart — the firecracker analogue of a docker volume. It is a
|
||||
# plain ext4 file: `sudo mount -o loop <path>` on the host (with the VM
|
||||
# stopped) to inspect bot-bottle.db directly.
|
||||
_REGISTRY_SIZE = "512M"
|
||||
|
||||
|
||||
def registry_volume_path() -> Path:
|
||||
return _infra_dir() / "registry.ext4"
|
||||
|
||||
|
||||
def _ensure_registry_volume() -> Path:
|
||||
"""Create the empty ext4 registry volume on first use; reuse it after."""
|
||||
vol = registry_volume_path()
|
||||
if vol.exists():
|
||||
return vol
|
||||
info(f"creating infra registry volume {vol} ({_REGISTRY_SIZE})")
|
||||
proc = subprocess.run(
|
||||
["mke2fs", "-q", "-t", "ext4", "-F", str(vol), _REGISTRY_SIZE],
|
||||
capture_output=True, text=True, check=False,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
vol.unlink(missing_ok=True)
|
||||
die(f"creating registry volume failed: {proc.stderr.strip()}")
|
||||
return vol
|
||||
|
||||
|
||||
def _stable_keypair() -> tuple[Path, str]:
|
||||
"""The infra VM's SSH keypair — generated once and reused, so any later
|
||||
launcher can SSH in (fetch CA / provision) even though a different process
|
||||
booted the VM. The pubkey is re-injected on every boot via the cmdline."""
|
||||
d = _infra_dir()
|
||||
key, pub = d / "id_ed25519", d / "id_ed25519.pub"
|
||||
if key.exists() and pub.exists():
|
||||
return key, pub.read_text().strip()
|
||||
key.unlink(missing_ok=True)
|
||||
pub.unlink(missing_ok=True)
|
||||
subprocess.run(
|
||||
["ssh-keygen", "-t", "ed25519", "-N", "", "-q", "-f", str(key),
|
||||
"-C", "bot-bottle-infra"],
|
||||
check=True,
|
||||
)
|
||||
return key, pub.read_text().strip()
|
||||
|
||||
|
||||
def _kill_pidfile() -> None:
|
||||
"""SIGTERM (then SIGKILL) the recorded infra VMM, if it's still ours.
|
||||
Guards against a recycled PID by checking the process is firecracker."""
|
||||
try:
|
||||
pid = int(_pid_file().read_text().strip())
|
||||
except (OSError, ValueError):
|
||||
return
|
||||
try:
|
||||
comm = Path(f"/proc/{pid}/comm").read_text().strip()
|
||||
except OSError:
|
||||
return # already gone
|
||||
if comm != "firecracker":
|
||||
return # PID recycled by an unrelated process
|
||||
try:
|
||||
os.kill(pid, signal.SIGTERM)
|
||||
for _ in range(50):
|
||||
if not Path(f"/proc/{pid}").exists():
|
||||
return
|
||||
time.sleep(0.1)
|
||||
os.kill(pid, signal.SIGKILL)
|
||||
except OSError:
|
||||
pass
|
||||
|
||||
|
||||
def _health_ok(url: str) -> bool:
|
||||
try:
|
||||
with urllib.request.urlopen(f"{url}/health", timeout=1.0) as resp:
|
||||
return resp.status == 200
|
||||
except (urllib.error.URLError, TimeoutError, OSError):
|
||||
return False
|
||||
|
||||
|
||||
class SshGatewayTransport:
|
||||
"""`GatewayTransport` for the gateway running in the infra VM — the docker
|
||||
exec/cp equivalents over SSH (dropbear + the stable infra key)."""
|
||||
|
||||
def __init__(self, private_key: Path, guest_ip: str) -> None:
|
||||
self._key = private_key
|
||||
self._ip = guest_ip
|
||||
|
||||
def exec(self, argv: list[str]) -> None:
|
||||
proc = subprocess.run(
|
||||
util.ssh_base_argv(self._key, self._ip) + [shlex.join(argv)],
|
||||
capture_output=True, text=True, timeout=60, check=False,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"infra gateway exec {argv!r} failed: {proc.stderr.strip()}")
|
||||
|
||||
def cp_into(self, src: str, dest: str) -> None:
|
||||
# Preserve the source mode (docker cp does): the access-hook is staged
|
||||
# 0700 and git-http execs it directly — a plain `cat >` would land it
|
||||
# 0644 and the exec fails with EACCES; keys stay 0600.
|
||||
mode = stat.S_IMODE(os.stat(src).st_mode)
|
||||
q = shlex.quote(dest)
|
||||
proc = subprocess.run(
|
||||
util.ssh_base_argv(self._key, self._ip)
|
||||
+ [f"cat > {q} && chmod {mode:o} {q}"],
|
||||
input=Path(src).read_bytes(), capture_output=True, timeout=30, check=False,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"infra gateway cp {src} -> {dest} failed: "
|
||||
f"{proc.stderr.decode(errors='replace').strip()}")
|
||||
|
||||
|
||||
def gateway_transport() -> SshGatewayTransport:
|
||||
"""git-gate provisioning transport for the gateway in the infra VM, built
|
||||
from the stable key + the orchestrator link's guest IP. Needs no live VM
|
||||
handle, so teardown can use it too."""
|
||||
return SshGatewayTransport(
|
||||
_infra_dir() / "id_ed25519", netpool.orch_slot().guest_ip)
|
||||
|
||||
|
||||
def wait_for_health(
|
||||
infra: InfraVm, *, timeout: float = _HEALTH_TIMEOUT_SECONDS,
|
||||
) -> None:
|
||||
"""Poll the control plane's /health until it answers 200 or the deadline
|
||||
passes. Dies (with the console tail) if the VMM exits early."""
|
||||
url = f"{infra.control_plane_url}/health"
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if infra.vm is not None and not infra.vm.is_alive():
|
||||
die(f"infra VM exited during boot (rc={infra.vm.process.returncode}).\n"
|
||||
f"{firecracker_vm._console_tail(infra.vm.console_log)}")
|
||||
try:
|
||||
with urllib.request.urlopen(url, timeout=1.0) as resp:
|
||||
if resp.status == 200:
|
||||
info(f"infra control plane healthy at {infra.control_plane_url}")
|
||||
return
|
||||
except (urllib.error.URLError, TimeoutError, OSError):
|
||||
pass
|
||||
time.sleep(_HEALTH_POLL_SECONDS)
|
||||
tail = (firecracker_vm._console_tail(infra.vm.console_log)
|
||||
if infra.vm is not None else "")
|
||||
die(f"infra control plane at {url} did not become healthy within "
|
||||
f"{timeout:.0f}s.\n{tail}")
|
||||
|
||||
|
||||
def _infra_init() -> str:
|
||||
"""PID-1 init for the infra VM: mount the pseudo-filesystems, wire a
|
||||
resolver, start dropbear (debug SSH), then launch the control plane and
|
||||
the gateway data plane (multi-tenant against the local control plane)."""
|
||||
return f"""#!/bin/sh
|
||||
# bot-bottle Firecracker infra VM init (PID 1).
|
||||
mount -t proc proc /proc 2>/dev/null
|
||||
mount -t sysfs sys /sys 2>/dev/null
|
||||
mount -t devtmpfs dev /dev 2>/dev/null
|
||||
mkdir -p /dev/pts && mount -t devpts devpts /dev/pts 2>/dev/null
|
||||
mount -o remount,rw / 2>/dev/null
|
||||
|
||||
# Export a real PATH: a bare-init shell resolves its own execs via a
|
||||
# built-in default path, but that isn't in the *environment*, so
|
||||
# gateway_init's subprocess daemons (spawned as `python3 ...`) would
|
||||
# inherit no PATH and fail to find python3. Export it for all children.
|
||||
export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
# Direct upstream resolver (control-plane / gateway egress + buildah).
|
||||
printf 'nameserver {_INFRA_RESOLVER}\\n' > /etc/resolv.conf 2>/dev/null
|
||||
|
||||
# Debug SSH: install the per-boot pubkey from the kernel cmdline.
|
||||
KEY=$(sed -n 's/.*bb_pubkey=\\([^ ]*\\).*/\\1/p' /proc/cmdline | base64 -d 2>/dev/null)
|
||||
if [ -n "$KEY" ]; then
|
||||
mkdir -p /root/.ssh
|
||||
printf '%s\\n' "$KEY" > /root/.ssh/authorized_keys
|
||||
chmod 700 /root/.ssh && chmod 600 /root/.ssh/authorized_keys
|
||||
fi
|
||||
chown -R 0:0 /root 2>/dev/null || true
|
||||
mkdir -p /etc/dropbear /run /var/lib/bot-bottle
|
||||
|
||||
# Persistent registry volume (second virtio-block device, /dev/vdb) mounted
|
||||
# at the control plane's DB dir, so bot-bottle.db survives infra-VM restarts.
|
||||
mount -t ext4 /dev/vdb /var/lib/bot-bottle 2>/dev/null || true
|
||||
|
||||
/bb-dropbear -R -E -p 22 &
|
||||
|
||||
# Control plane. Source is baked at /app; the package is stdlib-only.
|
||||
cd /app
|
||||
BOT_BOTTLE_ROOT=/var/lib/bot-bottle python3 -m bot_bottle.orchestrator \\
|
||||
--host 0.0.0.0 --port {CONTROL_PLANE_PORT} --broker stub &
|
||||
|
||||
# Gateway data plane, multi-tenant: each request resolves source-IP ->
|
||||
# policy against the local control plane. The VM backend reaches git over
|
||||
# git-http (9420), so the git:// daemon (git-gate, needs a per-bottle
|
||||
# entrypoint the consolidated model doesn't use) is left out.
|
||||
BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise \\
|
||||
BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{CONTROL_PLANE_PORT} \\
|
||||
SUPERVISE_DB_PATH=/var/lib/bot-bottle/db/bot-bottle.db \\
|
||||
python3 /app/gateway_init.py &
|
||||
|
||||
# Reap as PID 1; children are backgrounded, so `wait` blocks.
|
||||
while : ; do wait ; done
|
||||
"""
|
||||
@@ -1,8 +1,7 @@
|
||||
"""Launch flow for the Firecracker backend (PRD 0070, consolidated).
|
||||
|
||||
Per bottle:
|
||||
1. build the agent rootfs in a builder VM (buildah, no host docker), or
|
||||
resume a frozen bottle from its committed rootfs tar; cache the ext4;
|
||||
1. build the agent image (docker), export it to a cached ext4 rootfs;
|
||||
2. ensure the per-host orchestrator + shared gateway are up;
|
||||
3. claim a free TAP pool slot (rootless flock);
|
||||
4. register the bottle on the orchestrator by the VM's guest IP (the
|
||||
@@ -32,7 +31,6 @@ from typing import Callable, Generator
|
||||
|
||||
from ...agent_provider import runtime_for
|
||||
from ...bottle_state import (
|
||||
committed_rootfs_path,
|
||||
egress_state_dir,
|
||||
git_gate_state_dir,
|
||||
read_committed_image,
|
||||
@@ -47,9 +45,10 @@ from ...git_gate import (
|
||||
)
|
||||
from ...log import info, warn
|
||||
from ...supervise import SUPERVISE_PORT
|
||||
from ..docker import util as docker_mod
|
||||
from ..docker.egress import EGRESS_PORT
|
||||
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
|
||||
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
|
||||
from . import firecracker_vm, isolation_probe, netpool, util
|
||||
from .bottle import FirecrackerBottle
|
||||
from .bottle_plan import FirecrackerBottlePlan
|
||||
from .consolidated_launch import (
|
||||
@@ -58,6 +57,7 @@ from .consolidated_launch import (
|
||||
)
|
||||
|
||||
|
||||
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
|
||||
_GIT_HTTP_PORT = 9420
|
||||
|
||||
|
||||
@@ -85,10 +85,10 @@ def launch(
|
||||
raise teardown_exc
|
||||
|
||||
try:
|
||||
# Step 1: agent rootfs. Built from the Dockerfile inside a Firecracker
|
||||
# builder VM (buildah, no host docker); a committed snapshot is reused
|
||||
# when present. Returns the base dir the per-bottle ext4 is made from.
|
||||
plan, agent_base = _build_agent_base(plan)
|
||||
# Step 1: agent image. The sidecar bundle image is built by the
|
||||
# orchestrator service (ensure_running → ensure_built); we only
|
||||
# build the agent image here. Use a committed snapshot when available.
|
||||
plan = _build_agent_image(plan)
|
||||
|
||||
# Step 2: mint the git-gate dynamic (gitea) deploy keys, if any.
|
||||
git_gate_plan = plan.git_gate_plan
|
||||
@@ -148,24 +148,17 @@ def launch(
|
||||
plan,
|
||||
git_gate_plan=git_gate_plan,
|
||||
egress_plan=egress_plan,
|
||||
identity_token=ctx.identity_token,
|
||||
# Deliver the identity token as egress proxy credentials — clients
|
||||
# honor `HTTPS_PROXY=http://id:token@gw` without app changes; the
|
||||
# gateway reads Proxy-Authorization, validates the (source_ip,
|
||||
# token) pair, and strips it before upstream.
|
||||
agent_proxy_url=(
|
||||
f"http://bottle:{ctx.identity_token}"
|
||||
f"@{slot.host_ip}:{EGRESS_PORT}"
|
||||
),
|
||||
agent_proxy_url=f"http://{slot.host_ip}:{EGRESS_PORT}",
|
||||
agent_git_gate_url=git_gate_url,
|
||||
agent_supervise_url=supervise_url,
|
||||
)
|
||||
|
||||
# Step 6: build the per-bottle rootfs + SSH key, then boot.
|
||||
base_dir = util.build_base_rootfs_dir(plan.image)
|
||||
run_dir = util.cache_dir() / "run" / plan.slug
|
||||
run_dir.mkdir(parents=True, exist_ok=True)
|
||||
rootfs = run_dir / "rootfs.ext4"
|
||||
util.build_rootfs_ext4(agent_base, rootfs)
|
||||
util.build_rootfs_ext4(base_dir, rootfs)
|
||||
private_key, pubkey = util.generate_keypair(run_dir)
|
||||
|
||||
vm = firecracker_vm.boot(
|
||||
@@ -206,24 +199,19 @@ def launch(
|
||||
teardown()
|
||||
|
||||
|
||||
def _build_agent_base(
|
||||
plan: FirecrackerBottlePlan,
|
||||
) -> tuple[FirecrackerBottlePlan, Path]:
|
||||
"""Produce the agent's base rootfs dir. Primary path: build the Dockerfile
|
||||
inside a Firecracker builder VM (buildah, no host docker), smoke-testing
|
||||
the image before export. A committed snapshot (freeze/migrate) is resumed
|
||||
directly from the rootfs tar the freezer wrote — no host docker either."""
|
||||
def _build_agent_image(plan: FirecrackerBottlePlan) -> FirecrackerBottlePlan:
|
||||
committed = read_committed_image(plan.slug)
|
||||
committed_tar = committed_rootfs_path(plan.slug)
|
||||
if committed and committed_tar.is_file():
|
||||
info(f"resuming from committed rootfs {committed_tar}")
|
||||
return plan, util.build_committed_rootfs_dir(committed_tar)
|
||||
base = image_builder.build_agent_rootfs_dir(
|
||||
Path(plan.dockerfile_path),
|
||||
image_tag=plan.image,
|
||||
smoke_test=runtime_for(plan.agent_provider_template).smoke_test,
|
||||
if committed and docker_mod.image_exists(committed):
|
||||
info(f"using committed image {committed!r}")
|
||||
return dataclasses.replace(
|
||||
plan,
|
||||
agent_provision=dataclasses.replace(plan.agent_provision, image=committed),
|
||||
)
|
||||
docker_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
|
||||
docker_mod.verify_agent_image(
|
||||
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
|
||||
)
|
||||
return plan, base
|
||||
return plan
|
||||
|
||||
|
||||
# --- agent guest env -------------------------------------------------
|
||||
@@ -232,8 +220,7 @@ def _agent_guest_env(plan: FirecrackerBottlePlan, host_ip: str) -> dict[str, str
|
||||
"""Env injected into every agent/exec call over SSH. The VM has no
|
||||
baked process env (it just runs init), so the proxy/CA/git/supervise
|
||||
wiring is applied per-invocation."""
|
||||
# Carries the identity token as proxy credentials (set in `launch`).
|
||||
proxy_url = plan.agent_proxy_url or f"http://{host_ip}:{EGRESS_PORT}"
|
||||
proxy_url = f"http://{host_ip}:{EGRESS_PORT}"
|
||||
no_proxy = f"localhost,127.0.0.1,{host_ip}"
|
||||
env: dict[str, str] = {
|
||||
"HTTPS_PROXY": proxy_url, "HTTP_PROXY": proxy_url,
|
||||
|
||||
@@ -15,11 +15,3 @@ BOT_BOTTLE_FC_POOL_SIZE=8
|
||||
BOT_BOTTLE_FC_IP_BASE=10.243.0.0
|
||||
BOT_BOTTLE_FC_IFACE_PREFIX=bbfc
|
||||
BOT_BOTTLE_FC_NFT_TABLE=bot_bottle_fc
|
||||
# The orchestrator/gateway VM's own TAP — a dedicated link OUTSIDE the
|
||||
# bbfc* agent pool. Unlike agent VMs (which reach only their gateway),
|
||||
# the orchestrator is trusted infra that needs real NAT'd internet
|
||||
# egress: to FROM-pull + apt/npm during in-VM agent-image builds
|
||||
# (buildah) and to forward agent egress upstream (Stage B gateway). Its
|
||||
# /31 is the top of the IP_BASE /16 (host x.y.255.0, guest x.y.255.1),
|
||||
# clear of the pool near the bottom of the block.
|
||||
BOT_BOTTLE_FC_ORCH_IFACE=bborch0
|
||||
|
||||
@@ -79,12 +79,6 @@ def _cfg(key: str) -> str:
|
||||
IFACE_PREFIX = _cfg("BOT_BOTTLE_FC_IFACE_PREFIX")
|
||||
NFT_TABLE = _cfg("BOT_BOTTLE_FC_NFT_TABLE")
|
||||
|
||||
# The orchestrator/gateway VM's dedicated TAP — outside the bbfc* agent
|
||||
# pool and, unlike it, NAT'd to the internet (see `orch_slot`). The
|
||||
# orchestrator is trusted infra: it builds agent images in-VM (buildah
|
||||
# needs to FROM-pull + apt/npm) and forwards agent egress upstream.
|
||||
ORCH_IFACE = _cfg("BOT_BOTTLE_FC_ORCH_IFACE")
|
||||
|
||||
|
||||
def pool_size() -> int:
|
||||
return int(_cfg("BOT_BOTTLE_FC_POOL_SIZE"))
|
||||
@@ -129,25 +123,6 @@ def all_slots() -> list[Slot]:
|
||||
return [slot(i) for i in range(pool_size())]
|
||||
|
||||
|
||||
def orch_slot() -> Slot:
|
||||
"""The orchestrator/gateway VM's dedicated link — its own TAP
|
||||
(`ORCH_IFACE`) on a /31 at the TOP of the IP_BASE /16 (host
|
||||
x.y.255.0, guest x.y.255.1), well clear of the agent pool near the
|
||||
bottom of the block. Unlike a pool `Slot`, this link is NAT'd out to
|
||||
the internet by the setup (the orchestrator is trusted infra), so it
|
||||
is deliberately *not* one of the isolated `bbfc*` slots.
|
||||
|
||||
`index` is -1 (sentinel: not a pool index)."""
|
||||
base16 = int(ipaddress.IPv4Address(ip_base())) & 0xFFFF0000
|
||||
host = base16 + 0xFF00
|
||||
return Slot(
|
||||
index=-1,
|
||||
iface=ORCH_IFACE,
|
||||
host_ip=str(ipaddress.IPv4Address(host)),
|
||||
guest_ip=str(ipaddress.IPv4Address(host + 1)),
|
||||
)
|
||||
|
||||
|
||||
# --- fail-closed verification ---------------------------------------
|
||||
|
||||
def _run_ok(argv: list[str]) -> bool:
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
"""Build the infra rootfs and publish it as a Gitea generic package.
|
||||
|
||||
The off-host (build / CI) half of PRD 0069 Stage 2: this DOES use Docker, but
|
||||
never on the launch host. It runs the same pipeline the launch host used to run
|
||||
locally — `docker build` the three fixed images, export to a rootfs dir, inject
|
||||
the guest boot, `mke2fs` to an ext4 with the buildah build slack — then gzips
|
||||
the ext4 and PUTs it (plus a `.sha256`) to
|
||||
`…/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<version>/`.
|
||||
|
||||
The `<version>` is `infra_artifact.infra_artifact_version(...)`, the content
|
||||
hash of the rootfs inputs, so a launch host at the same code checkout resolves
|
||||
the exact artifact this produced.
|
||||
|
||||
python3 -m bot_bottle.backend.firecracker.publish_infra [--dry-run] [--force]
|
||||
|
||||
Auth: a token with `write:package` on the target owner, from
|
||||
`BOT_BOTTLE_INFRA_ARTIFACT_TOKEN`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gzip
|
||||
import hashlib
|
||||
import shutil
|
||||
import sys
|
||||
import tempfile
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
from . import infra_artifact, infra_vm, util
|
||||
|
||||
_CHUNK = 1 << 20
|
||||
|
||||
# A human-readable description shipped alongside the artifact — generic packages
|
||||
# have no description field, so this file *is* the description on the package
|
||||
# page. Uploaded on every publish so it never goes stale.
|
||||
_ABOUT_NAME = "about.txt"
|
||||
_ABOUT_TEXT = (
|
||||
"bot-bottle infra rootfs for the Firecracker backend (PRD 0069 Stage 2, "
|
||||
"#348): the per-host infra VM (orchestrator control plane + gateway + "
|
||||
"buildah). Prebuilt off-host, gzip ext4; the launch host downloads + "
|
||||
"sha256-verifies + boots it, no host Docker. The version tag is a content "
|
||||
"hash of the rootfs inputs. Files: rootfs.ext4.gz + rootfs.ext4.gz.sha256.\n"
|
||||
)
|
||||
|
||||
|
||||
def _gzip(src: Path, dest: Path) -> None:
|
||||
with open(src, "rb") as fh, gzip.open(dest, "wb") as out:
|
||||
shutil.copyfileobj(fh, out, _CHUNK)
|
||||
|
||||
|
||||
def _sha256(path: Path) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as fh:
|
||||
for chunk in iter(lambda: fh.read(_CHUNK), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def _put(url: str, body: "bytes | Path", token: str) -> None:
|
||||
"""PUT `body` (raw bytes, or a Path streamed from disk) to `url`. The rootfs
|
||||
is hundreds of MB, so it is passed as a Path and streamed — `urlopen` reads
|
||||
the open file in blocks rather than materializing it in memory (with an
|
||||
explicit Content-Length, which Gitea requires and which also stops urllib
|
||||
from `len()`-ing a non-bytes body)."""
|
||||
handle = None
|
||||
if isinstance(body, Path):
|
||||
length = body.stat().st_size
|
||||
handle = open(body, "rb")
|
||||
data: object = handle
|
||||
else:
|
||||
length = len(body)
|
||||
data = body
|
||||
req = urllib.request.Request(url, data=data, method="PUT") # type: ignore[arg-type]
|
||||
req.add_header("Content-Length", str(length))
|
||||
if token:
|
||||
req.add_header("Authorization", f"token {token}")
|
||||
req.add_header("Content-Type", "application/octet-stream")
|
||||
try:
|
||||
with urllib.request.urlopen(req) as resp:
|
||||
print(f" uploaded {url} (HTTP {resp.status})")
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 409:
|
||||
raise SystemExit(
|
||||
f"artifact already published at {url} (HTTP 409); "
|
||||
f"bump the code version or pass --force to overwrite"
|
||||
)
|
||||
raise SystemExit(f"upload failed (HTTP {e.code}): {url}\n{e.read().decode(errors='replace')}")
|
||||
except urllib.error.URLError as e:
|
||||
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
|
||||
finally:
|
||||
if handle is not None:
|
||||
handle.close()
|
||||
|
||||
|
||||
def _delete(url: str, token: str) -> None:
|
||||
req = urllib.request.Request(url, method="DELETE")
|
||||
if token:
|
||||
req.add_header("Authorization", f"token {token}")
|
||||
try:
|
||||
with urllib.request.urlopen(req):
|
||||
pass
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code != 404:
|
||||
raise SystemExit(f"could not overwrite existing artifact (HTTP {e.code}): {url}")
|
||||
except urllib.error.URLError as e:
|
||||
raise SystemExit(f"registry unreachable: {url} ({e.reason})")
|
||||
|
||||
|
||||
def build_artifact(out_dir: Path) -> tuple[str, Path, Path]:
|
||||
"""Build the infra rootfs ext4, gzip it, and write the checksum. Returns
|
||||
`(version, gz_path, sha_path)`. Uses host Docker (off-host / CI)."""
|
||||
version = infra_artifact.infra_artifact_version(infra_vm._infra_init())
|
||||
print(f"building infra rootfs artifact {version} (docker)")
|
||||
infra_vm.build_infra_images_with_docker()
|
||||
base = infra_vm.build_infra_rootfs_dir()
|
||||
|
||||
ext4 = out_dir / "rootfs.ext4"
|
||||
util.build_rootfs_ext4(base, ext4, slack_mib=8192)
|
||||
gz = out_dir / "rootfs.ext4.gz"
|
||||
print("compressing rootfs")
|
||||
_gzip(ext4, gz)
|
||||
ext4.unlink(missing_ok=True)
|
||||
|
||||
sha = out_dir / "rootfs.ext4.gz.sha256"
|
||||
digest = _sha256(gz)
|
||||
sha.write_text(f"{digest} rootfs.ext4.gz\n")
|
||||
print(f" {gz.name}: {gz.stat().st_size / 1e6:.0f} MB sha256={digest}")
|
||||
return version, gz, sha
|
||||
|
||||
|
||||
def main(argv: list[str] | None = None) -> int:
|
||||
parser = argparse.ArgumentParser(
|
||||
prog="publish_infra", description="Build + publish the infra rootfs artifact.")
|
||||
parser.add_argument("--dry-run", action="store_true",
|
||||
help="build the artifact but do not upload")
|
||||
parser.add_argument("--force", action="store_true",
|
||||
help="overwrite an already-published artifact of this version")
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
_, _, token = infra_artifact._config()
|
||||
if not args.dry_run and not token:
|
||||
raise SystemExit(
|
||||
"no publish token: set BOT_BOTTLE_INFRA_ARTIFACT_TOKEN to a token "
|
||||
"with write:package")
|
||||
|
||||
with tempfile.TemporaryDirectory(prefix="bb-publish-infra.") as tmp:
|
||||
version, gz, sha = build_artifact(Path(tmp))
|
||||
gz_url = infra_artifact.artifact_url(version, gz.name)
|
||||
sha_url = infra_artifact.artifact_url(version, sha.name)
|
||||
about_url = infra_artifact.artifact_url(version, _ABOUT_NAME)
|
||||
if args.dry_run:
|
||||
print(f"dry-run: would upload -> {gz_url}")
|
||||
return 0
|
||||
if args.force:
|
||||
_delete(gz_url, token)
|
||||
_delete(sha_url, token)
|
||||
_delete(about_url, token)
|
||||
_put(gz_url, gz, token) # streamed from disk (hundreds of MB)
|
||||
_put(sha_url, sha.read_bytes(), token) # tiny, in-memory is fine
|
||||
_put(about_url, _ABOUT_TEXT.encode(), token) # package description
|
||||
print(f"published infra rootfs {version}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
@@ -14,7 +14,6 @@ and `./cli.py backend setup --backend=firecracker`.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
@@ -160,22 +159,15 @@ def docker_image_id(ref: str) -> str:
|
||||
return result.stdout.strip().replace("sha256:", "")[:16]
|
||||
|
||||
|
||||
def build_base_rootfs_dir(
|
||||
image_ref: str, *, variant: str = "", init_script: str | None = None,
|
||||
) -> Path:
|
||||
"""Export the image's filesystem and inject the guest init + static
|
||||
dropbear. Cached by image digest — the per-bottle bits
|
||||
def build_base_rootfs_dir(image_ref: str) -> Path:
|
||||
"""Export the agent image's filesystem and inject the guest init +
|
||||
static dropbear. Cached by image digest — the per-bottle bits
|
||||
(authorized_keys, IP) are passed at boot via the kernel cmdline, so
|
||||
this tree carries nothing bottle-specific and is safely shared.
|
||||
|
||||
`variant` suffixes the cache key so the same image can be prepared
|
||||
with a different `init_script` (e.g. the infra VM boots the same
|
||||
orchestrator image as the builder but runs the control plane as
|
||||
PID 1, not the SSH-only agent init) without a cache collision.
|
||||
|
||||
Returns the prepared directory (read as the `mke2fs -d` source)."""
|
||||
digest = docker_image_id(image_ref)
|
||||
base = cache_dir() / "rootfs" / f"{digest}{variant}"
|
||||
base = cache_dir() / "rootfs" / digest
|
||||
ready = base / ".bb-ready"
|
||||
if ready.is_file():
|
||||
return base
|
||||
@@ -208,85 +200,18 @@ def build_base_rootfs_dir(
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
|
||||
)
|
||||
|
||||
inject_guest_boot(base, init_script=init_script)
|
||||
_inject_guest_boot(base)
|
||||
ready.write_text("ok\n")
|
||||
return base
|
||||
|
||||
|
||||
def build_committed_rootfs_dir(tar_path: Path) -> Path:
|
||||
"""Prepare a base rootfs dir from a frozen-bottle snapshot tar (the
|
||||
freeze/resume path — no Docker). Extracts the snapshot, recreates the
|
||||
virtual mount points the freezer excluded, and injects the guest init +
|
||||
static dropbear, mirroring `build_base_rootfs_dir` but sourced from a tar
|
||||
we control rather than a Docker image.
|
||||
|
||||
Cached under the rootfs cache, keyed by the tar's size+mtime so a
|
||||
re-freeze re-extracts but repeated resumes of the same snapshot don't.
|
||||
Returns the prepared directory (read as the `mke2fs -d` source)."""
|
||||
st = tar_path.stat()
|
||||
fingerprint = hashlib.sha256(
|
||||
f"{tar_path}:{st.st_size}:{st.st_mtime_ns}".encode()
|
||||
).hexdigest()[:16]
|
||||
base = cache_dir() / "rootfs" / f"committed-{fingerprint}"
|
||||
ready = base / ".bb-ready"
|
||||
if ready.is_file():
|
||||
return base
|
||||
|
||||
if base.exists():
|
||||
shutil.rmtree(base, ignore_errors=True)
|
||||
base.mkdir(parents=True)
|
||||
|
||||
info(f"extracting committed rootfs {tar_path} -> {base}")
|
||||
result = subprocess.run(
|
||||
["tar", "-x", "-f", str(tar_path), "-C", str(base)],
|
||||
capture_output=True, text=True, check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
die(f"extracting committed rootfs {tar_path} failed: "
|
||||
f"{result.stderr.strip() or '<no stderr>'}")
|
||||
|
||||
# The freezer excludes the live/virtual filesystems from the snapshot;
|
||||
# recreate them as empty mount points so the guest init can mount
|
||||
# proc/sys/dev and dropbear has a writable /run.
|
||||
for mount_point in ("proc", "sys", "dev", "run"):
|
||||
(base / mount_point).mkdir(mode=0o755, exist_ok=True)
|
||||
|
||||
inject_guest_boot(base)
|
||||
ready.write_text("ok\n")
|
||||
return base
|
||||
|
||||
|
||||
def inject_guest_boot(rootfs: Path, init_script: str | None = None) -> None:
|
||||
"""Drop the static dropbear and the PID-1 init into the rootfs.
|
||||
`init_script` defaults to the SSH-only agent init; the infra VM
|
||||
passes its own (control plane + gateway) init.
|
||||
|
||||
A committed snapshot is guest-controlled, so `bb-dropbear`/`bb-init`
|
||||
may already exist as symlinks aimed at a host file (e.g. bb-init ->
|
||||
~/.bashrc). Replace whatever is there and create the files with
|
||||
O_EXCL|O_NOFOLLOW so the write always lands a fresh regular file in
|
||||
the staging tree and never follows a planted symlink out of it."""
|
||||
_write_staged_file(rootfs / "bb-dropbear", dropbear_path().read_bytes())
|
||||
_write_staged_file(rootfs / "bb-init", (init_script or _GUEST_INIT).encode())
|
||||
|
||||
|
||||
def _write_staged_file(path: Path, data: bytes) -> None:
|
||||
"""Write `data` to `path` (mode 0755) as a fresh regular file inside a
|
||||
staging rootfs, replacing any pre-existing entry without following a
|
||||
symlink at `path`. Fails closed on anything unexpected there."""
|
||||
if path.is_symlink() or path.exists():
|
||||
if path.is_dir() and not path.is_symlink():
|
||||
shutil.rmtree(path)
|
||||
else:
|
||||
path.unlink()
|
||||
fd = os.open(
|
||||
path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o755
|
||||
)
|
||||
try:
|
||||
os.write(fd, data)
|
||||
finally:
|
||||
os.close(fd)
|
||||
os.chmod(path, 0o755)
|
||||
def _inject_guest_boot(rootfs: Path) -> None:
|
||||
"""Drop the static dropbear and the PID-1 init into the rootfs."""
|
||||
shutil.copy2(dropbear_path(), rootfs / "bb-dropbear")
|
||||
os.chmod(rootfs / "bb-dropbear", 0o755)
|
||||
init = rootfs / "bb-init"
|
||||
init.write_text(_GUEST_INIT)
|
||||
os.chmod(init, 0o755)
|
||||
|
||||
|
||||
def build_rootfs_ext4(base_dir: Path, out_path: Path, *, slack_mib: int = 1024) -> None:
|
||||
|
||||
@@ -89,14 +89,6 @@ class MacosContainerBottleBackend(
|
||||
with _launch.launch(plan, provision=self.provision) as bottle:
|
||||
yield bottle
|
||||
|
||||
def ensure_orchestrator(self) -> str:
|
||||
"""Bring up the per-host infra container (control plane + gateway) and
|
||||
return its control-plane URL — the on-demand entry point operator tools
|
||||
(`supervise`) call when no control plane is running yet. Mirrors
|
||||
firecracker's infra-VM bring-up."""
|
||||
from .infra import MacosInfraService
|
||||
return MacosInfraService().ensure_running().control_plane_url
|
||||
|
||||
def prepare_cleanup(self) -> MacosContainerBottleCleanupPlan:
|
||||
return _cleanup.prepare_cleanup()
|
||||
|
||||
|
||||
@@ -52,7 +52,6 @@ class MacosContainerBottle(Bottle):
|
||||
terminal_title: str = "",
|
||||
terminal_color: str = "",
|
||||
agent_workdir: str = "/home/node",
|
||||
exec_env: dict[str, str] | None = None,
|
||||
):
|
||||
self.name = container
|
||||
self._teardown = teardown
|
||||
@@ -63,15 +62,6 @@ class MacosContainerBottle(Bottle):
|
||||
self.terminal_color = terminal_color
|
||||
self.agent_provider_template = agent_provider_template
|
||||
self.agent_workdir = agent_workdir
|
||||
# Env applied to the agent process at `container exec` time, on top of
|
||||
# what the container was run with. This is how the identity token
|
||||
# reaches the agent (PRD 0070): registration mints it *after* the
|
||||
# container exists — its source IP is the registration key and Apple
|
||||
# Container assigns that by DHCP — so it cannot be in the run-time env
|
||||
# the way docker's compose spec does it. `container exec --env` wins
|
||||
# over the run-time value, so the token-bearing proxy URL set here
|
||||
# supersedes the token-less one baked in at launch.
|
||||
self._exec_env = dict(exec_env or {})
|
||||
self._closed = False
|
||||
|
||||
def agent_argv(self, argv: list[str], *, tty: bool = True) -> list[str]:
|
||||
@@ -84,12 +74,6 @@ class MacosContainerBottle(Bottle):
|
||||
)
|
||||
)
|
||||
container_exec = ["container", "exec"]
|
||||
# Bare env names, same rule as the terminal hints below: the value
|
||||
# stays in the child env `exec_agent` builds and never reaches argv —
|
||||
# the proxy URL here carries the identity token, which `ps` would
|
||||
# otherwise expose to every process on the host.
|
||||
for name in sorted(self._exec_env):
|
||||
container_exec.extend(["--env", name])
|
||||
if tty:
|
||||
container_exec.extend(["--interactive", "--tty"])
|
||||
# Forward terminal capability hints so TUIs can enable modified-key
|
||||
@@ -110,33 +94,21 @@ class MacosContainerBottle(Bottle):
|
||||
|
||||
def exec_agent(self, argv: list[str], *, tty: bool = True) -> int:
|
||||
agent_argv = self.agent_argv(argv, tty=tty)
|
||||
# The values behind the bare `--env` names in `agent_argv`. `sh -lc`
|
||||
# below is in this process tree, so the child env reaches `container
|
||||
# exec` either way.
|
||||
env = {**os.environ, **self._exec_env} if self._exec_env else None
|
||||
script = (
|
||||
exec_shell_script(agent_argv, self.terminal_title, self.terminal_color)
|
||||
if tty else None
|
||||
)
|
||||
if script is None:
|
||||
return subprocess.run(agent_argv, env=env, check=False).returncode
|
||||
return subprocess.run(["sh", "-lc", script], env=env, check=False).returncode
|
||||
return subprocess.run(agent_argv, check=False).returncode
|
||||
return subprocess.run(["sh", "-lc", script], check=False).returncode
|
||||
|
||||
def exec(self, script: str, *, user: str = "node") -> ExecResult:
|
||||
# Carry the same exec env the agent gets: provisioning steps run
|
||||
# through here, and a provider whose provision step fetches anything
|
||||
# would egress without the identity token and be denied by /resolve.
|
||||
# Bare `--env NAME` again, so the token stays off argv.
|
||||
argv = ["container", "exec", "--user", user, "--interactive"]
|
||||
for name in sorted(self._exec_env):
|
||||
argv.extend(["--env", name])
|
||||
argv.extend([self.name, "sh", "-s"])
|
||||
result = subprocess.run(
|
||||
argv,
|
||||
["container", "exec", "--user", user, "--interactive",
|
||||
self.name, "sh", "-s"],
|
||||
input=script,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
env={**os.environ, **self._exec_env} if self._exec_env else None,
|
||||
check=False,
|
||||
)
|
||||
return ExecResult(
|
||||
|
||||
@@ -13,13 +13,9 @@ from .. import BottlePlan
|
||||
class MacosContainerBottlePlan(BottlePlan):
|
||||
slug: str
|
||||
forwarded_env: dict[str, str] = field(repr=False)
|
||||
agent_proxy_url: str = ""
|
||||
agent_git_gate_url: str = ""
|
||||
agent_supervise_url: str = ""
|
||||
# Read by provision-time consumers (git extraHeader, supervise MCP header)
|
||||
# via getattr(plan, "identity_token", ""); stamped in launch after the
|
||||
# bottle is registered. See launch.py's stamp for why it lives here and not
|
||||
# only in the exec-time proxy env.
|
||||
identity_token: str = ""
|
||||
|
||||
@property
|
||||
def container_name(self) -> str:
|
||||
|
||||
@@ -1,144 +0,0 @@
|
||||
"""Consolidated bottle launch sequence for the macOS backend (PRD 0070).
|
||||
|
||||
The docker backend allocates a free address, pins the agent to it with
|
||||
`--ip`, registers it, *then* starts the agent — registration precedes launch
|
||||
because the pinned address is known up front.
|
||||
|
||||
**Apple Container 1.0.0 has no `--ip`.** The `--network` flag takes only
|
||||
`<name>[,mac=…][,mtu=…]`; the address is assigned by vmnet's DHCP and is
|
||||
knowable only once the container is running. So the macOS order inverts:
|
||||
|
||||
ensure_gateway() -> caller starts the agent -> register_agent(source_ip)
|
||||
|
||||
That is why this module exposes two functions where docker has one — the
|
||||
caller has to start the agent in between. `ensure_gateway` runs first because
|
||||
the agent's proxy env needs the gateway's address at `container run` time; the
|
||||
agent's *own* address (the attribution key) only exists afterwards.
|
||||
|
||||
The control plane and the gateway are one **infra container** here (see
|
||||
`infra`), so `gateway_ip` and the control-plane host are the same address.
|
||||
|
||||
The consequence for the identity token: it is minted by registration, i.e.
|
||||
*after* the agent container exists, so it cannot be baked into the run-time
|
||||
env the way docker's compose spec does. It is delivered at `container exec`
|
||||
time instead — see `bottle.MacosContainerBottle`.
|
||||
|
||||
That delivery is load-bearing, not a nicety: `/resolve` requires a matching
|
||||
`(source_ip, identity_token)` pair and fail-closes with no source-IP-only
|
||||
fallback (#366). So egress that does not carry the token is denied — which is
|
||||
the safe direction, and is why the agent's init process is a bare `sleep` and
|
||||
every real command arrives through `container exec`.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
from ...egress import EgressPlan
|
||||
from ...git_gate import GitGatePlan
|
||||
from ...orchestrator.client import OrchestratorClient
|
||||
from ...orchestrator.registration import registration_inputs
|
||||
from ..docker.gateway_provision import deprovision_git_gate, provision_git_gate
|
||||
from .gateway import GATEWAY_NETWORK
|
||||
from .gateway_provision import AppleGatewayTransport
|
||||
from .infra import MacosInfraService, OrchestratorStartError
|
||||
|
||||
|
||||
class ConsolidatedLaunchError(RuntimeError):
|
||||
"""The consolidated register/provision sequence could not complete."""
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GatewayEndpoint:
|
||||
"""What the agent `container run` needs to reach the shared gateway (the
|
||||
infra container). `gateway_ip` is that container's host-only address, the
|
||||
same host the control-plane URL points at."""
|
||||
|
||||
orchestrator_url: str
|
||||
gateway_ip: str # the gateway's address — the agent's proxy target
|
||||
gateway_ca_pem: str # the shared CA the provisioner installs
|
||||
network: str # the shared host-only network to attach to
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class LaunchContext:
|
||||
"""What the running agent needs once it has been registered."""
|
||||
|
||||
bottle_id: str
|
||||
identity_token: str
|
||||
source_ip: str # the agent's DHCP-assigned address (attribution key)
|
||||
gateway_ip: str
|
||||
network: str
|
||||
orchestrator_url: str
|
||||
|
||||
|
||||
def ensure_gateway(
|
||||
*, service: MacosInfraService | None = None,
|
||||
) -> GatewayEndpoint:
|
||||
"""Ensure the per-host infra container (control plane + gateway) is up and
|
||||
report how to reach it. Idempotent — one singleton, so N bottle launches
|
||||
share it. Call before starting the agent container: the agent's proxy env
|
||||
needs `gateway_ip` at run time."""
|
||||
service = service or MacosInfraService()
|
||||
infra = service.ensure_running()
|
||||
return GatewayEndpoint(
|
||||
orchestrator_url=infra.control_plane_url,
|
||||
gateway_ip=infra.gateway_ip,
|
||||
gateway_ca_pem=service.ca_cert_pem(),
|
||||
network=service.network,
|
||||
)
|
||||
|
||||
|
||||
def register_agent(
|
||||
egress_plan: EgressPlan,
|
||||
git_gate_plan: GitGatePlan,
|
||||
*,
|
||||
source_ip: str,
|
||||
endpoint: GatewayEndpoint,
|
||||
image_ref: str = "",
|
||||
tokens: dict[str, str] | None = None,
|
||||
) -> LaunchContext:
|
||||
"""Register the (already running) agent by its address and provision its
|
||||
git-gate state into the gateway. `source_ip` must be read from the live
|
||||
container — it is the attribution key the gateway resolves policy by.
|
||||
Raises on failure; the caller tears down."""
|
||||
client = OrchestratorClient(endpoint.orchestrator_url)
|
||||
inputs = registration_inputs(egress_plan)
|
||||
reg = client.register_bottle(
|
||||
source_ip, image_ref=image_ref, policy=inputs.policy,
|
||||
metadata=inputs.metadata, tokens=tokens,
|
||||
)
|
||||
try:
|
||||
provision_git_gate(AppleGatewayTransport(), reg.bottle_id, git_gate_plan)
|
||||
except Exception:
|
||||
# Roll the registration back so a provisioning failure leaves no orphan.
|
||||
client.teardown_bottle(reg.bottle_id)
|
||||
raise
|
||||
return LaunchContext(
|
||||
bottle_id=reg.bottle_id,
|
||||
identity_token=reg.identity_token,
|
||||
source_ip=source_ip,
|
||||
gateway_ip=endpoint.gateway_ip,
|
||||
network=endpoint.network,
|
||||
orchestrator_url=endpoint.orchestrator_url,
|
||||
)
|
||||
|
||||
|
||||
def teardown_consolidated(bottle_id: str, *, orchestrator_url: str) -> None:
|
||||
"""Deregister the bottle and remove its git-gate state from the gateway.
|
||||
Both steps are idempotent so this is safe from a cleanup trap. Does NOT
|
||||
stop the gateway — it's a persistent per-host singleton."""
|
||||
OrchestratorClient(orchestrator_url).teardown_bottle(bottle_id)
|
||||
deprovision_git_gate(AppleGatewayTransport(), bottle_id)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"GatewayEndpoint",
|
||||
"LaunchContext",
|
||||
"ensure_gateway",
|
||||
"register_agent",
|
||||
"teardown_consolidated",
|
||||
"ConsolidatedLaunchError",
|
||||
"OrchestratorStartError",
|
||||
"GATEWAY_NETWORK",
|
||||
]
|
||||
@@ -1,11 +1,9 @@
|
||||
"""Host-side egress route-apply for the macos-container backend.
|
||||
|
||||
The per-bottle companion container this used to signal (`container kill
|
||||
--signal HUP <container>`) was removed in the companion-container removal
|
||||
(#385). In the consolidated model the shared gateway resolves egress policy
|
||||
per-request against the orchestrator rather than reloading a per-bottle routes
|
||||
file, so the live per-bottle reload is not supported here and fails closed
|
||||
until the gateway-side apply lands — same posture as the docker backend.
|
||||
--signal HUP <container>`) was removed in the companion-container removal (#385),
|
||||
along with the disabled macOS launch path. Fails closed until the macOS
|
||||
backend grows the consolidated gateway.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
@@ -18,8 +16,8 @@ class MacOSContainerEgressApplicator(EgressApplicator):
|
||||
del slug
|
||||
raise EgressApplyError(
|
||||
"live egress route-apply was removed with the per-bottle "
|
||||
"companion container (#385); route changes will flow through "
|
||||
"the consolidated gateway in a follow-up."
|
||||
"companion container (#385); the macos-container backend is "
|
||||
"disabled until it uses the consolidated gateway."
|
||||
)
|
||||
|
||||
|
||||
|
||||
@@ -1,42 +1,14 @@
|
||||
"""Active-agent enumeration for the macOS Apple Container backend."""
|
||||
"""Active-agent enumeration for the macOS Apple Container backend.
|
||||
|
||||
The backend is disabled during the companion-container removal (#385) — it can't
|
||||
launch bottles, so there are none to enumerate. Enumeration returns when
|
||||
the backend grows the consolidated gateway.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import subprocess
|
||||
|
||||
from ...bottle_state import read_metadata
|
||||
from .. import ActiveAgent
|
||||
from .infra import INFRA_NAME
|
||||
|
||||
_PREFIX = "bot-bottle-"
|
||||
# The shared per-host infra container carries the same prefix as agent
|
||||
# containers but is infrastructure, not a bottle — one control plane + gateway
|
||||
# serves every agent, so listing it as an agent would invent one per host.
|
||||
_INFRA_NAMES = frozenset({INFRA_NAME})
|
||||
|
||||
|
||||
def enumerate_active() -> list[ActiveAgent]:
|
||||
result = subprocess.run(
|
||||
["container", "list", "--quiet"],
|
||||
capture_output=True,
|
||||
text=True,
|
||||
check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
return []
|
||||
out: list[ActiveAgent] = []
|
||||
for name in sorted(line.strip() for line in result.stdout.splitlines()):
|
||||
if not name.startswith(_PREFIX) or name in _INFRA_NAMES:
|
||||
continue
|
||||
slug = name[len(_PREFIX):]
|
||||
metadata = read_metadata(slug)
|
||||
out.append(ActiveAgent(
|
||||
backend_name="macos-container",
|
||||
slug=slug,
|
||||
agent_name=metadata.agent_name if metadata else "?",
|
||||
started_at=metadata.started_at if metadata else "",
|
||||
services=(),
|
||||
label=metadata.label if metadata else "",
|
||||
color=metadata.color if metadata else "",
|
||||
))
|
||||
return out
|
||||
return []
|
||||
|
||||
@@ -1,46 +0,0 @@
|
||||
"""Shared network/image constants for the macOS consolidated infra container.
|
||||
|
||||
The gateway data plane no longer runs as its own Apple container — it shares a
|
||||
single per-host **infra container** with the control plane (see `infra`),
|
||||
because two Apple-Container guests writing one `bot-bottle.db` over virtiofs
|
||||
would race incoherent `fcntl` locks. This module holds the pieces both the
|
||||
infra service and the launch/provision glue need: the network names, the
|
||||
gateway image, and the network-creation helper.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
|
||||
from ...orchestrator.gateway import GatewayError
|
||||
from . import util as container_mod
|
||||
|
||||
# The shared host-only network the infra container and every agent bottle sit
|
||||
# on. The agent's address here is the attribution key. Distinct from the docker
|
||||
# names so both backends can coexist on one host.
|
||||
GATEWAY_NETWORK = "bot-bottle-mac-gateway"
|
||||
# The NAT network that gives the infra container (and only it) a route out.
|
||||
GATEWAY_EGRESS_NETWORK = "bot-bottle-mac-egress"
|
||||
|
||||
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
|
||||
|
||||
DEFAULT_CA_TIMEOUT_SECONDS = 30.0
|
||||
|
||||
|
||||
def ensure_networks(
|
||||
network: str = GATEWAY_NETWORK, egress_network: str = GATEWAY_EGRESS_NETWORK,
|
||||
) -> None:
|
||||
"""Create the shared host-only network + the NAT egress network. Idempotent
|
||||
— `create_network` tolerates 'already exists'."""
|
||||
container_mod.create_network(egress_network)
|
||||
container_mod.create_network(network, internal=True)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"GATEWAY_NETWORK",
|
||||
"GATEWAY_EGRESS_NETWORK",
|
||||
"GATEWAY_IMAGE",
|
||||
"GatewayError",
|
||||
"DEFAULT_CA_TIMEOUT_SECONDS",
|
||||
"ensure_networks",
|
||||
]
|
||||
@@ -1,44 +0,0 @@
|
||||
"""`GatewayTransport` for the Apple infra container (PRD 0070).
|
||||
|
||||
The provisioning *logic* (per-bottle creds dirs, namespaced repo init) is
|
||||
backend-neutral and lives in `backend.docker.gateway_provision`; this is only
|
||||
the transport — how files and commands reach the running gateway. Docker uses
|
||||
`docker exec`/`docker cp` and Firecracker uses SSH; Apple uses the `container`
|
||||
CLI's equivalents against the infra container that hosts the gateway daemons.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from ..docker.gateway_provision import GatewayProvisionError
|
||||
from . import util as container_mod
|
||||
from .infra import INFRA_NAME
|
||||
|
||||
|
||||
class AppleGatewayTransport:
|
||||
"""`GatewayTransport` for the gateway daemons in the Apple infra container."""
|
||||
|
||||
def __init__(self, gateway: str = INFRA_NAME) -> None:
|
||||
self.gateway = gateway
|
||||
|
||||
def exec(self, argv: list[str]) -> None:
|
||||
result = container_mod.run_container_argv(
|
||||
["container", "exec", self.gateway, *argv]
|
||||
)
|
||||
if result.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"gateway exec {argv!r} failed: "
|
||||
f"{(result.stderr or '').strip() or '<no stderr>'}"
|
||||
)
|
||||
|
||||
def cp_into(self, src: str, dest: str) -> None:
|
||||
result = container_mod.run_container_argv(
|
||||
["container", "cp", src, f"{self.gateway}:{dest}"]
|
||||
)
|
||||
if result.returncode != 0:
|
||||
raise GatewayProvisionError(
|
||||
f"gateway cp {src} -> {dest} failed: "
|
||||
f"{(result.stderr or '').strip() or '<no stderr>'}"
|
||||
)
|
||||
|
||||
|
||||
__all__ = ["AppleGatewayTransport", "GatewayProvisionError"]
|
||||
@@ -1,305 +0,0 @@
|
||||
"""The per-host infra container for the macOS backend (PRD 0070).
|
||||
|
||||
A single persistent Apple container that runs BOTH the orchestrator control
|
||||
plane and the gateway data plane — the macOS analogue of the Firecracker infra
|
||||
VM (`backend/firecracker/infra_vm.py`), not the docker backend's two separate
|
||||
containers.
|
||||
|
||||
Why one container, not two: Apple Containers are lightweight VMs, each with its
|
||||
own kernel. The docker backend runs the orchestrator and gateway as two
|
||||
containers safely because they share the host kernel, so their concurrent
|
||||
writes to the one `bot-bottle.db` (the orchestrator's registry + the gateway
|
||||
supervise daemon's queue) are serialized by coherent `fcntl` locks. Across two
|
||||
*guest* kernels sharing a virtiofs-mounted DB those locks are not coherent, and
|
||||
concurrent writers can corrupt the file. Firecracker solved this by putting
|
||||
both services in one guest with the DB on a device only that guest mounts; this
|
||||
does the same with Apple primitives.
|
||||
|
||||
Two consequences fall out of the single container, both simplifications:
|
||||
|
||||
- **No DNS dance.** The control plane and the gateway daemons reach each other
|
||||
over `127.0.0.1`, so nothing depends on Apple's (absent) container DNS and
|
||||
there is no orchestrator-before-gateway ordering to get right.
|
||||
- **The DB is never host-shared.** It lives on a container-only volume, so no
|
||||
host process opens the live file. The host CLI reaches registry + supervise
|
||||
state through the control-plane HTTP surface (`cli/supervise.py` already uses
|
||||
`OrchestratorClient`), exactly as it does for firecracker.
|
||||
|
||||
The control-plane source is bind-mounted (like the docker orchestrator), so a
|
||||
code change takes effect on the next launch without an image rebuild; the
|
||||
gateway daemons are baked in the gateway image and rebuild through its own
|
||||
digest check.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import time
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from ... import log
|
||||
from ...orchestrator.gateway import GATEWAY_CA_CERT
|
||||
from ...orchestrator.lifecycle import (
|
||||
DEFAULT_PORT,
|
||||
DEFAULT_STARTUP_TIMEOUT_SECONDS,
|
||||
OrchestratorStartError,
|
||||
source_hash,
|
||||
)
|
||||
from ...paths import (
|
||||
CONTROL_PLANE_TOKEN_ENV,
|
||||
HOST_DB_FILENAME,
|
||||
host_control_plane_token,
|
||||
)
|
||||
from . import util as container_mod
|
||||
from .gateway import (
|
||||
DEFAULT_CA_TIMEOUT_SECONDS,
|
||||
GATEWAY_EGRESS_NETWORK,
|
||||
GATEWAY_IMAGE,
|
||||
GATEWAY_NETWORK,
|
||||
GatewayError,
|
||||
ensure_networks,
|
||||
)
|
||||
|
||||
# The one per-host infra container: control plane + gateway data plane.
|
||||
INFRA_NAME = "bot-bottle-mac-infra"
|
||||
INFRA_LABEL = "bot-bottle-mac-infra=1"
|
||||
# Container-only volume holding bot-bottle.db. No host bind-mount, so the DB is
|
||||
# written by exactly one kernel (this container's). Survives recreation.
|
||||
INFRA_DB_VOLUME = "bot-bottle-mac-db"
|
||||
|
||||
# BOT_BOTTLE_ROOT inside the container; host_db_path() resolves the DB to
|
||||
# <root>/db/<filename> and the supervise daemon writes the same file.
|
||||
_DB_ROOT_IN_CONTAINER = "/var/lib/bot-bottle"
|
||||
_DB_PATH_IN_CONTAINER = f"{_DB_ROOT_IN_CONTAINER}/db/{HOST_DB_FILENAME}"
|
||||
_SRC_IN_CONTAINER = "/bot-bottle-src"
|
||||
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[3]
|
||||
|
||||
_HEALTH_POLL_SECONDS = 0.25
|
||||
_HEALTH_REQUEST_TIMEOUT_SECONDS = 1.0
|
||||
_CA_POLL_SECONDS = 0.5
|
||||
|
||||
# The gateway subset the consolidated model runs (no per-bottle git:// daemon).
|
||||
_GATEWAY_DAEMONS = "egress,git-http,supervise"
|
||||
|
||||
|
||||
def _init_script(port: int) -> str:
|
||||
"""PID-1 init: start the control plane and the gateway daemons, both in
|
||||
this container, reaching each other over loopback. Backgrounded so `wait`
|
||||
reaps as PID 1. No `set -e` — a transient daemon failure must not kill the
|
||||
whole container (gateway_init applies the same 'stay up' policy)."""
|
||||
return (
|
||||
"export PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n"
|
||||
f"mkdir -p $(dirname {_DB_PATH_IN_CONTAINER})\n"
|
||||
# Control plane, from the bind-mounted source (stdlib-only package).
|
||||
f"( cd {_SRC_IN_CONTAINER} && BOT_BOTTLE_ROOT={_DB_ROOT_IN_CONTAINER} "
|
||||
f"python3 -m bot_bottle.orchestrator --host 0.0.0.0 --port {port} "
|
||||
"--broker stub ) &\n"
|
||||
# Gateway data plane, multi-tenant against the local control plane.
|
||||
f"( cd /app && BOT_BOTTLE_GATEWAY_DAEMONS={_GATEWAY_DAEMONS} "
|
||||
f"BOT_BOTTLE_ORCHESTRATOR_URL=http://127.0.0.1:{port} "
|
||||
f"SUPERVISE_DB_PATH={_DB_PATH_IN_CONTAINER} python3 /app/gateway_init.py ) &\n"
|
||||
"while : ; do wait ; done\n"
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InfraEndpoint:
|
||||
"""How to reach the running infra container. The control plane and the
|
||||
gateway are the same container, so one address serves both."""
|
||||
|
||||
control_plane_url: str # http://<infra ip>:8099 — host CLI + registration
|
||||
gateway_ip: str # same container; agents' proxy / git-http / MCP target
|
||||
|
||||
|
||||
class MacosInfraService:
|
||||
"""Manages the single per-host infra container. Callers use
|
||||
`ensure_running()` (returns the endpoint) and `ca_cert_pem()`."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
*,
|
||||
port: int = DEFAULT_PORT,
|
||||
network: str = GATEWAY_NETWORK,
|
||||
egress_network: str = GATEWAY_EGRESS_NETWORK,
|
||||
image: str = GATEWAY_IMAGE,
|
||||
repo_root: Path = _REPO_ROOT,
|
||||
name: str = INFRA_NAME,
|
||||
db_volume: str = INFRA_DB_VOLUME,
|
||||
) -> None:
|
||||
self.port = port
|
||||
self.network = network
|
||||
self.egress_network = egress_network
|
||||
self.image = image
|
||||
self._repo_root = repo_root
|
||||
self._name = name
|
||||
self._db_volume = db_volume
|
||||
|
||||
def _resolve_url(self) -> str:
|
||||
"""The control-plane URL, or "" while the container has no address."""
|
||||
ip = container_mod.try_container_ipv4_on_network(self._name, self.network)
|
||||
return f"http://{ip}:{self.port}" if ip else ""
|
||||
|
||||
def is_healthy(
|
||||
self, url: str, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS,
|
||||
) -> bool:
|
||||
if not url:
|
||||
return False
|
||||
try:
|
||||
with urllib.request.urlopen(f"{url}/health", timeout=timeout) as resp:
|
||||
return resp.status == 200
|
||||
except (urllib.error.URLError, TimeoutError, OSError):
|
||||
return False
|
||||
|
||||
def _source_current(self, current_hash: str) -> bool:
|
||||
"""True iff the running infra container was created from the current
|
||||
bind-mounted control-plane source. The control-plane process loads that
|
||||
code at startup and won't reload it, so a stale container keeps serving
|
||||
OLD code."""
|
||||
if not container_mod.container_is_running(self._name):
|
||||
return False
|
||||
env = container_mod.container_env(self._name)
|
||||
if not env:
|
||||
return True # can't compare → don't churn a working container
|
||||
return env.get("BOT_BOTTLE_SOURCE_HASH") == current_hash
|
||||
|
||||
def _running_healthy_endpoint(self, current_hash: str) -> InfraEndpoint | None:
|
||||
"""The endpoint if the running container is BOTH source-current and
|
||||
answering /health, else None (→ recreate). Health, not just the source
|
||||
label, is what lets a wedged-but-current container self-heal instead of
|
||||
being polled to death forever."""
|
||||
if not self._source_current(current_hash):
|
||||
return None
|
||||
url = self._resolve_url()
|
||||
if url and self.is_healthy(url):
|
||||
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
|
||||
return None
|
||||
|
||||
def ensure_built(self) -> None:
|
||||
"""Ensure the gateway data-plane image exists. The control-plane source
|
||||
is bind-mounted, not baked, so only the gateway image needs building."""
|
||||
container_mod.build_image(
|
||||
self.image, str(self._repo_root), dockerfile="Dockerfile.gateway",
|
||||
)
|
||||
|
||||
def ensure_running(
|
||||
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
|
||||
) -> InfraEndpoint:
|
||||
"""Ensure the single infra container is up; return how to reach it.
|
||||
Idempotent per-host singleton — a healthy container on current source
|
||||
is left untouched, so N launches share the one control plane + gateway.
|
||||
Raises `OrchestratorStartError` on startup timeout."""
|
||||
current_hash = source_hash(self._repo_root)
|
||||
endpoint = self._running_healthy_endpoint(current_hash)
|
||||
if endpoint is not None:
|
||||
return endpoint
|
||||
self.ensure_built()
|
||||
log.info("starting infra container", context={"name": self._name})
|
||||
self._run_container(current_hash)
|
||||
return self._wait_healthy(startup_timeout)
|
||||
|
||||
def _run_container(self, current_hash: str) -> None:
|
||||
ensure_networks(self.network, self.egress_network)
|
||||
container_mod.force_remove_container(self._name)
|
||||
argv = [
|
||||
"container", "run", "--detach",
|
||||
"--name", self._name,
|
||||
"--label", "bot-bottle.backend=macos-container",
|
||||
"--label", INFRA_LABEL,
|
||||
# NAT network FIRST so the gateway's egress has a default route;
|
||||
# the host-only network is where agents (and the host CLI) reach it.
|
||||
"--network", self.egress_network,
|
||||
"--network", self.network,
|
||||
"--dns", container_mod.dns_server(),
|
||||
# Container-only DB volume: one kernel writes bot-bottle.db, never
|
||||
# shared with the host or another guest.
|
||||
"--volume", f"{self._db_volume}:{_DB_ROOT_IN_CONTAINER}",
|
||||
# Bind-mount the control-plane source (read-only); a code change
|
||||
# takes effect on relaunch with no image rebuild.
|
||||
"--mount",
|
||||
container_mod.bind_mount_spec(
|
||||
str(self._repo_root), _SRC_IN_CONTAINER, readonly=True),
|
||||
# Baked onto the container so `_source_current` can detect a real
|
||||
# control-plane code change and recreate.
|
||||
"--env", f"BOT_BOTTLE_SOURCE_HASH={current_hash}",
|
||||
# The control-plane secret, for BOTH the control plane (to require
|
||||
# it) and the gateway's PolicyResolver (to present it) — they share
|
||||
# this one container. Bare `--env NAME` inherits the value from the
|
||||
# run process below, so the secret never lands on argv or in
|
||||
# `container inspect`'s command line. The agent runs in a SEPARATE
|
||||
# container that is never given this var, which is the whole point.
|
||||
"--env", CONTROL_PLANE_TOKEN_ENV,
|
||||
"--entrypoint", "sh",
|
||||
self.image,
|
||||
"-c", _init_script(self.port),
|
||||
]
|
||||
run_env = {**os.environ, CONTROL_PLANE_TOKEN_ENV: host_control_plane_token()}
|
||||
result = container_mod.run_container_argv(argv, env=run_env)
|
||||
if result.returncode != 0:
|
||||
raise OrchestratorStartError(
|
||||
f"infra container failed to start: "
|
||||
f"{(result.stderr or '').strip() or '<no stderr>'}"
|
||||
)
|
||||
|
||||
def _wait_healthy(self, startup_timeout: float) -> InfraEndpoint:
|
||||
deadline = time.monotonic() + startup_timeout
|
||||
while True:
|
||||
url = self._resolve_url()
|
||||
if url and self.is_healthy(url):
|
||||
log.info("infra container healthy", context={"url": url})
|
||||
return InfraEndpoint(control_plane_url=url, gateway_ip=_ip_of(url))
|
||||
if time.monotonic() >= deadline:
|
||||
raise OrchestratorStartError(
|
||||
f"infra container did not become healthy within "
|
||||
f"{startup_timeout:g}s"
|
||||
)
|
||||
time.sleep(_HEALTH_POLL_SECONDS)
|
||||
|
||||
def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
|
||||
"""The gateway's mitmproxy CA (PEM) agents install to trust its TLS
|
||||
interception. Read out of the container (the CA lives on a
|
||||
container-internal path, not a host mount); polls because mitmproxy
|
||||
writes it a beat after start."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
result = container_mod.run_container_argv(
|
||||
["container", "exec", self._name, "cat", GATEWAY_CA_CERT])
|
||||
if result.returncode == 0 and result.stdout.strip():
|
||||
return result.stdout
|
||||
if time.monotonic() >= deadline:
|
||||
raise GatewayError(
|
||||
f"gateway CA not available in {self._name} after {timeout:g}s: "
|
||||
f"{(result.stderr or '').strip() or 'empty'}"
|
||||
)
|
||||
time.sleep(_CA_POLL_SECONDS)
|
||||
|
||||
def stop(self) -> None:
|
||||
"""Remove the infra container (idempotent). The DB volume persists."""
|
||||
container_mod.force_remove_container(self._name)
|
||||
|
||||
|
||||
def _ip_of(url: str) -> str:
|
||||
"""The host from an http://host:port URL."""
|
||||
return url.split("://", 1)[-1].rsplit(":", 1)[0]
|
||||
|
||||
|
||||
def probe_control_plane_url(port: int = DEFAULT_PORT) -> str:
|
||||
"""The running infra container's control-plane URL, or "" if it isn't up.
|
||||
Used by host-side control-plane discovery (`discover_orchestrator_url`);
|
||||
safe to call on any host — returns "" when the container or the `container`
|
||||
CLI isn't present."""
|
||||
ip = container_mod.try_container_ipv4_on_network(INFRA_NAME, GATEWAY_NETWORK)
|
||||
return f"http://{ip}:{port}" if ip else ""
|
||||
|
||||
|
||||
__all__ = [
|
||||
"MacosInfraService",
|
||||
"InfraEndpoint",
|
||||
"OrchestratorStartError",
|
||||
"GatewayError",
|
||||
"INFRA_NAME",
|
||||
"INFRA_DB_VOLUME",
|
||||
]
|
||||
@@ -1,74 +1,24 @@
|
||||
"""Launch flow for the macOS Apple Container backend (PRD 0070).
|
||||
"""Launch flow for the macOS Apple Container backend — disabled (#385).
|
||||
|
||||
The agent container attaches to the **shared host-only gateway network** and
|
||||
proxies egress through the one per-host gateway, replacing the per-bottle
|
||||
companion container removed in #385.
|
||||
This backend launched a per-bottle companion container (the egress /
|
||||
git-gate / supervise data plane) alongside the agent container, with the
|
||||
agent's proxy env pointed at the companion's host-only IP. That
|
||||
per-bottle-companion architecture was removed in the companion-container removal;
|
||||
the macOS backend will be re-enabled once it grows the consolidated
|
||||
per-host gateway the docker backend already uses.
|
||||
|
||||
The order differs from docker's, forced by Apple Container 1.0.0 having no
|
||||
`--ip` (see `consolidated_launch`): the agent is started *before* it is
|
||||
registered, because its DHCP-assigned address — the attribution key — does not
|
||||
exist until then.
|
||||
|
||||
gateway up -> run agent -> read its IP -> register it -> provision
|
||||
|
||||
Two things follow from that inversion:
|
||||
|
||||
- The **identity token** is minted by registration and so cannot be in the
|
||||
agent's run-time env; it rides the proxy URL applied at `container exec`
|
||||
time (`bottle.MacosContainerBottle`). `/resolve` requires it (#366), so
|
||||
egress without it is denied — hence the bare `sleep` init: every real agent
|
||||
command goes through exec and therefore carries the token.
|
||||
- The agent is run with `--cap-drop CAP_NET_RAW`. Apple Container grants
|
||||
NET_RAW by default, which would let an agent open a raw socket and forge a
|
||||
neighbour's source address on the shared segment. NET_ADMIN is already
|
||||
absent (the agent cannot change its own address or route), so dropping
|
||||
NET_RAW is what closes the source-address half of PRD 0070's invariant:
|
||||
"a packet's source address, as seen by the orchestrator, provably identifies
|
||||
the originating bottle." The identity token is the other half — an attacker
|
||||
would need to forge the address *and* steal the token — but the invariant is
|
||||
a stated precondition of consolidation, so it is enforced on its own terms
|
||||
rather than left to the token.
|
||||
Until then, launching a macOS bottle fails closed. `prepare` / `status`
|
||||
/ cleanup still work.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import dataclasses
|
||||
import os
|
||||
import subprocess
|
||||
from contextlib import ExitStack, contextmanager
|
||||
from pathlib import Path
|
||||
from contextlib import contextmanager
|
||||
from typing import Callable, Generator
|
||||
|
||||
from ...bottle_state import (
|
||||
egress_state_dir,
|
||||
git_gate_state_dir,
|
||||
read_committed_image,
|
||||
)
|
||||
from ...egress import (
|
||||
egress_agent_env_entries,
|
||||
egress_resolve_token_values,
|
||||
)
|
||||
from ...git_gate import (
|
||||
provision_git_gate_dynamic_keys,
|
||||
revoke_git_gate_provisioned_keys,
|
||||
)
|
||||
from ...git_http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
|
||||
from ...log import die, info, warn
|
||||
from ...supervise import SUPERVISE_PORT
|
||||
from ..docker.egress import EGRESS_PORT
|
||||
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
|
||||
from . import util as container_mod
|
||||
from ...log import die
|
||||
from .bottle import MacosContainerBottle
|
||||
from .bottle_plan import MacosContainerBottlePlan
|
||||
from .consolidated_launch import (
|
||||
GatewayEndpoint,
|
||||
ensure_gateway,
|
||||
register_agent,
|
||||
teardown_consolidated,
|
||||
)
|
||||
|
||||
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
|
||||
_AGENT_SLEEP_SECONDS = "2147483647"
|
||||
|
||||
|
||||
@contextmanager
|
||||
@@ -77,274 +27,13 @@ def launch(
|
||||
*,
|
||||
provision: Callable[[MacosContainerBottlePlan, "MacosContainerBottle"], str | None],
|
||||
) -> Generator[MacosContainerBottle, None, None]:
|
||||
"""Build, run, register, provision, and yield an Apple Container bottle on
|
||||
the shared per-host gateway."""
|
||||
stack = ExitStack()
|
||||
bottle_for_revoke = plan.manifest.bottle
|
||||
git_gate_dir_for_revoke = git_gate_state_dir(plan.slug)
|
||||
|
||||
def teardown() -> None:
|
||||
teardown_exc: BaseException | None = None
|
||||
try:
|
||||
stack.close()
|
||||
except BaseException as exc: # noqa: W0718 - teardown must continue
|
||||
teardown_exc = exc
|
||||
warn(f"macos-container teardown failed: {exc!r}")
|
||||
revoke_git_gate_provisioned_keys(bottle_for_revoke, git_gate_dir_for_revoke)
|
||||
if teardown_exc is not None:
|
||||
raise teardown_exc
|
||||
|
||||
try:
|
||||
plan = _build_images(plan)
|
||||
|
||||
# Step 1: the per-host singletons. Must precede the agent run — its
|
||||
# proxy env needs the gateway's address at `container run` time.
|
||||
endpoint = ensure_gateway()
|
||||
|
||||
# Step 2: mint this bottle's deploy keys, then point it at the SHARED
|
||||
# gateway's CA + git-http/supervise ports.
|
||||
plan = _provision_git_gate_keys(plan)
|
||||
plan = _install_gateway_ca(plan, endpoint)
|
||||
plan = _stamp_agent_urls(plan, endpoint)
|
||||
|
||||
# Step 3: run the agent. It has no identity token yet — registration
|
||||
# needs the address this run assigns.
|
||||
container_mod.force_remove_container(plan.container_name)
|
||||
_start_agent(plan, endpoint)
|
||||
stack.callback(container_mod.force_remove_container, plan.container_name)
|
||||
|
||||
# Step 4: read the assigned address and register by it. This is the
|
||||
# attribution key; `--cap-drop CAP_NET_RAW` at run is what makes it
|
||||
# unforgeable. Poll: `container run --detach` can return before vmnet's
|
||||
# DHCP has assigned the address.
|
||||
source_ip = container_mod.wait_container_ipv4_on_network(
|
||||
plan.container_name, endpoint.network,
|
||||
)
|
||||
if not source_ip:
|
||||
die(
|
||||
f"agent {plan.container_name} never got an address on "
|
||||
f"{endpoint.network}"
|
||||
)
|
||||
effective_env = {**os.environ, **plan.agent_provision.provisioned_env}
|
||||
token_values = egress_resolve_token_values(
|
||||
plan.egress_plan.token_env_map, effective_env,
|
||||
)
|
||||
ctx = register_agent(
|
||||
plan.egress_plan,
|
||||
plan.git_gate_plan,
|
||||
source_ip=source_ip,
|
||||
endpoint=endpoint,
|
||||
image_ref=plan.image,
|
||||
tokens=token_values,
|
||||
)
|
||||
stack.callback(
|
||||
teardown_consolidated, ctx.bottle_id,
|
||||
orchestrator_url=ctx.orchestrator_url,
|
||||
)
|
||||
info(
|
||||
f"agent {plan.container_name} registered "
|
||||
f"(gateway {endpoint.gateway_ip}, ip {source_ip})"
|
||||
)
|
||||
|
||||
# Stamp the token onto the plan so provision-time consumers can read it,
|
||||
# not only the exec-time egress proxy. git-gate's gitconfig extraHeader
|
||||
# and the supervise MCP --header both reach the gateway on NO_PROXY (they
|
||||
# bypass the egress proxy that carries the token), so without this the
|
||||
# gateway's /resolve fail-closes and every git fetch/push and supervise
|
||||
# call from the bottle is denied. Registration already produced the
|
||||
# token above, so — unlike the run-time env — the plan CAN carry it.
|
||||
plan = dataclasses.replace(plan, identity_token=ctx.identity_token)
|
||||
|
||||
bottle = MacosContainerBottle(
|
||||
plan.container_name,
|
||||
teardown,
|
||||
None,
|
||||
agent_command=plan.agent_command,
|
||||
agent_prompt_mode=plan.agent_prompt_mode,
|
||||
agent_provider_template=plan.agent_provider_template,
|
||||
terminal_title=(
|
||||
f"{plan.spec.label} ({plan.spec.agent_name})"
|
||||
if plan.spec.label else plan.spec.agent_name
|
||||
),
|
||||
terminal_color=plan.spec.color,
|
||||
agent_workdir=plan.workspace_plan.workdir,
|
||||
exec_env=_identity_proxy_env(endpoint, ctx.identity_token),
|
||||
)
|
||||
bottle.prompt_path = provision(plan, bottle)
|
||||
|
||||
yield bottle
|
||||
finally:
|
||||
teardown()
|
||||
|
||||
|
||||
def _build_images(plan: MacosContainerBottlePlan) -> MacosContainerBottlePlan:
|
||||
"""Build the agent image. The gateway's own image is built by
|
||||
`ensure_gateway` — it belongs to the shared singleton, not to a bottle."""
|
||||
committed = read_committed_image(plan.slug)
|
||||
if committed and container_mod.image_exists(committed):
|
||||
info(f"using committed image {committed!r}")
|
||||
return dataclasses.replace(
|
||||
plan,
|
||||
agent_provision=dataclasses.replace(
|
||||
plan.agent_provision, image=committed,
|
||||
),
|
||||
)
|
||||
container_mod.build_image(
|
||||
plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path,
|
||||
"""Fail closed: the macOS backend is disabled until it grows the
|
||||
consolidated per-host gateway (the companion-container path it used
|
||||
was removed in #385)."""
|
||||
del plan, provision
|
||||
die(
|
||||
"the macos-container backend is temporarily disabled during the "
|
||||
"companion-container removal (#385); it will return once it uses "
|
||||
"the consolidated gateway. Use --backend=docker for now."
|
||||
)
|
||||
return plan
|
||||
|
||||
|
||||
def _provision_git_gate_keys(
|
||||
plan: MacosContainerBottlePlan,
|
||||
) -> MacosContainerBottlePlan:
|
||||
if not plan.git_gate_plan.upstreams:
|
||||
return plan
|
||||
git_gate_plan = provision_git_gate_dynamic_keys(
|
||||
plan.manifest.bottle,
|
||||
plan.git_gate_plan,
|
||||
git_gate_state_dir(plan.slug),
|
||||
)
|
||||
return dataclasses.replace(plan, git_gate_plan=git_gate_plan)
|
||||
|
||||
|
||||
def _install_gateway_ca(
|
||||
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
|
||||
) -> MacosContainerBottlePlan:
|
||||
"""Stage the SHARED gateway's CA for the provisioner to install, replacing
|
||||
the per-bottle CA the companion container used to mint. Every bottle on
|
||||
this host trusts this one CA."""
|
||||
ca_dir = egress_state_dir(plan.slug) / "gateway-ca"
|
||||
ca_dir.mkdir(parents=True, exist_ok=True)
|
||||
ca_file = ca_dir / "gateway-ca.pem"
|
||||
ca_file.write_text(endpoint.gateway_ca_pem)
|
||||
egress_plan = dataclasses.replace(
|
||||
plan.egress_plan,
|
||||
mitmproxy_ca_host_path=ca_file,
|
||||
mitmproxy_ca_cert_only_host_path=ca_file,
|
||||
)
|
||||
return dataclasses.replace(plan, egress_plan=egress_plan)
|
||||
|
||||
|
||||
def _stamp_agent_urls(
|
||||
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
|
||||
) -> MacosContainerBottlePlan:
|
||||
"""Point the agent's git-gate insteadOf rewrites + supervise MCP at the
|
||||
shared gateway's ports. Both bypass the egress proxy (NO_PROXY covers the
|
||||
gateway address)."""
|
||||
git_gate_url = (
|
||||
f"http://{endpoint.gateway_ip}:{_GIT_HTTP_PORT}"
|
||||
if plan.git_gate_plan.upstreams else ""
|
||||
)
|
||||
supervise_url = (
|
||||
f"http://{endpoint.gateway_ip}:{SUPERVISE_PORT}/"
|
||||
if plan.supervise_plan is not None else ""
|
||||
)
|
||||
return dataclasses.replace(
|
||||
plan,
|
||||
agent_git_gate_url=git_gate_url,
|
||||
agent_supervise_url=supervise_url,
|
||||
)
|
||||
|
||||
|
||||
def _proxy_url(gateway_ip: str, identity_token: str = "") -> str:
|
||||
"""The agent's egress proxy URL. The identity token rides as proxy
|
||||
credentials — the gateway reads Proxy-Authorization, resolves the
|
||||
(source_ip, token) pair against the control plane, and strips it before
|
||||
upstream. Without a valid pair `/resolve` denies the request (#366)."""
|
||||
cred = f"bottle:{identity_token}@" if identity_token else ""
|
||||
return f"http://{cred}{gateway_ip}:{EGRESS_PORT}"
|
||||
|
||||
|
||||
def _no_proxy(gateway_ip: str) -> str:
|
||||
# git-http + supervise live on the gateway and must NOT go through the
|
||||
# egress proxy — the agent reaches them directly by its address.
|
||||
return f"localhost,127.0.0.1,{gateway_ip}"
|
||||
|
||||
|
||||
def _identity_proxy_env(
|
||||
endpoint: GatewayEndpoint, identity_token: str,
|
||||
) -> dict[str, str]:
|
||||
"""The token-bearing proxy env applied at `container exec`. It supersedes
|
||||
the token-less run-time value (exec `--env` wins), which is the only way to
|
||||
get the token in: it does not exist until after the container runs."""
|
||||
if not identity_token:
|
||||
return {}
|
||||
url = _proxy_url(endpoint.gateway_ip, identity_token)
|
||||
return {
|
||||
"HTTPS_PROXY": url, "HTTP_PROXY": url,
|
||||
"https_proxy": url, "http_proxy": url,
|
||||
}
|
||||
|
||||
|
||||
def _start_agent(plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint) -> None:
|
||||
argv = _agent_run_argv(plan, endpoint)
|
||||
env = {**os.environ, **plan.forwarded_env}
|
||||
info(f"container run agent {plan.container_name}")
|
||||
result = subprocess.run(
|
||||
argv, capture_output=True, text=True, env=env, check=False,
|
||||
)
|
||||
if result.returncode != 0:
|
||||
die(
|
||||
f"container run for agent {plan.container_name} failed: "
|
||||
f"{(result.stderr or '').strip() or '<no stderr>'}"
|
||||
)
|
||||
|
||||
|
||||
def _agent_run_argv(
|
||||
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
|
||||
) -> list[str]:
|
||||
argv = [
|
||||
"container", "run",
|
||||
"--name", plan.container_name,
|
||||
"--detach",
|
||||
"--label", "bot-bottle.backend=macos-container",
|
||||
"--network", endpoint.network,
|
||||
# The attribution invariant: without NET_RAW the agent cannot open a
|
||||
# raw socket, so it cannot source-IP-spoof its neighbours on the shared
|
||||
# segment. NET_ADMIN is not granted by default, so its address and
|
||||
# route are already fixed. See the module docstring.
|
||||
"--cap-drop", "CAP_NET_RAW",
|
||||
]
|
||||
for entry in _agent_env_entries(plan, endpoint):
|
||||
argv += ["--env", entry]
|
||||
# The init process is a no-op: every agent command arrives via
|
||||
# `container exec`, which is also how the identity token gets in.
|
||||
argv += [plan.image, "sleep", _AGENT_SLEEP_SECONDS]
|
||||
return argv
|
||||
|
||||
|
||||
def _agent_env_entries(
|
||||
plan: MacosContainerBottlePlan, endpoint: GatewayEndpoint,
|
||||
) -> tuple[str, ...]:
|
||||
# Token-less at run time — the token does not exist yet (see
|
||||
# `_identity_proxy_env`). Anything egressing before the exec-time override
|
||||
# is denied by `/resolve`, which is the safe direction.
|
||||
proxy_url = _proxy_url(endpoint.gateway_ip)
|
||||
no_proxy = _no_proxy(endpoint.gateway_ip)
|
||||
env = [
|
||||
f"HTTPS_PROXY={proxy_url}",
|
||||
f"HTTP_PROXY={proxy_url}",
|
||||
f"https_proxy={proxy_url}",
|
||||
f"http_proxy={proxy_url}",
|
||||
f"NO_PROXY={no_proxy}",
|
||||
f"no_proxy={no_proxy}",
|
||||
f"NODE_EXTRA_CA_CERTS={AGENT_CA_PATH}",
|
||||
f"SSL_CERT_FILE={AGENT_CA_BUNDLE}",
|
||||
f"REQUESTS_CA_BUNDLE={AGENT_CA_BUNDLE}",
|
||||
]
|
||||
if plan.agent_git_gate_url:
|
||||
env.append(f"GIT_GATE_URL={plan.agent_git_gate_url}")
|
||||
if plan.agent_supervise_url:
|
||||
env.append(f"MCP_SUPERVISE_URL={plan.agent_supervise_url}")
|
||||
for name, value in sorted(plan.agent_provision.guest_env.items()):
|
||||
env.append(f"{name}={value}")
|
||||
# Forwarded vars: bare name → inherits from the `container run` process env
|
||||
# so the secret value never lands on argv.
|
||||
for name in sorted(plan.forwarded_env.keys()):
|
||||
env.append(name)
|
||||
env.extend(egress_agent_env_entries(plan.egress_plan))
|
||||
return tuple(env)
|
||||
|
||||
|
||||
__all__ = ["launch"]
|
||||
yield # unreachable — `die` raises; keeps this a generator/contextmanager
|
||||
|
||||
@@ -437,145 +437,22 @@ def inspect_container(name: str) -> dict[str, object]:
|
||||
|
||||
|
||||
def container_ipv4_on_network(name: str, network: str) -> str:
|
||||
"""The container's IPv4 address on `network`. Fatal if absent — callers
|
||||
that can tolerate "not yet" want `try_container_ipv4_on_network`."""
|
||||
ip = try_container_ipv4_on_network(name, network)
|
||||
if not ip:
|
||||
die(f"container {name} has no IPv4 address on {network}")
|
||||
return ip
|
||||
|
||||
|
||||
def run_container_argv(
|
||||
argv: list[str], *, env: dict[str, str] | None = None,
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
"""Run a `container` command, returning the result for the caller to
|
||||
interpret. Unlike the `die`-on-failure helpers above, this lets callers
|
||||
that raise their own typed errors (the gateway / orchestrator lifecycle)
|
||||
keep control of the failure path.
|
||||
|
||||
`env` sets the child process environment — used to hand a secret to a bare
|
||||
`--env NAME` flag (Apple's "just key → inherit from host" form) so the
|
||||
value is inherited from this process, never written onto argv or into
|
||||
`container inspect`'s recorded command line."""
|
||||
return subprocess.run(
|
||||
argv, capture_output=True, text=True, check=False, env=env)
|
||||
|
||||
|
||||
def bind_mount_spec(source: str, target: str, *, readonly: bool = False) -> str:
|
||||
"""A `container run --mount` bind spec. One definition so the gateway and
|
||||
orchestrator emit an identical string — a divergence here would silently
|
||||
break one backend's mounts while the other kept working."""
|
||||
spec = f"type=bind,source={source},target={target}"
|
||||
if readonly:
|
||||
spec += ",readonly"
|
||||
return spec
|
||||
|
||||
|
||||
def _normalize_digest(value: str) -> str:
|
||||
return value.split(":", 1)[1] if ":" in value else value
|
||||
|
||||
|
||||
def _inspect_first(argv: list[str]) -> dict[str, object]:
|
||||
"""Run an inspect command and return its first JSON object, or {} on any
|
||||
failure (non-zero exit, malformed JSON, unexpected shape). {} is the shared
|
||||
'don't know' signal all the non-fatal inspect readers below build on — a
|
||||
caller comparing against it treats it as 'leave the working container
|
||||
alone', never as a mismatch."""
|
||||
result = run_container_argv(argv)
|
||||
if result.returncode != 0:
|
||||
return {}
|
||||
try:
|
||||
data = json.loads(result.stdout or "[]")
|
||||
except json.JSONDecodeError:
|
||||
return {}
|
||||
if isinstance(data, list):
|
||||
data = data[0] if data else {}
|
||||
return data if isinstance(data, dict) else {}
|
||||
|
||||
|
||||
def _descriptor_digest(node: object) -> str:
|
||||
"""The normalized digest under a `{... "descriptor": {"digest": ...}}`
|
||||
node, or "". Both the image and container inspect shapes nest the image's
|
||||
identity this way, so the digest readers stay symmetric — a difference
|
||||
between them is what would spuriously recreate a container."""
|
||||
if not isinstance(node, dict):
|
||||
return ""
|
||||
descriptor = node.get("descriptor")
|
||||
if isinstance(descriptor, dict) and descriptor.get("digest"):
|
||||
return _normalize_digest(str(descriptor["digest"]))
|
||||
return ""
|
||||
|
||||
|
||||
def image_digest(ref: str) -> str:
|
||||
"""The digest of image `ref`, or "" if it can't be read. Reads exactly the
|
||||
field `container_image_digest` reads (`configuration.descriptor.digest`) so
|
||||
the two are comparable; "" means 'don't know' → callers don't churn."""
|
||||
data = _inspect_first([_CONTAINER, "image", "inspect", ref])
|
||||
return _descriptor_digest(data.get("configuration"))
|
||||
|
||||
|
||||
def container_image_digest(name: str) -> str:
|
||||
"""The digest of the image container `name` was created from, or "" if it
|
||||
can't be read. Compare with `image_digest(ref)` to tell whether a running
|
||||
container predates an image rebuild."""
|
||||
config = _inspect_first([_CONTAINER, "inspect", name]).get("configuration")
|
||||
image = config.get("image") if isinstance(config, dict) else None
|
||||
return _descriptor_digest(image)
|
||||
|
||||
|
||||
def container_env(name: str) -> dict[str, str]:
|
||||
"""The env container `name` was started with, or {} if unreadable. Lets a
|
||||
caller tell whether a running container's baked-in configuration still
|
||||
matches what it would pass today."""
|
||||
config = _inspect_first([_CONTAINER, "inspect", name]).get("configuration")
|
||||
init = config.get("initProcess") if isinstance(config, dict) else None
|
||||
entries = init.get("environment") if isinstance(init, dict) else None
|
||||
if not isinstance(entries, list):
|
||||
return {}
|
||||
env: dict[str, str] = {}
|
||||
for entry in entries:
|
||||
if isinstance(entry, str) and "=" in entry:
|
||||
key, value = entry.split("=", 1)
|
||||
env[key] = value
|
||||
return env
|
||||
|
||||
|
||||
def try_container_ipv4_on_network(name: str, network: str) -> str:
|
||||
"""`container_ipv4_on_network` without the fatal exit: "" when the address
|
||||
isn't readable yet. For pollers — a container is created before it has an
|
||||
address, so "not yet" is an expected state there, not an error."""
|
||||
status = _inspect_first([_CONTAINER, "inspect", name]).get("status")
|
||||
data = inspect_container(name)
|
||||
status = data.get("status")
|
||||
networks = status.get("networks") if isinstance(status, dict) else None
|
||||
if not isinstance(networks, list):
|
||||
return ""
|
||||
die(f"container inspect {name} did not include status.networks")
|
||||
for entry in networks:
|
||||
if not isinstance(entry, dict) or entry.get("network") != network:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
if entry.get("network") != network:
|
||||
continue
|
||||
raw = entry.get("ipv4Address")
|
||||
if isinstance(raw, str) and raw:
|
||||
return raw.split("/", 1)[0]
|
||||
return ""
|
||||
|
||||
|
||||
def wait_container_ipv4_on_network(
|
||||
name: str, network: str, *, timeout: float = 15.0, poll: float = 0.25,
|
||||
) -> str:
|
||||
"""Poll for the container's DHCP-assigned address on `network`, returning
|
||||
it once available or "" on timeout.
|
||||
|
||||
Apple Container has no `--ip`: `container run --detach` can return before
|
||||
vmnet's DHCP has populated `status.networks[].ipv4Address`, so a bare read
|
||||
right after start races the assignment. Callers that need the address (the
|
||||
attribution key, the gateway's proxy target) poll through here instead of
|
||||
the fatal `container_ipv4_on_network`."""
|
||||
deadline = time.monotonic() + timeout
|
||||
while True:
|
||||
ip = try_container_ipv4_on_network(name, network)
|
||||
if ip:
|
||||
return ip
|
||||
if time.monotonic() >= deadline:
|
||||
return ""
|
||||
time.sleep(poll)
|
||||
if not isinstance(raw, str) or not raw:
|
||||
die(f"container {name} has no IPv4 address on {network}")
|
||||
return raw.split("/", 1)[0]
|
||||
die(f"container {name} is not attached to network {network}")
|
||||
raise AssertionError("unreachable")
|
||||
|
||||
|
||||
def image_id(ref: str) -> str:
|
||||
|
||||
@@ -31,7 +31,6 @@ from __future__ import annotations
|
||||
import dataclasses
|
||||
import json
|
||||
import secrets
|
||||
import socket
|
||||
import string
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
@@ -44,7 +43,6 @@ from .paths import bot_bottle_root
|
||||
_STATE_SUBDIR = "state"
|
||||
_PER_BOTTLE_DOCKERFILE_NAME = "Dockerfile"
|
||||
_COMMITTED_IMAGE_NAME = "committed-image"
|
||||
_COMMITTED_ROOTFS_NAME = "committed-rootfs.tar"
|
||||
_TRANSCRIPT_SUBDIR = "transcript"
|
||||
# Per-daemon scratch subdirs. PRD 0018 chunk 2: bind-mount sources
|
||||
# live here so chunk 3's `docker compose up` can find them at stable
|
||||
@@ -89,14 +87,6 @@ def bottle_identity(agent_name: str) -> str:
|
||||
return f"{slug}-{suffix}"
|
||||
|
||||
|
||||
def globalize_slug(slug: str) -> str:
|
||||
"""Return a globally-unique slug qualified with the current hostname.
|
||||
|
||||
Assumes slug is a value returned from mint_slug. Use wherever a slug
|
||||
must be unique across hosts (e.g. deploy-key titles)."""
|
||||
return f"{socket.gethostname()}-{slug}"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class BottleMetadata:
|
||||
"""Persistent record of how a bottle was launched, written at
|
||||
@@ -201,15 +191,6 @@ def committed_image_path(identity: str) -> Path:
|
||||
return bottle_state_dir(identity) / _COMMITTED_IMAGE_NAME
|
||||
|
||||
|
||||
def committed_rootfs_path(identity: str) -> Path:
|
||||
"""Where the Firecracker freezer stores a snapshot of the bottle's
|
||||
guest rootfs (a plain tar). This is the resumable/migratable artifact
|
||||
the Firecracker backend boots from — no Docker image involved. The
|
||||
matching `committed-image` state file records that a snapshot exists
|
||||
(and its path); `resume` boots from this tar when both are present."""
|
||||
return bottle_state_dir(identity) / _COMMITTED_ROOTFS_NAME
|
||||
|
||||
|
||||
def write_committed_image(identity: str, image_tag: str) -> Path:
|
||||
"""Persist the committed image tag for `identity`. The next
|
||||
`cli.py resume <identity>` will boot from this image instead of
|
||||
@@ -359,12 +340,10 @@ __all__ = [
|
||||
"BottleMetadata",
|
||||
"agent_state_dir",
|
||||
"bottle_identity",
|
||||
"globalize_slug",
|
||||
"bottle_state_dir",
|
||||
"cleanup_state",
|
||||
"clear_preserve_marker",
|
||||
"committed_image_path",
|
||||
"committed_rootfs_path",
|
||||
"egress_state_dir",
|
||||
"git_gate_state_dir",
|
||||
"is_preserved",
|
||||
|
||||
+87
-84
@@ -20,19 +20,32 @@ from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
from ..paths import bot_bottle_root
|
||||
from ..log import Die, error, info
|
||||
from ..orchestrator.client import (
|
||||
OrchestratorClient,
|
||||
OrchestratorClientError,
|
||||
discover_orchestrator_url,
|
||||
from ..bottle_state import read_metadata
|
||||
from ..backend.docker.egress_apply import (
|
||||
EgressApplyError,
|
||||
applicator as _docker_applicator,
|
||||
)
|
||||
from ..backend.macos_container.egress_apply import (
|
||||
applicator as _macos_applicator,
|
||||
)
|
||||
from ..log import Die, error, info
|
||||
|
||||
from ..supervise import (
|
||||
COMPONENT_FOR_TOOL,
|
||||
AuditEntry,
|
||||
Proposal,
|
||||
Response,
|
||||
STATUS_APPROVED,
|
||||
STATUS_MODIFIED,
|
||||
STATUS_REJECTED,
|
||||
TOOL_EGRESS_ALLOW,
|
||||
TOOL_EGRESS_BLOCK,
|
||||
TOOL_GITLEAKS_ALLOW,
|
||||
TOOL_EGRESS_TOKEN_ALLOW,
|
||||
list_all_pending_proposals,
|
||||
render_diff,
|
||||
write_audit_entry,
|
||||
write_response,
|
||||
)
|
||||
from ._common import PROG
|
||||
|
||||
@@ -47,61 +60,30 @@ _REPORT_ONLY_TOOLS: tuple[str, ...] = (TOOL_GITLEAKS_ALLOW, TOOL_EGRESS_TOKEN_AL
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class QueuedProposal:
|
||||
"""A pending proposal from the supervise queue.
|
||||
|
||||
`label` is the operator-facing bottle name (the human slug the
|
||||
orchestrator resolved from the registry); `proposal.bottle_slug` is the
|
||||
opaque bottle_id every operator action is keyed by. Display uses `label`;
|
||||
respond calls use `proposal.bottle_slug`."""
|
||||
"""A pending proposal from the supervise queue."""
|
||||
|
||||
proposal: Proposal
|
||||
label: str = ""
|
||||
|
||||
|
||||
# A failed operator action (orchestrator unreachable, bottle torn down,
|
||||
# 409) is caught by the TUI key handlers and surfaced in the status line so
|
||||
# the proposal stays pending rather than crashing curses.
|
||||
ApplyError = (OrchestratorClientError,)
|
||||
# Errors any remediation engine may raise. Caught by the TUI key
|
||||
# handlers and surfaced in the status line so a failed apply keeps
|
||||
# the proposal pending rather than crashing curses.
|
||||
ApplyError = (EgressApplyError,)
|
||||
|
||||
|
||||
# The one per-host orchestrator, discovered lazily on first use. Every
|
||||
# operator action — list, approve, reject — goes through its HTTP control
|
||||
# plane (the orchestrator owns the single DB + live policy); there is no
|
||||
# direct-DB path and no backend branching here.
|
||||
_client_instance: OrchestratorClient | None = None
|
||||
|
||||
|
||||
def _resolve_orchestrator_url() -> str:
|
||||
"""URL of the running orchestrator control plane, starting one on demand.
|
||||
|
||||
Supervise is often the first thing an operator runs — before any bottle
|
||||
has booted the control plane. So when discovery finds nothing, bring up
|
||||
the selected backend's orchestrator + gateway (idempotent) rather than
|
||||
failing with "launch a bottle first"."""
|
||||
try:
|
||||
return discover_orchestrator_url()
|
||||
except OrchestratorClientError:
|
||||
from ..backend import get_bottle_backend
|
||||
backend = get_bottle_backend()
|
||||
info(f"no orchestrator control plane running; starting one ({backend.name})…")
|
||||
return backend.ensure_orchestrator()
|
||||
|
||||
|
||||
def _client() -> OrchestratorClient:
|
||||
global _client_instance # noqa: PLW0603 — CLI-session singleton
|
||||
if _client_instance is None:
|
||||
_client_instance = OrchestratorClient(_resolve_orchestrator_url())
|
||||
return _client_instance
|
||||
def apply_routes_change(slug: str, content: str) -> tuple[str, str]:
|
||||
meta = read_metadata(slug)
|
||||
backend = meta.backend if meta is not None else ""
|
||||
if backend == "macos-container":
|
||||
return _macos_applicator.apply_routes_change(slug, content)
|
||||
return _docker_applicator.apply_routes_change(slug, content)
|
||||
|
||||
|
||||
def discover_pending() -> list[QueuedProposal]:
|
||||
"""Collect pending proposals across bottles from the orchestrator."""
|
||||
"""Collect pending proposals across bottles."""
|
||||
out = [
|
||||
QueuedProposal(
|
||||
proposal=Proposal.from_dict(d),
|
||||
label=str(d.get("bottle_label") or d.get("bottle_slug") or ""),
|
||||
)
|
||||
for d in _client().supervise_pending()
|
||||
QueuedProposal(proposal=proposal)
|
||||
for proposal in list_all_pending_proposals()
|
||||
]
|
||||
out.sort(key=lambda q: q.proposal.arrival_timestamp)
|
||||
return out
|
||||
@@ -109,8 +91,8 @@ def discover_pending() -> list[QueuedProposal]:
|
||||
|
||||
def _approval_status(qp: QueuedProposal, verb: str) -> str:
|
||||
"""Status-line text after a successful approval."""
|
||||
base = f"{verb} {qp.proposal.tool} for [{qp.label}]"
|
||||
return f"{base}; resume: ./cli.py resume {qp.label}"
|
||||
base = f"{verb} {qp.proposal.tool} for [{qp.proposal.bottle_slug}]"
|
||||
return f"{base}; resume: ./cli.py resume {qp.proposal.bottle_slug}"
|
||||
|
||||
|
||||
def _detail_lines(
|
||||
@@ -121,7 +103,7 @@ def _detail_lines(
|
||||
"""Return the detail-view body as (text, curses-attr) tuples."""
|
||||
p = qp.proposal
|
||||
out: list[tuple[str, int]] = [
|
||||
(f"bottle: {qp.label}", 0),
|
||||
(f"bottle: {p.bottle_slug}", 0),
|
||||
(f"tool: {p.tool}", 0),
|
||||
(f"id: {p.id}", 0),
|
||||
(f"arrived: {p.arrival_timestamp}", 0),
|
||||
@@ -154,27 +136,39 @@ def approve(
|
||||
notes: str = "",
|
||||
final_file: str | None = None,
|
||||
) -> None:
|
||||
"""Approve (or, with `final_file`, modify-then-approve) via the
|
||||
orchestrator: it applies the route change to the bottle's live policy,
|
||||
writes the response that unblocks the agent, and audits it — one atomic
|
||||
server-side op. Raises `OrchestratorClientError` on failure."""
|
||||
_client().supervise_respond(
|
||||
qp.proposal.id,
|
||||
bottle_slug=qp.proposal.bottle_slug,
|
||||
decision="modify" if final_file is not None else "approve",
|
||||
"""Apply the proposal, write the waiting response, and audit it."""
|
||||
status = STATUS_MODIFIED if final_file is not None else STATUS_APPROVED
|
||||
file_to_apply = final_file if final_file is not None else qp.proposal.proposed_file
|
||||
|
||||
diff_before, diff_after = "", ""
|
||||
if qp.proposal.tool in (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK):
|
||||
diff_before, diff_after = apply_routes_change(
|
||||
qp.proposal.bottle_slug,
|
||||
file_to_apply,
|
||||
)
|
||||
|
||||
response = Response(
|
||||
proposal_id=qp.proposal.id,
|
||||
status=status,
|
||||
notes=notes,
|
||||
final_file=final_file,
|
||||
)
|
||||
|
||||
write_response(qp.proposal.bottle_slug, response)
|
||||
_write_audit(
|
||||
qp, action=status, notes=notes,
|
||||
diff_before=diff_before, diff_after=diff_after,
|
||||
)
|
||||
|
||||
def reject(qp: QueuedProposal, *, reason: str) -> None:
|
||||
"""Reject via the orchestrator (writes the response + audit)."""
|
||||
_client().supervise_respond(
|
||||
qp.proposal.id,
|
||||
bottle_slug=qp.proposal.bottle_slug,
|
||||
decision="reject",
|
||||
"""Write a rejection response and an audit entry."""
|
||||
response = Response(
|
||||
proposal_id=qp.proposal.id,
|
||||
status=STATUS_REJECTED,
|
||||
notes=reason,
|
||||
final_file=None,
|
||||
)
|
||||
write_response(qp.proposal.bottle_slug, response)
|
||||
_write_audit(qp, action=STATUS_REJECTED, notes=reason, diff_before="", diff_after="")
|
||||
|
||||
|
||||
def _approve_from_tui(
|
||||
@@ -194,6 +188,29 @@ def _approve_from_tui(
|
||||
return _approval_status(qp, verb)
|
||||
|
||||
|
||||
def _write_audit(
|
||||
qp: QueuedProposal,
|
||||
*,
|
||||
action: str,
|
||||
notes: str,
|
||||
diff_before: str,
|
||||
diff_after: str,
|
||||
) -> None:
|
||||
"""Audit log for egress tool."""
|
||||
component = COMPONENT_FOR_TOOL.get(qp.proposal.tool)
|
||||
if component is None:
|
||||
return
|
||||
write_audit_entry(AuditEntry(
|
||||
timestamp=datetime.now(timezone.utc).isoformat(),
|
||||
bottle_slug=qp.proposal.bottle_slug,
|
||||
component=component,
|
||||
operator_action=action,
|
||||
operator_notes=notes,
|
||||
justification=qp.proposal.justification,
|
||||
diff=render_diff(diff_before, diff_after, label=component),
|
||||
))
|
||||
|
||||
|
||||
# --- $EDITOR integration --------------------------------------------------
|
||||
|
||||
|
||||
@@ -228,20 +245,6 @@ def cmd_supervise(argv: list[str]) -> int:
|
||||
)
|
||||
args = parser.parse_args(argv)
|
||||
|
||||
# Establish the orchestrator connection up front so a missing control
|
||||
# plane is a clean one-line error, not a curses crash mid-loop. This also
|
||||
# starts the orchestrator on demand when none is running (see `_client`).
|
||||
try:
|
||||
_client()
|
||||
except OrchestratorClientError as e:
|
||||
error(str(e))
|
||||
return 1
|
||||
except Die as e:
|
||||
# Backend has no orchestrator to start (e.g. macos-container).
|
||||
if e.message:
|
||||
error(e.message)
|
||||
return e.code if isinstance(e.code, int) else 1
|
||||
|
||||
if args.once:
|
||||
return _list_once()
|
||||
try:
|
||||
@@ -296,7 +299,7 @@ def _list_once() -> int:
|
||||
for qp in pending:
|
||||
sys.stdout.write(
|
||||
f"{qp.proposal.arrival_timestamp} "
|
||||
f"[{qp.label}] "
|
||||
f"[{qp.proposal.bottle_slug}] "
|
||||
f"{qp.proposal.tool} "
|
||||
f"{qp.proposal.id}\n"
|
||||
)
|
||||
@@ -393,7 +396,7 @@ def _main_loop(stdscr: "curses._CursesWindow") -> None: # type: ignore # pragm
|
||||
reason = _prompt(stdscr, "reject reason: ")
|
||||
if reason:
|
||||
reject(qp, reason=reason)
|
||||
status_line = f"rejected {qp.proposal.tool} for [{qp.label}]"
|
||||
status_line = f"rejected {qp.proposal.tool} for [{qp.proposal.bottle_slug}]"
|
||||
else:
|
||||
status_line = "reject aborted (empty reason)"
|
||||
|
||||
@@ -432,7 +435,7 @@ def _render(
|
||||
cursor = "> " if i == selected else " "
|
||||
line = (
|
||||
f"{cursor}{ts_short} "
|
||||
f"[{qp.label}] {p.tool:<18} {p.id[:8]}"
|
||||
f"[{p.bottle_slug}] {p.tool:<18} {p.id[:8]}"
|
||||
)
|
||||
attr = curses.A_REVERSE if i == selected else curses.A_NORMAL
|
||||
stdscr.addnstr(row, 0, line, w - 1, attr)
|
||||
|
||||
@@ -32,8 +32,6 @@ if TYPE_CHECKING:
|
||||
|
||||
|
||||
_SUPERVISE_MCP_NAME = "supervise"
|
||||
# App-layer identity token header (mirrors egress_addon / git_http_backend).
|
||||
_IDENTITY_HEADER = "x-bot-bottle-identity"
|
||||
|
||||
|
||||
def _skills_dir(guest_home: str) -> str:
|
||||
@@ -303,15 +301,9 @@ class ClaudeAgentProvider(AgentProvider):
|
||||
if plan.supervise_plan is None:
|
||||
return
|
||||
info(f"registering supervise MCP server in agent claude config → {supervise_url}")
|
||||
# Deliver the identity token as an MCP request header — the supervise
|
||||
# daemon requires it (mandatory (source_ip, token) attribution).
|
||||
token = getattr(plan, "identity_token", "")
|
||||
header = (
|
||||
f" --header {shlex.quote(f'{_IDENTITY_HEADER}: {token}')}" if token else ""
|
||||
)
|
||||
r = bottle.exec(
|
||||
f"claude mcp add --scope user --transport http "
|
||||
f"{_SUPERVISE_MCP_NAME} {supervise_url}{header}",
|
||||
f"{_SUPERVISE_MCP_NAME} {supervise_url}",
|
||||
user="node",
|
||||
)
|
||||
if r.returncode != 0:
|
||||
|
||||
@@ -9,7 +9,6 @@ invocation that registers the supervise daemon in Codex's
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import base64
|
||||
import os
|
||||
import shlex
|
||||
from pathlib import Path
|
||||
@@ -27,7 +26,7 @@ from ...agent_provider import (
|
||||
)
|
||||
from .codex_auth import codex_host_access_token, write_codex_dummy_auth_file
|
||||
from ...egress import CODEX_HOST_CREDENTIAL_TOKEN_REF, EgressRoute
|
||||
from ...log import die, info
|
||||
from ...log import die, info, warn
|
||||
|
||||
|
||||
if TYPE_CHECKING:
|
||||
@@ -35,8 +34,6 @@ if TYPE_CHECKING:
|
||||
|
||||
|
||||
_SUPERVISE_MCP_NAME = "supervise"
|
||||
# App-layer identity token header (mirrors egress_addon / git_http_backend).
|
||||
_IDENTITY_HEADER = "x-bot-bottle-identity"
|
||||
_CODEX_CLI = "/home/node/.codex/packages/standalone/current/bin/codex"
|
||||
_CODEX_CLI_PATH = (
|
||||
"/home/node/.local/bin:"
|
||||
@@ -45,41 +42,6 @@ _CODEX_CLI_PATH = (
|
||||
)
|
||||
|
||||
|
||||
def _toml_basic_string(value: str) -> str:
|
||||
"""Quote `value` as a TOML basic (double-quoted) string."""
|
||||
escaped = (
|
||||
value.replace("\\", "\\\\")
|
||||
.replace('"', '\\"')
|
||||
.replace("\n", "\\n")
|
||||
.replace("\t", "\\t")
|
||||
)
|
||||
return f'"{escaped}"'
|
||||
|
||||
|
||||
def _supervise_mcp_config_toml(supervise_url: str, token: str) -> str:
|
||||
"""Render the `[mcp_servers.supervise]` streamable-HTTP entry for
|
||||
Codex's `config.toml`.
|
||||
|
||||
The Codex CLI has no `mcp add --header` flag; a static request
|
||||
header on an HTTP MCP server is only expressible via the
|
||||
`http_headers` config key (see `RawMcpServerConfig` /
|
||||
`McpServerTransportConfig::StreamableHttp`). We deliver the
|
||||
mandatory identity token (source_ip, token attribution) that way.
|
||||
Only Codex-supported streamable-HTTP keys (`url`, `http_headers`)
|
||||
are emitted."""
|
||||
lines = [
|
||||
"",
|
||||
f"[mcp_servers.{_SUPERVISE_MCP_NAME}]",
|
||||
f"url = {_toml_basic_string(supervise_url)}",
|
||||
]
|
||||
if token:
|
||||
key = _toml_basic_string(_IDENTITY_HEADER)
|
||||
val = _toml_basic_string(token)
|
||||
lines.append(f"http_headers = {{ {key} = {val} }}")
|
||||
lines.append("")
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _skills_dir(guest_home: str) -> str:
|
||||
# Codex agents still read skills from the claude-code convention
|
||||
# (~/.claude/skills/) — the bot-bottle-codex image follows the
|
||||
@@ -304,39 +266,25 @@ class CodexAgentProvider(AgentProvider):
|
||||
bottle: "Bottle",
|
||||
supervise_url: str,
|
||||
) -> None:
|
||||
"""Register the supervise daemon as a streamable-HTTP MCP
|
||||
server in Codex's user config (`~/.codex/config.toml`).
|
||||
"""Run `codex mcp add` inside the agent guest to register the
|
||||
supervise daemon in Codex's user config (~/.codex/config.toml).
|
||||
|
||||
We write the `[mcp_servers.supervise]` entry directly rather
|
||||
than shelling out to `codex mcp add`: the CLI's `add` has no
|
||||
way to attach a static request header, and the identity token
|
||||
(mandatory (source_ip, token) attribution) MUST ride on the
|
||||
MCP request as `http_headers`. Failure is FATAL when supervise
|
||||
is enabled — a silently-unregistered server leaves the agent
|
||||
with no supervise access and, under mandatory attribution, no
|
||||
way to recover from inside the bottle."""
|
||||
Mirrors the Claude provider's `claude mcp add` flow — failure
|
||||
is logged but not fatal."""
|
||||
if plan.supervise_plan is None:
|
||||
return
|
||||
info(f"registering supervise MCP server in agent codex config → {supervise_url}")
|
||||
token = getattr(plan, "identity_token", "")
|
||||
block = _supervise_mcp_config_toml(supervise_url, token)
|
||||
auth_dir = plan.agent_provision.guest_env.get("CODEX_HOME") \
|
||||
or f"{plan.guest_home}/.codex"
|
||||
config_path = f"{auth_dir}/config.toml"
|
||||
# Append via base64 so the TOML payload never has to survive a
|
||||
# shell-quoting round trip. node owns the config file, so append
|
||||
# as node to preserve ownership/mode.
|
||||
payload = base64.b64encode(block.encode()).decode()
|
||||
script = (
|
||||
f"printf %s {shlex.quote(payload)} | base64 -d "
|
||||
f">> {shlex.quote(config_path)}"
|
||||
r = bottle.exec(
|
||||
f"{shlex.quote(_CODEX_CLI)} mcp add {_SUPERVISE_MCP_NAME} --url "
|
||||
f"{shlex.quote(supervise_url)}",
|
||||
user="node",
|
||||
)
|
||||
r = bottle.exec(script, user="node")
|
||||
if r.returncode != 0:
|
||||
die(
|
||||
"agent provider provisioning: could not register supervise "
|
||||
f"MCP server in {config_path}: "
|
||||
f"{(r.stderr or r.stdout or '').strip()}"
|
||||
warn(
|
||||
f"`codex mcp add supervise` failed (exit {r.returncode}): "
|
||||
f"{(r.stderr or r.stdout or '').strip()}. Inside the bottle, "
|
||||
f"register manually with: "
|
||||
f"codex mcp add supervise --url {shlex.quote(supervise_url)}"
|
||||
)
|
||||
|
||||
def headless_prompt(self, prompt: str) -> list[str]:
|
||||
|
||||
@@ -14,20 +14,13 @@ from __future__ import annotations
|
||||
import subprocess
|
||||
|
||||
|
||||
def run_docker(
|
||||
argv: list[str], *, env: dict[str, str] | None = None,
|
||||
) -> subprocess.CompletedProcess[str]:
|
||||
def run_docker(argv: list[str]) -> subprocess.CompletedProcess[str]:
|
||||
"""Run a `docker` command, capturing stdout/stderr as text. Never raises
|
||||
on a non-zero exit — callers inspect `returncode` / `stderr` so they can
|
||||
stay fail-closed or tolerate idempotent no-ops (e.g. removing an
|
||||
already-absent container).
|
||||
|
||||
`env` sets the child process environment — used to hand a secret to a bare
|
||||
`--env NAME` flag (docker inherits its value from this process) so the
|
||||
value never lands on argv or in `docker inspect`'s recorded command line."""
|
||||
already-absent container)."""
|
||||
return subprocess.run(
|
||||
argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True,
|
||||
check=False, env=env,
|
||||
argv, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True, check=False,
|
||||
)
|
||||
|
||||
|
||||
|
||||
+25
-138
@@ -6,8 +6,6 @@ egress container."""
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
import os
|
||||
import signal
|
||||
@@ -71,38 +69,10 @@ INTROSPECT_HOST = "_egress.local"
|
||||
# → legacy per-bottle single-tenant mode (unchanged).
|
||||
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
|
||||
|
||||
# App-layer identity token. Delivered as proxy credentials
|
||||
# (`HTTPS_PROXY=http://<bottle_id>:<token>@gw`): clients honor it as part of
|
||||
# the proxy protocol without app changes, and the addon reads + strips it so
|
||||
# it never leaks upstream. The legacy `x-bot-bottle-identity` request header
|
||||
# is still stripped defensively (git-http uses that header on its own port).
|
||||
# App-layer identity token (defense-in-depth over the source-IP invariant);
|
||||
# the agent injects it, the addon strips it so it never leaks upstream.
|
||||
IDENTITY_HEADER = "x-bot-bottle-identity"
|
||||
|
||||
# Per-flow key under which `request()` stashes the resolved (Config, supervise
|
||||
# slug, env) so the later `response()` and `websocket_message()` hooks scan
|
||||
# against the *calling bottle's* policy. In the consolidated (multi-tenant)
|
||||
# gateway the static `self.config` is empty — every request's real policy comes
|
||||
# from the per-request `/resolve` — so a hook that fell back to `self.config`
|
||||
# would find no route and silently skip its DLP scan (fail-open). Resolving once
|
||||
# at the request and reusing it also avoids a `/resolve` round-trip per response
|
||||
# and per WebSocket frame.
|
||||
_FLOW_CTX_KEY = "bot_bottle_egress_ctx"
|
||||
|
||||
|
||||
def _token_from_proxy_auth(header: str) -> str:
|
||||
"""Extract the identity token (the password) from a `Proxy-Authorization:
|
||||
Basic base64(<bottle_id>:<token>)` header. Empty on any malformed value —
|
||||
the mandatory `/resolve` then fail-closes on the empty token."""
|
||||
scheme, _, encoded = header.partition(" ")
|
||||
if scheme.lower() != "basic" or not encoded:
|
||||
return ""
|
||||
try:
|
||||
decoded = base64.b64decode(encoded, validate=True).decode("utf-8")
|
||||
except (binascii.Error, ValueError, UnicodeDecodeError):
|
||||
return ""
|
||||
_, _, password = decoded.partition(":")
|
||||
return password
|
||||
|
||||
# Seconds the egress proxy holds a token-blocked request open waiting for the
|
||||
# operator's supervisor decision (PRD 0062), overridable via env.
|
||||
DEFAULT_TOKEN_ALLOW_TIMEOUT_SECONDS = 300.0
|
||||
@@ -122,10 +92,6 @@ class EgressAddon:
|
||||
# Class default so addons built via __new__ (e.g. in tests) default to
|
||||
# single-tenant; __init__ sets the instance attribute for real runs.
|
||||
_resolver: "PolicyResolver | None" = None
|
||||
# Class default so __new__-built addons have it (real runs get a fresh
|
||||
# per-instance dict in __init__; only http_connect mutates it, which the
|
||||
# request-flow tests don't exercise).
|
||||
_conn_tokens: "dict[str, str]" = {}
|
||||
|
||||
def __init__(self) -> None:
|
||||
self.routes_path = os.environ.get("EGRESS_ROUTES", DEFAULT_ROUTES_PATH)
|
||||
@@ -140,10 +106,6 @@ class EgressAddon:
|
||||
# scan. In-memory only (a restart re-prompts); mutated only from the
|
||||
# asyncio loop that runs the addon hooks, so no lock is needed.
|
||||
self._safe_tokens: dict[str, set[str]] = {}
|
||||
# Per-client-connection identity token captured from the CONNECT's
|
||||
# `Proxy-Authorization` (HTTPS tunnels don't repeat it on the bumped
|
||||
# inner requests). Keyed by client_conn.id; cleared on disconnect.
|
||||
self._conn_tokens: dict[str, str] = {}
|
||||
self._supervise_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "").strip()
|
||||
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
|
||||
self._reload(initial=True)
|
||||
@@ -233,39 +195,31 @@ class EgressAddon:
|
||||
{"Content-Type": "text/plain; charset=utf-8"},
|
||||
)
|
||||
|
||||
def _log_request(
|
||||
self, flow: http.HTTPFlow, env: "typing.Mapping[str, str]",
|
||||
) -> None:
|
||||
# `env` is the per-flow resolved overlay (process env + this bottle's
|
||||
# /resolve tokens), so the log redaction scrubs the calling bottle's
|
||||
# provisioned secrets — not just the process-level ones in os.environ.
|
||||
def _log_request(self, flow: http.HTTPFlow) -> None:
|
||||
headers = {
|
||||
k: redact_tokens(v, env=env)
|
||||
k: redact_tokens(v, env=os.environ)
|
||||
for k, v in flow.request.headers.items()
|
||||
if k.lower() != "authorization"
|
||||
}
|
||||
body = redact_tokens(flow.request.get_text(strict=False) or "", env=env)
|
||||
body = redact_tokens(flow.request.get_text(strict=False) or "", env=os.environ)
|
||||
sys.stderr.write(
|
||||
json.dumps({
|
||||
"event": "egress_request",
|
||||
"host": redact_tokens(flow.request.pretty_host, env=env),
|
||||
"host": redact_tokens(flow.request.pretty_host, env=os.environ),
|
||||
"method": flow.request.method,
|
||||
"path": redact_tokens(flow.request.path, env=env),
|
||||
"path": redact_tokens(flow.request.path, env=os.environ),
|
||||
"headers": headers,
|
||||
"body": body,
|
||||
})
|
||||
+ "\n"
|
||||
)
|
||||
|
||||
def _log_response(
|
||||
self, flow: http.HTTPFlow, env: "typing.Mapping[str, str]",
|
||||
) -> None:
|
||||
# Per-flow env overlay (see _log_request): redact this bottle's tokens.
|
||||
def _log_response(self, flow: http.HTTPFlow) -> None:
|
||||
headers = {
|
||||
k: redact_tokens(v, env=env)
|
||||
k: redact_tokens(v, env=os.environ)
|
||||
for k, v in flow.response.headers.items()
|
||||
}
|
||||
body = redact_tokens(flow.response.get_text(strict=False) or "", env=env)
|
||||
body = redact_tokens(flow.response.get_text(strict=False) or "", env=os.environ)
|
||||
sys.stderr.write(
|
||||
json.dumps({
|
||||
"event": "egress_response",
|
||||
@@ -293,72 +247,12 @@ class EgressAddon:
|
||||
return self.config, self._supervise_slug, os.environ
|
||||
conn = flow.client_conn
|
||||
client_ip = conn.peername[0] if conn and conn.peername else ""
|
||||
token = self._request_token(flow)
|
||||
token = flow.request.headers.get(IDENTITY_HEADER, "")
|
||||
flow.request.headers.pop(IDENTITY_HEADER, None)
|
||||
config, slug, tokens = resolve_client_context(self._resolver, client_ip, token)
|
||||
env = {**os.environ, **tokens} if tokens else os.environ
|
||||
return config, slug, env
|
||||
|
||||
def _stash_flow_ctx(
|
||||
self,
|
||||
flow: http.HTTPFlow,
|
||||
config: Config,
|
||||
slug: str,
|
||||
env: "typing.Mapping[str, str]",
|
||||
) -> None:
|
||||
"""Remember the per-flow context `request()` resolved, so the later
|
||||
`response()` / `websocket_message()` hooks reuse it — scanning against
|
||||
the same bottle's policy the request was decided on, with one `/resolve`
|
||||
per flow rather than one per frame."""
|
||||
meta = getattr(flow, "metadata", None)
|
||||
if isinstance(meta, dict):
|
||||
meta[_FLOW_CTX_KEY] = (config, slug, env)
|
||||
|
||||
def _flow_ctx(
|
||||
self, flow: http.HTTPFlow,
|
||||
) -> "tuple[Config, str, typing.Mapping[str, str]]":
|
||||
"""The `(Config, supervise slug, env)` `request()` resolved for this
|
||||
flow, so a later hook scans against the calling bottle's policy — not the
|
||||
empty static config the consolidated gateway carries. Falls back to the
|
||||
single-tenant static values for a flow that never passed through
|
||||
`request()` (or a flow object without metadata)."""
|
||||
meta = getattr(flow, "metadata", None)
|
||||
if isinstance(meta, dict):
|
||||
ctx = meta.get(_FLOW_CTX_KEY)
|
||||
if ctx is not None:
|
||||
return ctx
|
||||
return self.config, self._supervise_slug, os.environ
|
||||
|
||||
def _request_token(self, flow: http.HTTPFlow) -> str:
|
||||
"""The per-bottle identity token for this request, from the proxy
|
||||
credentials — the delivery mechanism (`HTTPS_PROXY=http://id:token@gw`)
|
||||
that clients honor without app changes. Plain-HTTP requests carry
|
||||
`Proxy-Authorization` directly; HTTPS bumped requests inherit the token
|
||||
captured from their tunnel's CONNECT. Read then stripped so it never
|
||||
leaks upstream (also strips the legacy header, if present)."""
|
||||
token = _token_from_proxy_auth(
|
||||
flow.request.headers.get("Proxy-Authorization", ""))
|
||||
flow.request.headers.pop("Proxy-Authorization", None)
|
||||
flow.request.headers.pop(IDENTITY_HEADER, None)
|
||||
conn = flow.client_conn
|
||||
if not token and conn is not None:
|
||||
token = self._conn_tokens.get(getattr(conn, "id", ""), "")
|
||||
return token
|
||||
|
||||
def http_connect(self, flow: http.HTTPFlow) -> None:
|
||||
"""Capture the identity token from an HTTPS tunnel's CONNECT (the inner
|
||||
bumped requests won't carry `Proxy-Authorization`), keyed by client
|
||||
connection, and strip it so it never reaches upstream."""
|
||||
token = _token_from_proxy_auth(
|
||||
flow.request.headers.get("Proxy-Authorization", ""))
|
||||
flow.request.headers.pop("Proxy-Authorization", None)
|
||||
conn = flow.client_conn
|
||||
if conn is not None and getattr(conn, "id", ""):
|
||||
self._conn_tokens[conn.id] = token
|
||||
|
||||
def client_disconnected(self, client: typing.Any) -> None:
|
||||
"""Drop the per-connection token when the client goes away."""
|
||||
self._conn_tokens.pop(getattr(client, "id", ""), None)
|
||||
|
||||
async def request(self, flow: http.HTTPFlow) -> None:
|
||||
request_path, _, query = flow.request.path.partition("?")
|
||||
|
||||
@@ -367,9 +261,6 @@ class EgressAddon:
|
||||
return
|
||||
|
||||
config, slug, env = self._resolve_flow(flow)
|
||||
# Stash for the response / websocket hooks so their DLP scans use this
|
||||
# bottle's resolved policy, not the empty static config (see _flow_ctx).
|
||||
self._stash_flow_ctx(flow, config, slug, env)
|
||||
|
||||
# DLP outbound scan BEFORE stripping auth — catches tokens the
|
||||
# agent tried to smuggle in any header, path, query param, or body.
|
||||
@@ -428,7 +319,7 @@ class EgressAddon:
|
||||
flow.request.headers["authorization"] = decision.inject_authorization
|
||||
|
||||
if config.log >= LOG_FULL:
|
||||
self._log_request(flow, env)
|
||||
self._log_request(flow)
|
||||
|
||||
def _block_dlp(self, flow: http.HTTPFlow, result: ScanResult) -> None:
|
||||
ctx = self._req_ctx(flow)
|
||||
@@ -637,17 +528,14 @@ class EgressAddon:
|
||||
await asyncio.sleep(TOKEN_ALLOW_POLL_INTERVAL_SECONDS)
|
||||
|
||||
def response(self, flow: http.HTTPFlow) -> None:
|
||||
"""DLP inbound scan on response headers and body, against the calling
|
||||
bottle's resolved config (multi-tenant) or the static config
|
||||
(single-tenant) — see `_flow_ctx`."""
|
||||
config, _slug, env = self._flow_ctx(flow)
|
||||
route = match_route(config.routes, flow.request.pretty_host)
|
||||
"""DLP inbound scan on response headers and body."""
|
||||
route = match_route(self.config.routes, flow.request.pretty_host)
|
||||
if route is None:
|
||||
return
|
||||
if flow.response is None:
|
||||
return
|
||||
if config.log >= LOG_FULL:
|
||||
self._log_response(flow, env)
|
||||
if self.config.log >= LOG_FULL:
|
||||
self._log_response(flow)
|
||||
resp_headers = {k.lower(): v for k, v in flow.response.headers.items()}
|
||||
body = flow.response.get_text(strict=False) or ""
|
||||
scan_text = build_inbound_scan_text(resp_headers, body)
|
||||
@@ -664,7 +552,7 @@ class EgressAddon:
|
||||
resp_ctx = {**resp_ctx, "context": result.context}
|
||||
if result.severity == "block":
|
||||
self._block(flow, f"egress DLP: {result.reason}", ctx=resp_ctx)
|
||||
elif result.severity == "warn" and config.log >= LOG_BLOCKS:
|
||||
elif result.severity == "warn" and self.config.log >= LOG_BLOCKS:
|
||||
sys.stderr.write(
|
||||
json.dumps({
|
||||
"event": "egress_warn",
|
||||
@@ -675,10 +563,7 @@ class EgressAddon:
|
||||
)
|
||||
|
||||
def websocket_message(self, flow: http.HTTPFlow) -> None:
|
||||
"""DLP scan on WebSocket frames, against the calling bottle's resolved
|
||||
config (see `_flow_ctx`). `request()` resolves and stashes the per-flow
|
||||
(config, slug, env) at the upgrade, so both the multi-tenant and
|
||||
single-tenant gateways scan here.
|
||||
"""DLP scan on WebSocket frames.
|
||||
|
||||
Outbound frames (from_client) are scanned for credential leakage;
|
||||
inbound frames are scanned for prompt injection. On a block the
|
||||
@@ -687,8 +572,10 @@ class EgressAddon:
|
||||
"""
|
||||
if flow.websocket is None: # type: ignore[union-attr]
|
||||
return
|
||||
config, slug, env = self._flow_ctx(flow)
|
||||
route = match_route(config.routes, flow.request.pretty_host)
|
||||
# WebSocket DLP runs against the static config only (single-tenant); in
|
||||
# the consolidated gateway self.config has no routes, so this is inert
|
||||
# until websocket routing is made source-IP-aware (a separate slice).
|
||||
route = match_route(self.config.routes, flow.request.pretty_host)
|
||||
if route is None:
|
||||
return
|
||||
message = flow.websocket.messages[-1] # type: ignore[union-attr]
|
||||
@@ -697,8 +584,8 @@ class EgressAddon:
|
||||
# A WebSocket data frame is not an HTTP request line, so CRLF is
|
||||
# not an injection vector here — scan only for credential leakage.
|
||||
result = scan_outbound(
|
||||
route, content, env,
|
||||
safe_tokens=self._safe_tokens_for(slug), crlf_text="",
|
||||
route, content, os.environ,
|
||||
safe_tokens=self._safe_tokens_for(self._supervise_slug), crlf_text="",
|
||||
)
|
||||
if result is not None and result.severity == "block":
|
||||
sys.stderr.write(f"egress DLP: {result.reason}\n")
|
||||
|
||||
@@ -13,7 +13,6 @@ import dataclasses
|
||||
from pathlib import Path
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
from .bottle_state import globalize_slug
|
||||
from .errors import MissingEnvVarError
|
||||
from .log import info
|
||||
from .manifest import ManifestBottle, ManifestGitEntry
|
||||
@@ -47,7 +46,7 @@ def _provision_dynamic_key(
|
||||
owner_repo = entry.UpstreamPath
|
||||
if owner_repo.endswith(".git"):
|
||||
owner_repo = owner_repo[:-4]
|
||||
title = f"bot-bottle:{globalize_slug(slug)}:{entry.Name}"
|
||||
title = f"bot-bottle:{slug}:{entry.Name}"
|
||||
|
||||
info(f"provisioning deploy key for git-gate.repos[{entry.Name!r}]")
|
||||
key_id, private_key_bytes = provisioner.create(owner_repo, title)
|
||||
|
||||
@@ -19,9 +19,6 @@ from .manifest import ManifestBottle, ManifestGitEntry
|
||||
# Short network alias for git-gate inside the gateway. The
|
||||
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
|
||||
GIT_GATE_HOSTNAME = "git-gate"
|
||||
# App-layer identity token header the agent's git sends to git-http and the
|
||||
# gateway validates (mirrors egress_addon / git_http_backend IDENTITY_HEADER).
|
||||
IDENTITY_HEADER = "x-bot-bottle-identity"
|
||||
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
|
||||
# git daemon (--timeout/--init-timeout), the access-hook subprocess in
|
||||
# git_http_backend, and the git http-backend CGI subprocess.
|
||||
@@ -78,7 +75,6 @@ def _gitconfig_validate_value(field: str, value: str) -> None:
|
||||
|
||||
def git_gate_render_gitconfig(
|
||||
entries: tuple[ManifestGitEntry, ...], gate_host: str, *, scheme: str = "git",
|
||||
identity_token: str = "",
|
||||
) -> str:
|
||||
"""Render the agent's ~/.gitconfig content for git-gate
|
||||
`insteadOf` rewrites. Pure host-side, no docker / VM;
|
||||
@@ -100,15 +96,6 @@ def git_gate_render_gitconfig(
|
||||
"# the upstream bidirectionally (gitleaks-scanned push;\n",
|
||||
"# fetch-from-upstream-before-every-upload-pack via access-hook).\n",
|
||||
]
|
||||
# Over the smart-HTTP transport (VM backends), attach the per-bottle
|
||||
# identity token as a request header on requests to the gate, scoped to
|
||||
# its URL so it never goes to any other remote. git-http requires it (the
|
||||
# gateway's mandatory (source_ip, token) attribution). git:// (single-tenant
|
||||
# docker) carries no header — attribution there is the network alias.
|
||||
if identity_token and scheme == "http":
|
||||
_gitconfig_validate_value("identity_token", identity_token)
|
||||
out.append(f'[http "http://{gate_host}/"]\n')
|
||||
out.append(f"\textraHeader = {IDENTITY_HEADER}: {identity_token}\n")
|
||||
for entry in entries:
|
||||
_gitconfig_validate_value(f"repos[{entry.Name!r}].url", entry.Upstream)
|
||||
out.append(f'[url "{scheme}://{gate_host}/{entry.Name}.git"]\n')
|
||||
|
||||
@@ -16,7 +16,6 @@ import secrets
|
||||
from pathlib import Path
|
||||
|
||||
from .. import log
|
||||
from ..store_manager import StoreManager
|
||||
from .broker import LaunchBroker, StubBroker
|
||||
from .control_plane import make_server
|
||||
from .docker_broker import DockerBroker
|
||||
@@ -46,11 +45,6 @@ def main(argv: list[str] | None = None) -> int:
|
||||
|
||||
registry = RegistryStore(args.db)
|
||||
registry.migrate()
|
||||
# One DB per host: the supervise queue + audit tables live in the SAME
|
||||
# SQLite file the registry owns, so the control plane is the single
|
||||
# source of truth. The in-VM supervise daemon writes here; the host
|
||||
# operator reaches it over HTTP (never a second, disconnected DB).
|
||||
StoreManager(registry.db_path).migrate()
|
||||
|
||||
# An ephemeral signing secret ties the orchestrator (signer) to its
|
||||
# broker (verifier). 'stub' records launches instead of starting
|
||||
|
||||
@@ -17,22 +17,9 @@ import urllib.error
|
||||
import urllib.request
|
||||
from dataclasses import dataclass
|
||||
|
||||
from ..paths import host_control_plane_token
|
||||
from .control_plane import CONTROL_AUTH_HEADER
|
||||
|
||||
DEFAULT_TIMEOUT_SECONDS = 5.0
|
||||
|
||||
|
||||
def _host_auth_token() -> str:
|
||||
"""The per-host control-plane secret, or "" if it can't be read. "" means
|
||||
'send no auth header' — correct against an open (unconfigured) control
|
||||
plane, and harmlessly rejected by a secured one."""
|
||||
try:
|
||||
return host_control_plane_token()
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
|
||||
class OrchestratorClientError(RuntimeError):
|
||||
"""A control-plane call failed (unreachable, or an unexpected status)."""
|
||||
|
||||
@@ -47,24 +34,11 @@ class RegisteredBottle:
|
||||
|
||||
|
||||
class OrchestratorClient:
|
||||
"""Trusted host-side client for the orchestrator control plane.
|
||||
"""Trusted host-side client for the orchestrator control plane."""
|
||||
|
||||
Presents the per-host control-plane secret on every call (the header the
|
||||
control plane requires on all routes but `/health`). The secret is read
|
||||
from the host file — this client only ever runs host-side (CLI, launcher,
|
||||
discovery), so it can read what an agent can't. `auth_token` is overridable
|
||||
for tests; the default reads the host file, minting it on first use."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_url: str,
|
||||
*,
|
||||
timeout: float = DEFAULT_TIMEOUT_SECONDS,
|
||||
auth_token: str | None = None,
|
||||
) -> None:
|
||||
def __init__(self, base_url: str, *, timeout: float = DEFAULT_TIMEOUT_SECONDS) -> None:
|
||||
self._base = base_url.rstrip("/")
|
||||
self._timeout = timeout
|
||||
self._auth_token = auth_token if auth_token is not None else _host_auth_token()
|
||||
|
||||
def _request(
|
||||
self, method: str, path: str, body: dict[str, object] | None = None,
|
||||
@@ -75,8 +49,6 @@ class OrchestratorClient:
|
||||
callers can treat 404 as a meaningful "no such bottle"."""
|
||||
data = json.dumps(body).encode() if body is not None else None
|
||||
headers = {"Content-Type": "application/json"} if data is not None else {}
|
||||
if self._auth_token:
|
||||
headers[CONTROL_AUTH_HEADER] = self._auth_token
|
||||
req = urllib.request.Request(
|
||||
f"{self._base}{path}", data=data, method=method, headers=headers,
|
||||
)
|
||||
@@ -163,78 +135,10 @@ class OrchestratorClient:
|
||||
bottles = payload.get("bottles")
|
||||
return bottles if isinstance(bottles, list) else []
|
||||
|
||||
# --- supervise queue (operator TUI) ------------------------------------
|
||||
|
||||
def supervise_pending(self) -> list[dict[str, object]]:
|
||||
"""Pending supervise proposals across all bottles
|
||||
(`GET /supervise/proposals`)."""
|
||||
payload = self._ok("GET", "/supervise/proposals")
|
||||
proposals = payload.get("proposals")
|
||||
return proposals if isinstance(proposals, list) else []
|
||||
|
||||
def supervise_respond(
|
||||
self,
|
||||
proposal_id: str,
|
||||
*,
|
||||
bottle_slug: str,
|
||||
decision: str,
|
||||
notes: str = "",
|
||||
final_file: str | None = None,
|
||||
) -> None:
|
||||
"""Record an operator decision (`POST /supervise/respond`). `decision`
|
||||
is approve/modify/reject. Raises `OrchestratorClientError` if the
|
||||
proposal is gone or the bottle can no longer be applied to (409)."""
|
||||
body: dict[str, object] = {
|
||||
"proposal_id": proposal_id,
|
||||
"bottle_slug": bottle_slug,
|
||||
"decision": decision,
|
||||
"notes": notes,
|
||||
}
|
||||
if final_file is not None:
|
||||
body["final_file"] = final_file
|
||||
self._ok("POST", "/supervise/respond", body)
|
||||
|
||||
|
||||
def discover_orchestrator_url(*, timeout: float = 2.0) -> str:
|
||||
"""The URL of the one running per-host orchestrator control plane, probing
|
||||
the backends' well-known control-plane addresses (both on port 8099):
|
||||
docker publishes it on loopback; the firecracker infra VM serves it on the
|
||||
orchestrator TAP. Returns the first that answers `/health`; raises if none
|
||||
do (no orchestrator up — launch a bottle first)."""
|
||||
candidates: list[str] = []
|
||||
try: # docker: loopback-published control plane
|
||||
from .lifecycle import DEFAULT_PORT as _DOCKER_PORT
|
||||
candidates.append(f"http://127.0.0.1:{_DOCKER_PORT}")
|
||||
except Exception: # noqa: BLE001 — backend optional
|
||||
candidates.append("http://127.0.0.1:8099")
|
||||
try: # firecracker: infra VM control plane on the orchestrator TAP
|
||||
from ..backend.firecracker import netpool
|
||||
from ..backend.firecracker.infra_vm import CONTROL_PLANE_PORT
|
||||
candidates.append(
|
||||
f"http://{netpool.orch_slot().guest_ip}:{CONTROL_PLANE_PORT}")
|
||||
except Exception: # noqa: BLE001 — backend optional / not firecracker
|
||||
pass
|
||||
try: # macOS: infra container control plane on its host-only address
|
||||
from ..backend.macos_container.infra import probe_control_plane_url
|
||||
url = probe_control_plane_url()
|
||||
if url:
|
||||
candidates.append(url)
|
||||
except Exception: # noqa: BLE001 — backend optional / not macOS
|
||||
pass
|
||||
for url in candidates:
|
||||
if OrchestratorClient(url, timeout=timeout).health():
|
||||
return url
|
||||
raise OrchestratorClientError(
|
||||
"no running orchestrator control plane found (tried "
|
||||
+ ", ".join(candidates)
|
||||
+ "); launch a bottle first"
|
||||
)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"OrchestratorClient",
|
||||
"OrchestratorClientError",
|
||||
"RegisteredBottle",
|
||||
"DEFAULT_TIMEOUT_SECONDS",
|
||||
"discover_orchestrator_url",
|
||||
]
|
||||
|
||||
@@ -16,10 +16,6 @@ vsock / unix-socket portability caveats):
|
||||
POST /attribute -> 200 {"bottle_id"} | 403
|
||||
POST /resolve -> 200 {"bottle_id","policy"} | 403
|
||||
body: {"source_ip","identity_token"}
|
||||
GET /supervise/proposals -> 200 {"proposals": [ <proposal>, ...]}
|
||||
POST /supervise/respond -> 200 {"responded": true} | 409 (operator)
|
||||
body: {"proposal_id","bottle_slug",
|
||||
"decision", ["notes"],["final_file"]}
|
||||
|
||||
`POST /bottles` / `DELETE` drive the full launch lifecycle: they mint (or
|
||||
tear down) the bottle in the registry AND broker the backend-native launch
|
||||
@@ -34,7 +30,6 @@ returned only once, to the caller that launches the bottle.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hmac
|
||||
import http.server
|
||||
import json
|
||||
import os
|
||||
@@ -43,20 +38,11 @@ import sys
|
||||
import typing
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from ..paths import CONTROL_PLANE_TOKEN_ENV
|
||||
from .service import Orchestrator
|
||||
|
||||
# JSON body payload type (parsed request / rendered response).
|
||||
Json = dict[str, object]
|
||||
|
||||
# The request header carrying the per-host control-plane secret. Every route
|
||||
# except `GET /health` requires it (see `dispatch`). The trusted callers hold
|
||||
# the secret (the gateway's PolicyResolver, the host CLI's OrchestratorClient);
|
||||
# an agent that can merely *reach* the port cannot present it, so it can't
|
||||
# enumerate bottles, rewrite policy, read injected upstream tokens, or approve
|
||||
# its own supervise proposals.
|
||||
CONTROL_AUTH_HEADER = "x-bot-bottle-control-auth"
|
||||
|
||||
|
||||
def _parse_json_object(body: bytes) -> Json:
|
||||
"""Parse a JSON object body. Raises ValueError for non-objects / bad JSON."""
|
||||
@@ -69,29 +55,15 @@ def _parse_json_object(body: bytes) -> Json:
|
||||
|
||||
|
||||
def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
|
||||
orch: Orchestrator, method: str, path: str, body: bytes, *, authorized: bool = True,
|
||||
orch: Orchestrator, method: str, path: str, body: bytes
|
||||
) -> tuple[int, Json]:
|
||||
"""Route one control-plane request to a (status, payload) pair. Pure —
|
||||
no I/O beyond the orchestrator — so it is fully testable without a socket.
|
||||
|
||||
`authorized` is whether the request presented the control-plane secret (or
|
||||
no secret is configured — see `ControlPlaneServer`). Every route except
|
||||
`GET /health` requires it: the source-IP + identity-token checks inside
|
||||
`/resolve` and `/attribute` authenticate the *bottle* a request is about,
|
||||
not the *caller*, so without this gate any agent that can reach the port
|
||||
could rewrite another bottle's policy, read the injected upstream tokens,
|
||||
or approve its own supervise proposals. Defaults True so unit tests of the
|
||||
routing logic don't have to thread it through."""
|
||||
no I/O beyond the orchestrator — so it is fully testable without a socket."""
|
||||
route = urlsplit(path).path.rstrip("/") or "/"
|
||||
|
||||
if method == "GET" and route == "/health":
|
||||
return 200, {"status": "ok"}
|
||||
|
||||
if not authorized:
|
||||
# Everything below is a trusted-caller operation. Deny before touching
|
||||
# the registry / broker / supervise store.
|
||||
return 401, {"error": "control-plane authentication required"}
|
||||
|
||||
if method == "GET" and route == "/gateway":
|
||||
return 200, orch.gateway_status()
|
||||
|
||||
@@ -155,44 +127,10 @@ def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
|
||||
return 403, {"error": "unattributed"}
|
||||
return 200, {"bottle_id": rec.bottle_id}
|
||||
|
||||
if method == "GET" and route == "/supervise/proposals":
|
||||
# Operator TUI: pending supervise proposals across all bottles.
|
||||
return 200, {"proposals": orch.supervise_pending()}
|
||||
|
||||
if method == "POST" and route == "/supervise/respond":
|
||||
# Operator decision: apply (approve/modify rewrites egress policy),
|
||||
# write the queued response, audit — all server-side on the one DB.
|
||||
try:
|
||||
data = _parse_json_object(body)
|
||||
except ValueError as e:
|
||||
return 400, {"error": f"invalid JSON: {e}"}
|
||||
proposal_id = data.get("proposal_id")
|
||||
bottle_slug = data.get("bottle_slug")
|
||||
decision = data.get("decision")
|
||||
if not (isinstance(proposal_id, str) and proposal_id):
|
||||
return 400, {"error": "proposal_id (string) is required"}
|
||||
if not (isinstance(bottle_slug, str) and bottle_slug):
|
||||
return 400, {"error": "bottle_slug (string) is required"}
|
||||
if not (isinstance(decision, str) and decision):
|
||||
return 400, {"error": "decision (string) is required"}
|
||||
notes = data.get("notes")
|
||||
final_file = data.get("final_file")
|
||||
ok, err = orch.supervise_respond(
|
||||
proposal_id,
|
||||
bottle_slug=bottle_slug,
|
||||
decision=decision,
|
||||
notes=notes if isinstance(notes, str) else "",
|
||||
final_file=final_file if isinstance(final_file, str) else None,
|
||||
)
|
||||
if ok:
|
||||
return 200, {"responded": True}
|
||||
return 409, {"error": err}
|
||||
|
||||
if method == "POST" and route == "/resolve":
|
||||
# The per-request lookup the multi-tenant gateway makes: returns the
|
||||
# bottle's policy. Requires a matching (source_ip, identity_token)
|
||||
# pair — a missing/empty/mismatched token fail-closes (403), no
|
||||
# source-IP-only fallback.
|
||||
# bottle's policy. identity_token is OPTIONAL — absent means resolve
|
||||
# by source IP alone (network-layer attribution).
|
||||
try:
|
||||
data = _parse_json_object(body)
|
||||
except ValueError as e:
|
||||
@@ -233,10 +171,8 @@ class Handler(http.server.BaseHTTPRequestHandler):
|
||||
assert isinstance(server, ControlPlaneServer)
|
||||
length = int(self.headers.get("Content-Length") or 0)
|
||||
body = self.rfile.read(length) if length > 0 else b""
|
||||
authorized = server.is_authorized(self.headers.get(CONTROL_AUTH_HEADER, ""))
|
||||
try:
|
||||
status, payload = dispatch(
|
||||
server.orchestrator, method, self.path, body, authorized=authorized)
|
||||
status, payload = dispatch(server.orchestrator, method, self.path, body)
|
||||
except Exception as e: # noqa: BLE001 — the control plane must stay up
|
||||
sys.stderr.write(f"orchestrator: {method} {self.path} failed: {e!r}\n")
|
||||
sys.stderr.flush()
|
||||
@@ -262,40 +198,15 @@ class Handler(http.server.BaseHTTPRequestHandler):
|
||||
|
||||
|
||||
class ControlPlaneServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
|
||||
"""Threading HTTP server that carries the orchestrator for its handlers.
|
||||
|
||||
Holds the per-host control-plane secret (from `$BOT_BOTTLE_CONTROL_PLANE_TOKEN`,
|
||||
injected by the launcher into this container only). When a secret is set,
|
||||
every route but `/health` requires it; when it is unset the server runs
|
||||
**open** and says so loudly at startup — a fail-visible fallback for tests
|
||||
and any backend that hasn't wired the secret yet (e.g. Firecracker, whose
|
||||
nft boundary already blocks agents from the control-plane port)."""
|
||||
"""Threading HTTP server that carries the orchestrator for its handlers."""
|
||||
|
||||
daemon_threads = True
|
||||
allow_reuse_address = True
|
||||
|
||||
def __init__(self, address: tuple[str, int], orchestrator: Orchestrator) -> None:
|
||||
self.orchestrator = orchestrator
|
||||
self._auth_token = os.environ.get(CONTROL_PLANE_TOKEN_ENV, "").strip()
|
||||
if not self._auth_token:
|
||||
sys.stderr.write(
|
||||
"orchestrator: WARNING — no control-plane secret "
|
||||
f"(${CONTROL_PLANE_TOKEN_ENV}); running WITHOUT caller "
|
||||
"authentication. Any client that can reach this port can drive "
|
||||
"it. Backends that put the control plane on an agent-reachable "
|
||||
"network MUST set this.\n"
|
||||
)
|
||||
sys.stderr.flush()
|
||||
super().__init__(address, Handler)
|
||||
|
||||
def is_authorized(self, presented: str) -> bool:
|
||||
"""True iff the request may proceed past `/health`: either no secret is
|
||||
configured (open mode) or the presented header matches it. Constant-time
|
||||
compare so a wrong token leaks nothing timing-wise."""
|
||||
if not self._auth_token:
|
||||
return True
|
||||
return hmac.compare_digest(presented, self._auth_token)
|
||||
|
||||
|
||||
def make_server(
|
||||
orchestrator: Orchestrator, host: str = "127.0.0.1", port: int = 0
|
||||
@@ -305,7 +216,4 @@ def make_server(
|
||||
return ControlPlaneServer((host, port), orchestrator)
|
||||
|
||||
|
||||
__all__ = [
|
||||
"dispatch", "Handler", "ControlPlaneServer", "make_server", "Json",
|
||||
"CONTROL_AUTH_HEADER",
|
||||
]
|
||||
__all__ = ["dispatch", "Handler", "ControlPlaneServer", "make_server", "Json"]
|
||||
|
||||
@@ -23,17 +23,6 @@ import time
|
||||
from pathlib import Path
|
||||
|
||||
from ..docker_cmd import run_docker
|
||||
from ..paths import (
|
||||
CONTROL_PLANE_TOKEN_ENV,
|
||||
host_control_plane_token,
|
||||
host_db_path,
|
||||
)
|
||||
from ..supervise import DB_PATH_IN_CONTAINER
|
||||
|
||||
# The host DB dir is bind-mounted here so the gateway's supervise daemon
|
||||
# writes its queued proposals into the ONE host DB (the same file the
|
||||
# orchestrator container opens and the operator reaches over HTTP).
|
||||
_SUPERVISE_DB_DIR_IN_CONTAINER = os.path.dirname(DB_PATH_IN_CONTAINER)
|
||||
|
||||
# The gateway's mitmproxy writes its CA a beat after the container starts, so
|
||||
# reads poll for it rather than assuming it's there on a fresh launch.
|
||||
@@ -65,14 +54,6 @@ GATEWAY_DOCKERFILE = "Dockerfile.gateway"
|
||||
_REPO_ROOT = Path(__file__).resolve().parents[2]
|
||||
|
||||
|
||||
def _host_db_dir() -> str:
|
||||
"""The host DB directory (created if missing), for the gateway's
|
||||
supervise-DB bind-mount."""
|
||||
db_dir = host_db_path().parent
|
||||
db_dir.mkdir(parents=True, exist_ok=True)
|
||||
return str(db_dir)
|
||||
|
||||
|
||||
class GatewayError(Exception):
|
||||
"""The shared gateway failed to build/start/stop (non-zero `docker` exit)."""
|
||||
|
||||
@@ -211,27 +192,15 @@ class DockerGateway(Gateway):
|
||||
# Persist the self-generated CA so it survives restarts (agents
|
||||
# trust it) — see GATEWAY_CA_VOLUME.
|
||||
"--volume", f"{GATEWAY_CA_VOLUME}:{MITMPROXY_HOME}",
|
||||
# Share the one host DB: the supervise daemon queues proposals
|
||||
# into the same file the orchestrator (and the operator, over
|
||||
# HTTP) reads — no second, disconnected DB in the container.
|
||||
"--volume", f"{_host_db_dir()}:{_SUPERVISE_DB_DIR_IN_CONTAINER}",
|
||||
"--env", f"SUPERVISE_DB_PATH={DB_PATH_IN_CONTAINER}",
|
||||
]
|
||||
for port in self._host_port_bindings:
|
||||
argv += ["--publish", f"0.0.0.0:{port}:{port}"]
|
||||
run_env = dict(os.environ)
|
||||
if self._orchestrator_url:
|
||||
# Makes the gateway's egress / git / supervise daemons multi-tenant:
|
||||
# each request resolves source-IP -> policy against the control plane.
|
||||
argv += ["--env", f"BOT_BOTTLE_ORCHESTRATOR_URL={self._orchestrator_url}"]
|
||||
# ...and presents the control-plane secret on those /resolve calls
|
||||
# (the control plane requires it). Bare `--env NAME` keeps the value
|
||||
# off argv / `docker inspect`; only the gateway (not the agent) is
|
||||
# given it. Only needed in multi-tenant mode, where /resolve is used.
|
||||
argv += ["--env", CONTROL_PLANE_TOKEN_ENV]
|
||||
run_env[CONTROL_PLANE_TOKEN_ENV] = host_control_plane_token()
|
||||
argv.append(self.image_ref)
|
||||
proc = run_docker(argv, env=run_env)
|
||||
proc = run_docker(argv)
|
||||
if proc.returncode != 0:
|
||||
raise GatewayError(f"gateway failed to start: {proc.stderr.strip()}")
|
||||
|
||||
|
||||
@@ -25,8 +25,8 @@ from pathlib import Path
|
||||
|
||||
from .. import log
|
||||
from ..docker_cmd import run_docker
|
||||
from ..paths import CONTROL_PLANE_TOKEN_ENV, bot_bottle_root, host_control_plane_token
|
||||
from .gateway import GATEWAY_IMAGE, GATEWAY_NAME, GATEWAY_NETWORK, DockerGateway, GatewayError
|
||||
from ..paths import bot_bottle_root
|
||||
from .gateway import GATEWAY_IMAGE, GATEWAY_NETWORK, DockerGateway, GatewayError
|
||||
|
||||
DEFAULT_PORT = 8099
|
||||
ORCHESTRATOR_NAME = "bot-bottle-orchestrator"
|
||||
@@ -41,7 +41,7 @@ ORCHESTRATOR_IMAGE = os.environ.get(
|
||||
ORCHESTRATOR_DOCKERFILE = "Dockerfile.orchestrator"
|
||||
# Baked onto the container as a label so `ensure_running` can tell whether the
|
||||
# running process is executing the *current* bind-mounted source — see
|
||||
# `source_hash`.
|
||||
# `_source_hash`.
|
||||
ORCHESTRATOR_SOURCE_HASH_LABEL = "bot-bottle-orchestrator-source-hash"
|
||||
|
||||
# The repo root is bind-mounted into the control-plane container so
|
||||
@@ -60,7 +60,7 @@ class OrchestratorStartError(RuntimeError):
|
||||
"""The orchestrator container did not become healthy within the timeout."""
|
||||
|
||||
|
||||
def source_hash(repo_root: Path) -> str:
|
||||
def _source_hash(repo_root: Path) -> str:
|
||||
"""Content hash of the orchestrator's bind-mounted Python source (the
|
||||
`bot_bottle` package the control-plane process imports). This only
|
||||
changes when the code that would actually run inside the container
|
||||
@@ -83,11 +83,8 @@ class OrchestratorService:
|
||||
`orchestrator_name` / `orchestrator_label` let backends run independent
|
||||
orchestrators on the same host without name collisions (e.g. the
|
||||
Firecracker backend uses `bot-bottle-fc-orchestrator` alongside the Docker
|
||||
backend's `bot-bottle-orchestrator`); `gateway_name` gives the paired
|
||||
gateway container the same treatment (e.g. isolated integration tests
|
||||
that can't share the production `GATEWAY_NAME` singleton). Subclass and
|
||||
override `_gateway()` for anything `_gateway_image`/`gateway_name` can't
|
||||
express (a genuinely backend-specific gateway variant)."""
|
||||
backend's `bot-bottle-orchestrator`). Subclass and override `_gateway()`
|
||||
to supply a backend-specific gateway variant."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
@@ -96,7 +93,6 @@ class OrchestratorService:
|
||||
network: str = GATEWAY_NETWORK,
|
||||
image: str = ORCHESTRATOR_IMAGE,
|
||||
gateway_image: str = GATEWAY_IMAGE,
|
||||
gateway_name: str = GATEWAY_NAME,
|
||||
repo_root: Path = _REPO_ROOT,
|
||||
host_root: Path | None = None,
|
||||
orchestrator_name: str = ORCHESTRATOR_NAME,
|
||||
@@ -110,7 +106,6 @@ class OrchestratorService:
|
||||
# were one conflated image before the split.
|
||||
self.image = image
|
||||
self._gateway_image = gateway_image
|
||||
self._gateway_name = gateway_name
|
||||
self._repo_root = repo_root
|
||||
self._host_root = host_root or bot_bottle_root()
|
||||
self._orchestrator_name = orchestrator_name
|
||||
@@ -139,24 +134,20 @@ class OrchestratorService:
|
||||
proc = run_docker(["docker", "ps", "--filter", f"name=^/{name}$", "--format", "{{.Names}}"])
|
||||
return name in proc.stdout.split()
|
||||
|
||||
def _run_orchestrator_container(self, current_hash: str) -> None:
|
||||
def _run_orchestrator_container(self, source_hash: str) -> None:
|
||||
"""Start the control-plane container (idempotent: clears a stale
|
||||
fixed-name container first). Register-only broker → no docker socket.
|
||||
Labels the container with `current_hash` so a later `ensure_running`
|
||||
can detect a real code change (see `source_hash`)."""
|
||||
Labels the container with `source_hash` so a later `ensure_running`
|
||||
can detect a real code change (see `_source_hash`)."""
|
||||
run_docker(["docker", "rm", "--force", self._orchestrator_name])
|
||||
proc = run_docker([
|
||||
"docker", "run", "--detach",
|
||||
"--name", self._orchestrator_name,
|
||||
"--label", self._orchestrator_label,
|
||||
"--label", f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={current_hash}",
|
||||
"--label", f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={source_hash}",
|
||||
"--network", self.network,
|
||||
# Host CLI reaches the control plane here; bound to loopback so it
|
||||
# is not exposed on the host's external interfaces. NOTE: the
|
||||
# container is still on `self.network` (the shared gateway network),
|
||||
# so agents can reach it by container IP — which is exactly why the
|
||||
# control plane requires the secret below rather than trusting the
|
||||
# network boundary.
|
||||
# is not exposed on the host's external interfaces.
|
||||
"--publish", f"127.0.0.1:{self.port}:{self.port}",
|
||||
"--volume", f"{self._repo_root}:{_APP_DIR}:ro",
|
||||
"--workdir", _APP_DIR,
|
||||
@@ -164,15 +155,11 @@ class OrchestratorService:
|
||||
# orchestrator opens bot-bottle.db).
|
||||
"--volume", f"{self._host_root}:{_ROOT_IN_CONTAINER}",
|
||||
"--env", f"BOT_BOTTLE_ROOT={_ROOT_IN_CONTAINER}",
|
||||
# The control-plane secret it requires on every route but /health.
|
||||
# Bare `--env NAME` → docker inherits the value from the run env
|
||||
# below, so the secret never lands on argv / `docker inspect`.
|
||||
"--env", CONTROL_PLANE_TOKEN_ENV,
|
||||
"--entrypoint", "python3",
|
||||
self.image,
|
||||
"-m", "bot_bottle.orchestrator",
|
||||
"--host", "0.0.0.0", "--port", str(self.port), "--broker", "stub",
|
||||
], env={**os.environ, CONTROL_PLANE_TOKEN_ENV: host_control_plane_token()})
|
||||
])
|
||||
if proc.returncode != 0:
|
||||
raise OrchestratorStartError(
|
||||
f"orchestrator container failed to start: {proc.stderr.strip()}"
|
||||
@@ -180,10 +167,7 @@ class OrchestratorService:
|
||||
|
||||
def _gateway(self) -> DockerGateway:
|
||||
return DockerGateway(
|
||||
self._gateway_image,
|
||||
name=self._gateway_name,
|
||||
network=self.network,
|
||||
orchestrator_url=self.internal_url,
|
||||
self._gateway_image, network=self.network, orchestrator_url=self.internal_url
|
||||
)
|
||||
|
||||
def _ensure_orchestrator_image(self) -> None:
|
||||
@@ -240,7 +224,7 @@ class OrchestratorService:
|
||||
# launch (the prior behaviour) would drop every other active
|
||||
# bottle's in-memory egress tokens each time a new bottle starts,
|
||||
# since the orchestrator process holds them only in memory (#381).
|
||||
current_hash = source_hash(self._repo_root)
|
||||
current_hash = _source_hash(self._repo_root)
|
||||
if self.is_healthy() and self._orchestrator_source_current(current_hash):
|
||||
return self.url
|
||||
|
||||
|
||||
@@ -17,37 +17,9 @@ Launch lifecycle:
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from .broker import LaunchBroker, LaunchRequest, sign_request
|
||||
from .registry import BottleRecord, RegistryStore
|
||||
from .gateway import Gateway
|
||||
from ..supervise import (
|
||||
AuditEntry,
|
||||
COMPONENT_FOR_TOOL,
|
||||
Response,
|
||||
STATUS_APPROVED,
|
||||
STATUS_MODIFIED,
|
||||
STATUS_REJECTED,
|
||||
TOOL_EGRESS_ALLOW,
|
||||
TOOL_EGRESS_BLOCK,
|
||||
list_all_pending_proposals,
|
||||
read_proposal,
|
||||
render_diff,
|
||||
write_audit_entry,
|
||||
write_response,
|
||||
)
|
||||
|
||||
|
||||
# Operator decision → Response.status. The apply half (egress tools) runs
|
||||
# for approve/modify only.
|
||||
_RESPOND_STATUS = {
|
||||
"approve": STATUS_APPROVED,
|
||||
"modify": STATUS_MODIFIED,
|
||||
"reject": STATUS_REJECTED,
|
||||
}
|
||||
_APPLY_TOOLS = (TOOL_EGRESS_ALLOW, TOOL_EGRESS_BLOCK)
|
||||
|
||||
|
||||
class Orchestrator:
|
||||
@@ -127,134 +99,22 @@ class Orchestrator:
|
||||
"""Fail-closed attribution (delegates to the registry)."""
|
||||
return self.registry.attribute(source_ip, identity_token)
|
||||
|
||||
def resolve(self, source_ip: str, identity_token: str) -> BottleRecord | None:
|
||||
"""Resolve the bottle behind a request — the per-request lookup the
|
||||
multi-tenant gateway makes; the returned record carries its `policy`.
|
||||
|
||||
**Mandatory pair**: requires a matching `(source_ip, identity_token)`
|
||||
(constant-time). There is no source-IP-only fallback — the app-layer
|
||||
token is delivered on every attributed data plane (egress proxy
|
||||
credentials, git-gate/supervise headers), so a missing or mismatched
|
||||
token fail-closes. This keeps a spoofed source IP (which the /31 TAP
|
||||
alone does not prevent) from selecting another bottle's policy/tokens
|
||||
without also holding that bottle's unguessable token."""
|
||||
return self.registry.attribute(source_ip, identity_token)
|
||||
def resolve(self, source_ip: str, identity_token: str = "") -> BottleRecord | None:
|
||||
"""Resolve the bottle behind a request — the source-IP-keyed lookup
|
||||
the multi-tenant gateway makes per request; the returned record
|
||||
carries its `policy`. With a token, full attribution (source IP +
|
||||
token); without, network-layer attribution by source IP alone
|
||||
(valid where the IP is unspoofable and the control plane is
|
||||
gateway-only)."""
|
||||
if identity_token:
|
||||
return self.registry.attribute(source_ip, identity_token)
|
||||
return self.registry.by_source_ip(source_ip)
|
||||
|
||||
def set_policy(self, bottle_id: str, policy: str) -> bool:
|
||||
"""Update a bottle's gateway policy in place (live reload). False if
|
||||
the bottle is unknown."""
|
||||
return self.registry.set_policy(bottle_id, policy)
|
||||
|
||||
# --- supervise queue (operator approvals) ------------------------------
|
||||
#
|
||||
# The orchestrator owns the single DB *and* the live policy, so operator
|
||||
# decisions are applied here, server-side, and reached over HTTP by the
|
||||
# host TUI (no direct-DB access, one path for every backend).
|
||||
|
||||
def supervise_pending(self) -> list[dict[str, object]]:
|
||||
"""All pending proposals across bottles, FIFO, as JSON dicts
|
||||
(`Proposal.to_dict`, round-trippable via `Proposal.from_dict`).
|
||||
|
||||
Each dict carries an extra `bottle_label`: the bottle's human slug
|
||||
resolved from the registry (the proposal itself is keyed by the
|
||||
orchestrator-assigned bottle_id, which is opaque to an operator). The
|
||||
CLI renders the label but still responds against `bottle_slug`."""
|
||||
out: list[dict[str, object]] = []
|
||||
for p in list_all_pending_proposals():
|
||||
d = p.to_dict()
|
||||
d["bottle_label"] = self._label_for(p.bottle_slug)
|
||||
out.append(d)
|
||||
return out
|
||||
|
||||
def _label_for(self, bottle_slug: str) -> str:
|
||||
"""The human slug recorded in registry metadata for a proposal's
|
||||
bottle, or the bottle_slug unchanged when the bottle is gone or has no
|
||||
recorded slug — so the label is always non-empty."""
|
||||
rec = self.registry.get(bottle_slug)
|
||||
if rec is None:
|
||||
return bottle_slug
|
||||
try:
|
||||
meta = json.loads(rec.metadata) if rec.metadata else {}
|
||||
except ValueError:
|
||||
meta = {}
|
||||
slug = meta.get("slug") if isinstance(meta, dict) else None
|
||||
return slug if isinstance(slug, str) and slug else bottle_slug
|
||||
|
||||
def _record_for_slug(self, slug: str) -> BottleRecord | None:
|
||||
"""The live registry record for a proposal's bottle, or None (e.g. the
|
||||
bottle was torn down before the operator responded).
|
||||
|
||||
In consolidated mode the supervise server attributes each proposal to
|
||||
the orchestrator-assigned bottle_id and stores that as the proposal's
|
||||
`bottle_slug` (see supervise_server `_attributed_config`), so the fast
|
||||
path is a direct bottle_id lookup. The metadata-slug scan is the
|
||||
fallback for legacy single-tenant proposals keyed by the human slug."""
|
||||
rec = self.registry.get(slug)
|
||||
if rec is not None:
|
||||
return rec
|
||||
for rec in self.registry.all():
|
||||
try:
|
||||
meta = json.loads(rec.metadata) if rec.metadata else {}
|
||||
except ValueError:
|
||||
meta = {}
|
||||
if isinstance(meta, dict) and meta.get("slug") == slug:
|
||||
return rec
|
||||
return None
|
||||
|
||||
def supervise_respond(
|
||||
self,
|
||||
proposal_id: str,
|
||||
*,
|
||||
bottle_slug: str,
|
||||
decision: str,
|
||||
notes: str = "",
|
||||
final_file: str | None = None,
|
||||
) -> tuple[bool, str]:
|
||||
"""Record an operator decision on a queued proposal, applying it
|
||||
server-side. `decision` is approve/modify/reject.
|
||||
|
||||
Approve/modify on an egress tool rewrites the bottle's policy so the
|
||||
gateway serves the new routes on its next `/resolve` (the live apply);
|
||||
then the queued Response is written (unblocking the agent's MCP call)
|
||||
and an audit entry recorded — all against the one DB. Returns
|
||||
(ok, error): ok=False with a message when the proposal or decision is
|
||||
unknown, or the bottle is gone so an approval can't be applied."""
|
||||
status = _RESPOND_STATUS.get(decision)
|
||||
if status is None:
|
||||
return False, f"unknown decision {decision!r}"
|
||||
try:
|
||||
proposal = read_proposal(bottle_slug, proposal_id)
|
||||
except FileNotFoundError:
|
||||
return False, "no such proposal"
|
||||
|
||||
diff_before, diff_after = "", ""
|
||||
if status in (STATUS_APPROVED, STATUS_MODIFIED) and proposal.tool in _APPLY_TOOLS:
|
||||
new_policy = final_file if final_file is not None else proposal.proposed_file
|
||||
rec = self._record_for_slug(bottle_slug)
|
||||
if rec is None:
|
||||
return False, (
|
||||
f"bottle {bottle_slug!r} is no longer registered; "
|
||||
"cannot apply the route change"
|
||||
)
|
||||
diff_before, diff_after = rec.policy, new_policy
|
||||
self.set_policy(rec.bottle_id, new_policy)
|
||||
|
||||
write_response(bottle_slug, Response(
|
||||
proposal_id=proposal_id, status=status, notes=notes, final_file=final_file,
|
||||
))
|
||||
component = COMPONENT_FOR_TOOL.get(proposal.tool)
|
||||
if component is not None:
|
||||
write_audit_entry(AuditEntry(
|
||||
timestamp=datetime.now(timezone.utc).isoformat(),
|
||||
bottle_slug=bottle_slug,
|
||||
component=component,
|
||||
operator_action=status,
|
||||
operator_notes=notes,
|
||||
justification=proposal.justification,
|
||||
diff=render_diff(diff_before, diff_after, label=component),
|
||||
))
|
||||
return True, ""
|
||||
|
||||
# --- consolidated gateway ----------------------------------------------
|
||||
|
||||
def ensure_gateway(self) -> None:
|
||||
|
||||
+1
-59
@@ -16,8 +16,6 @@ layer (and to COPY flat into the gateway).
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import secrets
|
||||
import stat
|
||||
from pathlib import Path
|
||||
|
||||
# The single shared host state DB. All bot-bottle SQLite stores (supervise
|
||||
@@ -25,14 +23,6 @@ from pathlib import Path
|
||||
# TableMigrations schema_key namespaces each store's tables.
|
||||
HOST_DB_FILENAME = "bot-bottle.db"
|
||||
|
||||
# The per-host control-plane secret file, and the env var the launchers inject
|
||||
# its value into. The control plane requires this secret on every mutating /
|
||||
# reading route (see orchestrator/control_plane.py); it is held only by the
|
||||
# trusted callers (control plane, gateway, host CLI) and never handed to an
|
||||
# agent, so an agent that can reach the control-plane port still can't drive it.
|
||||
CONTROL_PLANE_TOKEN_FILENAME = "control-plane-token"
|
||||
CONTROL_PLANE_TOKEN_ENV = "BOT_BOTTLE_CONTROL_PLANE_TOKEN"
|
||||
|
||||
|
||||
def bot_bottle_root() -> Path:
|
||||
"""The app data root — `$BOT_BOTTLE_ROOT` if set, else `~/.bot-bottle`."""
|
||||
@@ -50,52 +40,4 @@ def host_db_path() -> Path:
|
||||
return bot_bottle_root() / "db" / HOST_DB_FILENAME
|
||||
|
||||
|
||||
def host_db_dir() -> Path:
|
||||
"""The directory holding the shared host state DB, created if missing.
|
||||
Backends bind-mount this into their gateway so the supervise daemon writes
|
||||
to the one DB the orchestrator (and the operator over HTTP) reads."""
|
||||
db_dir = host_db_path().parent
|
||||
db_dir.mkdir(parents=True, exist_ok=True)
|
||||
return db_dir
|
||||
|
||||
|
||||
def host_control_plane_token() -> str:
|
||||
"""The per-host control-plane secret, minted (256-bit, url-safe) and
|
||||
persisted 0600 on first use, then reused.
|
||||
|
||||
This is the shared secret the launchers inject into the control-plane and
|
||||
gateway containers and that the host CLI presents on every call. It is a
|
||||
*host* artifact — the file lives under the root the agent never mounts, and
|
||||
the env var is set only on the trusted containers — so reading it here is
|
||||
safe on the host launch path but the value never reaches a bottle."""
|
||||
path = bot_bottle_root() / CONTROL_PLANE_TOKEN_FILENAME
|
||||
try:
|
||||
existing = path.read_text().strip()
|
||||
if existing:
|
||||
return existing
|
||||
except OSError:
|
||||
pass
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
token = secrets.token_urlsafe(32)
|
||||
# Create 0600 up front (O_EXCL loses a concurrent race harmlessly — we
|
||||
# re-read the winner's token below) so the secret is never briefly world-
|
||||
# readable between write and chmod.
|
||||
try:
|
||||
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
|
||||
except FileExistsError:
|
||||
return path.read_text().strip()
|
||||
with os.fdopen(fd, "w") as f:
|
||||
f.write(token)
|
||||
os.chmod(path, stat.S_IRUSR | stat.S_IWUSR)
|
||||
return token
|
||||
|
||||
|
||||
__all__ = [
|
||||
"HOST_DB_FILENAME",
|
||||
"CONTROL_PLANE_TOKEN_FILENAME",
|
||||
"CONTROL_PLANE_TOKEN_ENV",
|
||||
"bot_bottle_root",
|
||||
"host_db_path",
|
||||
"host_db_dir",
|
||||
"host_control_plane_token",
|
||||
]
|
||||
__all__ = ["HOST_DB_FILENAME", "bot_bottle_root", "host_db_path"]
|
||||
|
||||
@@ -29,29 +29,11 @@ the gateway.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
DEFAULT_TIMEOUT_SECONDS = 2.0
|
||||
|
||||
# The control-plane secret this gateway presents on every /resolve call, read
|
||||
# from the env the launcher injects into the gateway container. The control
|
||||
# plane requires it (orchestrator/control_plane.py). Constant + env-var name are
|
||||
# duplicated here rather than imported because this module is COPYed flat into
|
||||
# the gateway image, free of bot-bottle imports — same rationale as
|
||||
# IDENTITY_HEADER in egress_addon / git_http_backend.
|
||||
CONTROL_AUTH_HEADER = "x-bot-bottle-control-auth"
|
||||
CONTROL_PLANE_TOKEN_ENV = "BOT_BOTTLE_CONTROL_PLANE_TOKEN"
|
||||
|
||||
|
||||
def _control_auth_headers() -> dict[str, str]:
|
||||
"""The auth header to send, or {} when no secret is configured (an open
|
||||
control plane, e.g. Firecracker behind its nft boundary — sending nothing
|
||||
is correct there and harmlessly ignored)."""
|
||||
token = os.environ.get(CONTROL_PLANE_TOKEN_ENV, "").strip()
|
||||
return {CONTROL_AUTH_HEADER: token} if token else {}
|
||||
|
||||
|
||||
class PolicyResolveError(RuntimeError):
|
||||
"""The orchestrator was unreachable or returned an unexpected status —
|
||||
@@ -75,7 +57,7 @@ class PolicyResolver:
|
||||
).encode()
|
||||
req = urllib.request.Request(
|
||||
f"{self._base}/resolve", data=body, method="POST",
|
||||
headers={"Content-Type": "application/json", **_control_auth_headers()},
|
||||
headers={"Content-Type": "application/json"},
|
||||
)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=self._timeout) as resp:
|
||||
|
||||
@@ -51,16 +51,12 @@ from dataclasses import dataclass, replace
|
||||
try:
|
||||
# Same-directory imports inside the bundle container; these files are
|
||||
# COPYed flat under /app by Dockerfile.gateway.
|
||||
from egress_addon_core import (
|
||||
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
|
||||
)
|
||||
from egress_addon_core import LOG_OFF, load_config
|
||||
from policy_resolver import PolicyResolveError, PolicyResolver
|
||||
import supervise as _sv
|
||||
except ModuleNotFoundError:
|
||||
# Package imports for host-side tests and tooling.
|
||||
from .egress_addon_core import (
|
||||
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
|
||||
)
|
||||
from .egress_addon_core import LOG_OFF, load_config
|
||||
from .policy_resolver import PolicyResolveError, PolicyResolver
|
||||
from . import supervise as _sv
|
||||
|
||||
@@ -69,8 +65,6 @@ except ModuleNotFoundError:
|
||||
|
||||
|
||||
MCP_PROTOCOL_VERSION = "2024-11-05"
|
||||
# App-layer identity token header (mirrors egress_addon / git_http_backend).
|
||||
IDENTITY_HEADER = "x-bot-bottle-identity"
|
||||
SERVER_NAME = "bot-bottle-supervise"
|
||||
SERVER_VERSION = "0.1.0"
|
||||
|
||||
@@ -531,17 +525,6 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
|
||||
if method == "tools/list":
|
||||
return handle_tools_list(req.params)
|
||||
if method == "tools/call":
|
||||
# `list-egress-routes` is read-only introspection. In consolidated
|
||||
# mode the gateway's *static* route table is empty (routes are
|
||||
# resolved per request by source IP), so answer it from the calling
|
||||
# bottle's resolved policy. Otherwise the agent sees an empty
|
||||
# allowlist and composes an egress proposal that *replaces* the live
|
||||
# routes instead of extending them — silently dropping base routes
|
||||
# like api.anthropic.com when the operator approves it.
|
||||
if req.params.get("name") == _sv.TOOL_LIST_EGRESS_ROUTES:
|
||||
resolved = self._resolved_routes_payload()
|
||||
if resolved is not None:
|
||||
return resolved
|
||||
# Attribute the proposal to the calling bottle. Single-tenant → the
|
||||
# env slug on `config`; consolidated → the source-IP-resolved
|
||||
# bottle id, so one shared server queues each bottle's proposal
|
||||
@@ -549,28 +532,6 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
|
||||
return handle_tools_call(req.params, self._attributed_config(config))
|
||||
raise _RpcClientError(ERR_METHOD_NOT_FOUND, f"method not found: {method}")
|
||||
|
||||
def _resolved_routes_payload(self) -> dict[str, object] | None:
|
||||
"""The calling bottle's live egress routes as the `list-egress-routes`
|
||||
JSON payload, resolved by (source_ip, identity token) — the same shape
|
||||
the single-tenant introspection endpoint returns. None when there is no
|
||||
resolver (single-tenant), so the caller falls back to that endpoint.
|
||||
|
||||
Fail-closed like `_attributed_config`: an unattributed source or an
|
||||
unreachable orchestrator yields an empty route list (never another
|
||||
bottle's), courtesy of `resolve_client_context`."""
|
||||
resolver = getattr(self.server, "policy_resolver", None)
|
||||
if resolver is None:
|
||||
return None
|
||||
headers = getattr(self, "headers", None)
|
||||
token = headers.get(IDENTITY_HEADER, "") if headers is not None else ""
|
||||
conf, _slug, _tokens = resolve_client_context(
|
||||
resolver, self.client_address[0], token,
|
||||
)
|
||||
body = json.dumps(
|
||||
{"routes": [route_to_yaml_dict(r) for r in conf.routes]}, indent=2,
|
||||
)
|
||||
return {"content": [{"type": "text", "text": body}], "isError": False}
|
||||
|
||||
def _attributed_config(self, config: ServerConfig) -> ServerConfig:
|
||||
"""The ServerConfig with `bottle_slug` bound to *this request's* bottle.
|
||||
Single-tenant (no resolver): unchanged. Consolidated: the bottle id
|
||||
@@ -580,13 +541,8 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
|
||||
resolver = getattr(self.server, "policy_resolver", None)
|
||||
if resolver is None:
|
||||
return config
|
||||
# The agent's MCP client sends the identity token as a request header
|
||||
# (provisioned via `mcp add --header`); the orchestrator requires the
|
||||
# (source_ip, token) pair, so a missing/wrong token fail-closes below.
|
||||
headers = getattr(self, "headers", None)
|
||||
token = headers.get(IDENTITY_HEADER, "") if headers is not None else ""
|
||||
try:
|
||||
bottle_id = resolver.resolve_bottle_id(self.client_address[0], token)
|
||||
bottle_id = resolver.resolve_bottle_id(self.client_address[0])
|
||||
except PolicyResolveError as e:
|
||||
raise _RpcInternalError(f"orchestrator unreachable, cannot attribute: {e}") from e
|
||||
if not bottle_id:
|
||||
|
||||
@@ -8,20 +8,16 @@
|
||||
> **Superseded in part by [PRD 0070](0070-per-host-orchestrator.md) (#351):**
|
||||
> the sidecar-consolidation framing here (Stage 1, per-host sidecar; Stage 4,
|
||||
> sidecar-as-VM) is taken over by 0070's per-host orchestrator. This PRD still
|
||||
> owns the docker-free **image-provisioning** work — Stage 2 (pull the fixed
|
||||
> images from an OCI registry instead of building them with host Docker, a
|
||||
> dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
|
||||
> owns the docker-free **image-building** work — Stage 2 (nix-built fixed
|
||||
> images, a dependency of 0070) and Stage 3 (in-VM Dockerfile builder).
|
||||
|
||||
## Summary
|
||||
|
||||
Make the Firecracker backend depend on **firecracker + KVM only**, removing
|
||||
Docker from the host. Two moves get us there: run the **sidecar bundle as a
|
||||
persistent, per-host service** (eventually a Firecracker VM) instead of a
|
||||
per-bottle container, and **provision rootfs images without a host Docker
|
||||
daemon** — pull the fixed images (orchestrator/gateway/infra) from an OCI
|
||||
registry and unpack them daemonlessly, and build user Dockerfiles in an in-VM
|
||||
builder. The images are still *built* with Docker, but off the launch host
|
||||
(CI / a publish step) and pushed to the registry; the launch host only pulls.
|
||||
per-bottle container, and **build agent rootfs images without a host Docker
|
||||
daemon** (nix for the fixed images; an in-VM builder for user Dockerfiles).
|
||||
|
||||
## Motivation
|
||||
|
||||
@@ -97,51 +93,13 @@ torn down at exit."
|
||||
Can ship as a container first (quick resource/ops win) and become a VM in
|
||||
Stage 4.
|
||||
|
||||
### Stage 2 — Fixed rootfs prebuilt + pulled as an artifact (no host Docker)
|
||||
### Stage 2 — Fixed images built with nix (no Docker)
|
||||
|
||||
The one fixed image the Firecracker backend needs at launch — the combined
|
||||
**infra** rootfs the infra VM boots (orchestrator control plane + gateway +
|
||||
buildah, with the control-plane init as PID 1) — is **prebuilt end-to-end off
|
||||
the launch host and published as a versioned, ready-to-boot ext4 artifact**.
|
||||
The launch host **downloads the `.ext4` and boots it directly** — no
|
||||
`docker build`, no `docker export`, no `mke2fs`, no image tooling at all.
|
||||
|
||||
This is possible because the infra rootfs is already **host- and
|
||||
bottle-agnostic**: the per-boot bits (authorized_keys, guest IP) arrive on the
|
||||
**kernel cmdline**, not in the rootfs (see `build_base_rootfs_dir`). So one
|
||||
published ext4 boots on any launch host.
|
||||
|
||||
- **Artifact.** `rootfs.ext4` + a `rootfs.ext4.sha256`, published as a Gitea
|
||||
**generic package** (`bot-bottle-firecracker-infra/<tag>`) — generic packages take
|
||||
arbitrary large binaries (no attachment size cap / file-type allowlist that
|
||||
release attachments impose). The matching `vmlinux` kernel can ship the same
|
||||
way, so the whole VM is fetchable.
|
||||
- **Pull.** The launch host `GET`s
|
||||
`…/api/packages/<owner>/generic/bot-bottle-firecracker-infra/<tag>/rootfs.ext4` (+
|
||||
`.sha256`) for its pinned tag, verifies the checksum, caches it under the
|
||||
tag, and attaches it as the infra VM's root disk. Host prerequisite is an
|
||||
HTTP client — nothing else. Public packages need no auth to pull; a token
|
||||
with `read:package` covers a private instance.
|
||||
- **Registry.** The artifact base URL + owner are configurable, defaulting to
|
||||
this deployment's Gitea (`https://gitea.dideric.is` / `didericis`);
|
||||
overridable via env for other deployments / air-gapped mirrors.
|
||||
- **Versioning.** A pinned tag bumped when the infra rootfs contents change
|
||||
(bot_bottle's shipped files, the base deps, or the init), so a launch host
|
||||
pulls the artifact matching its code and a content change can't silently
|
||||
boot a stale rootfs. A checksum mismatch fails closed.
|
||||
- **Publish.** A `publish` step (CLI subcommand / CI job) runs the full
|
||||
pipeline **on a build/CI host** — `docker build` the three Dockerfiles →
|
||||
export → inject guest boot → `mke2fs` → upload the `.ext4` + `.sha256`.
|
||||
Building still uses Docker, but never on the launch/runner host, which is
|
||||
the one #348 needs unprivileged.
|
||||
- **Dev escape hatch.** An explicit opt-in still builds the rootfs locally
|
||||
with Docker (for iterating on the Dockerfiles without a publish
|
||||
round-trip); it is never the default path.
|
||||
|
||||
Removes Docker from the launch host entirely for the fixed image, and the
|
||||
launch host needs no OCI/rootfs tooling — just fetch + boot. The build-time
|
||||
cache / build-time-egress open problems a from-scratch build would face don't
|
||||
arise: the launch host never builds, it downloads a finished disk.
|
||||
The images bot-bottle *ships* — the sidecar, the agent base, and the builder
|
||||
(Stage 3) — are built declaratively with nix (`nixos-generators` /
|
||||
`make-ext4-fs` / `pkgs.dockerTools` for the rootfs), producing an ext4 or
|
||||
tar with correct ownership. Removes Docker for everything we own and gives
|
||||
the rootless-rootfs correctness (#347) for free on these images.
|
||||
|
||||
### Stage 3 — User Dockerfiles built in a builder VM (the unlock)
|
||||
|
||||
|
||||
@@ -358,111 +358,3 @@ the Apple Container-specific constraints directly:
|
||||
|
||||
Do not implement the backend as a direct clone of Docker Compose
|
||||
service aliases. That assumption failed in this run.
|
||||
|
||||
## Addendum: consolidated-gateway findings (2026-07-17, PRD 0070)
|
||||
|
||||
Re-tested on Apple Container 1.0.0 while porting the backend to the
|
||||
per-host consolidated gateway (#351). The two-network shape above still
|
||||
holds; these are the additional constraints that shaped the port, each
|
||||
verified against the live CLI on this host.
|
||||
|
||||
### No static IP for a container
|
||||
|
||||
`container run --network` accepts only
|
||||
`<name>[,mac=XX:XX:XX:XX:XX:XX][,mtu=VALUE]`. There is no `--ip`. The
|
||||
address comes from vmnet's DHCP and is knowable only after the container
|
||||
is running:
|
||||
|
||||
```console
|
||||
$ container run --name a --network bb-net --detach alpine sleep 900
|
||||
$ container inspect a | jq -r '.[0].status.networks[0].ipv4Address'
|
||||
192.168.128.3/24
|
||||
```
|
||||
|
||||
Consequence: the docker backend's "allocate a free IP -> pin it with
|
||||
`--ip` -> register -> launch" order cannot be reproduced. macOS inverts
|
||||
it to "launch -> read the assigned address -> register". The identity
|
||||
token therefore cannot be in the agent's run-time env (registration mints
|
||||
it after the container exists) and is delivered at `container exec` time.
|
||||
|
||||
### Networks are fixed at run time
|
||||
|
||||
There is no `container network connect`; `container network` exposes only
|
||||
`create`, `delete`, `list`, `inspect`, `prune`. A network cannot be
|
||||
attached to a running container, so a *persistent* shared gateway rules
|
||||
out per-bottle networks — they would force a gateway restart per launch.
|
||||
One shared host-only network, created up front, is the only shape that
|
||||
keeps the gateway a singleton.
|
||||
|
||||
### No container DNS
|
||||
|
||||
Containers cannot resolve each other by name; the host-only network's
|
||||
resolver refuses the query:
|
||||
|
||||
```console
|
||||
$ container exec agent nslookup gw
|
||||
;; connection timed out; no servers could be reached
|
||||
$ container exec agent cat /etc/resolv.conf
|
||||
nameserver 192.168.128.1
|
||||
```
|
||||
|
||||
Consequence: the gateway is handed the control plane's **IP**, not a
|
||||
container name as on docker. That forces the startup order
|
||||
orchestrator -> read its address -> gateway.
|
||||
|
||||
### The host can reach the host-only network directly
|
||||
|
||||
```console
|
||||
$ container inspect c | jq -r '.[0].status.networks[0].ipv4Address'
|
||||
192.168.128.2/24
|
||||
$ curl -s http://192.168.128.2:8099/i
|
||||
ok
|
||||
```
|
||||
|
||||
So no `--publish` hop is needed: the host CLI and the gateway use the
|
||||
same control-plane URL. Docker needs `--publish 127.0.0.1:...` plus a
|
||||
separate internal URL for the same job.
|
||||
|
||||
### CAP_NET_RAW is granted by default — and matters for attribution
|
||||
|
||||
Apple grants NET_RAW but not NET_ADMIN. The agent therefore cannot change
|
||||
its own address or route:
|
||||
|
||||
```console
|
||||
$ container exec agent ip addr add 192.168.128.99/24 dev eth0
|
||||
ip: RTNETLINK answers: Operation not permitted
|
||||
$ container exec agent ip route replace default via 192.168.128.2 dev eth0
|
||||
ip: RTNETLINK answers: Operation not permitted
|
||||
$ container exec agent grep CapEff /proc/self/status
|
||||
CapEff: 00000000a80425fb # bit 13 (NET_RAW) set, bit 12 (NET_ADMIN) clear
|
||||
```
|
||||
|
||||
But NET_RAW permits raw sockets, i.e. source-address forgery against
|
||||
neighbours on the shared segment — directly against PRD 0070's invariant
|
||||
("a packet's source address, as seen by the orchestrator, provably
|
||||
identifies the originating bottle"). `--cap-drop CAP_NET_RAW` closes it:
|
||||
|
||||
```console
|
||||
$ container run --cap-drop CAP_NET_RAW ... alpine
|
||||
$ container exec nr grep CapEff /proc/self/status
|
||||
CapEff: 00000000a80405fb # bit 13 cleared
|
||||
$ container exec nr ping -c1 192.168.128.2
|
||||
ping: permission denied (are you root?)
|
||||
```
|
||||
|
||||
The agent is run with `--cap-drop CAP_NET_RAW` for this reason.
|
||||
|
||||
### `container exec` inherits run-time env, and `--env` overrides it
|
||||
|
||||
```console
|
||||
$ container run --name e --env FOO=from_run --detach alpine sleep 120
|
||||
$ container exec e sh -c 'echo $FOO'
|
||||
from_run
|
||||
$ container exec --env FOO=from_exec e sh -c 'echo $FOO'
|
||||
from_exec
|
||||
```
|
||||
|
||||
This is what makes exec-time identity-token delivery work: the token-less
|
||||
proxy URL baked in at launch is superseded by the token-bearing one at
|
||||
exec. Bare `--env NAME` (inherit from the parent process) keeps the token
|
||||
value off argv.
|
||||
|
||||
@@ -49,12 +49,10 @@ let
|
||||
# /31 alignment == an even final octet (only bit 0 matters for base+2i).
|
||||
lastOctet = lib.toInt (lib.last (lib.splitString "." cfg.ipBase));
|
||||
|
||||
# The script needs ip/nft/sysctl + the usual coreutils, plus iptables
|
||||
# for the orchestrator link's DOCKER-USER accept (best-effort; skipped
|
||||
# when Docker is absent). It gets every pool value via the unit's
|
||||
# Environment=, so it never reads the shared defaults file (which isn't
|
||||
# beside it once copied to the store).
|
||||
runtimePath = with pkgs; [ iproute2 nftables iptables procps coreutils gnused ];
|
||||
# The script needs ip/nft/sysctl + the usual coreutils. It gets every
|
||||
# pool value via the unit's Environment=, so it never reads the shared
|
||||
# defaults file (which isn't beside it once copied to the store).
|
||||
runtimePath = with pkgs; [ iproute2 nftables procps coreutils gnused ];
|
||||
|
||||
ownEnv =
|
||||
if cfg.group != null
|
||||
@@ -66,7 +64,6 @@ let
|
||||
BOT_BOTTLE_FC_IP_BASE = cfg.ipBase;
|
||||
BOT_BOTTLE_FC_IFACE_PREFIX = cfg.ifacePrefix;
|
||||
BOT_BOTTLE_FC_NFT_TABLE = cfg.tableName;
|
||||
BOT_BOTTLE_FC_ORCH_IFACE = cfg.orchIface;
|
||||
} // ownEnv;
|
||||
in
|
||||
{
|
||||
@@ -130,19 +127,6 @@ in
|
||||
description = "nftables table name for the isolation boundary. Must match netpool.NFT_TABLE.";
|
||||
};
|
||||
|
||||
orchIface = lib.mkOption {
|
||||
type = lib.types.str;
|
||||
default = defaults.BOT_BOTTLE_FC_ORCH_IFACE;
|
||||
defaultText = lib.literalMD "the shared `netpool.defaults.env` value";
|
||||
description = ''
|
||||
TAP name for the orchestrator/gateway VM's dedicated link. Unlike
|
||||
the isolated bbfc* agent pool, this link is NAT'd to the internet
|
||||
(the orchestrator is trusted infra that builds agent images in-VM
|
||||
and forwards agent egress upstream). Must match
|
||||
BOT_BOTTLE_FC_ORCH_IFACE / netpool.ORCH_IFACE.
|
||||
'';
|
||||
};
|
||||
|
||||
writeEnvFile = lib.mkOption {
|
||||
type = lib.types.bool;
|
||||
default = false;
|
||||
@@ -192,7 +176,6 @@ in
|
||||
BOT_BOTTLE_FC_IP_BASE=${cfg.ipBase}
|
||||
BOT_BOTTLE_FC_IFACE_PREFIX=${cfg.ifacePrefix}
|
||||
BOT_BOTTLE_FC_NFT_TABLE=${cfg.tableName}
|
||||
BOT_BOTTLE_FC_ORCH_IFACE=${cfg.orchIface}
|
||||
'';
|
||||
};
|
||||
};
|
||||
|
||||
@@ -63,13 +63,12 @@ POOL_SIZE="${BOT_BOTTLE_FC_POOL_SIZE:-$(_default BOT_BOTTLE_FC_POOL_SIZE)}"
|
||||
IP_BASE="${BOT_BOTTLE_FC_IP_BASE:-$(_default BOT_BOTTLE_FC_IP_BASE)}"
|
||||
PREFIX="${BOT_BOTTLE_FC_IFACE_PREFIX:-$(_default BOT_BOTTLE_FC_IFACE_PREFIX)}"
|
||||
TABLE="${BOT_BOTTLE_FC_NFT_TABLE:-$(_default BOT_BOTTLE_FC_NFT_TABLE)}"
|
||||
ORCH_IFACE="${BOT_BOTTLE_FC_ORCH_IFACE:-$(_default BOT_BOTTLE_FC_ORCH_IFACE)}"
|
||||
OWNER="${BOT_BOTTLE_FC_OWNER:-${SUDO_USER:-$USER}}"
|
||||
GROUP="${BOT_BOTTLE_FC_GROUP:-}"
|
||||
|
||||
# Fail loudly rather than provisioning a half/empty range if a value
|
||||
# resolved to nothing (env unset AND the shared file unreadable).
|
||||
for _v in POOL_SIZE IP_BASE PREFIX TABLE ORCH_IFACE; do
|
||||
for _v in POOL_SIZE IP_BASE PREFIX TABLE; do
|
||||
[ -n "${!_v}" ] || { echo "error: $_v unresolved (set BOT_BOTTLE_FC_* or fix $_DEFAULTS)" >&2; exit 1; }
|
||||
done
|
||||
|
||||
@@ -90,13 +89,6 @@ host_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 )); }
|
||||
guest_ip() { _int_to_ip $(( $(_ip_to_int "$IP_BASE") + 2*$1 + 1 )); }
|
||||
iface() { echo "${PREFIX}$1"; }
|
||||
|
||||
# Orchestrator/gateway VM link: a /31 at the TOP of the IP_BASE /16
|
||||
# (host x.y.255.0, guest x.y.255.1), well clear of the agent pool near
|
||||
# the bottom of the block. Must match netpool.py:orch_slot().
|
||||
_orch_base() { echo $(( ($(_ip_to_int "$IP_BASE") & 0xFFFF0000) + 0xFF00 )); }
|
||||
orch_host() { _int_to_ip "$(_orch_base)"; }
|
||||
orch_guest() { _int_to_ip $(( $(_orch_base) + 1 )); }
|
||||
|
||||
require_root() {
|
||||
if [ "$(id -u)" -ne 0 ]; then
|
||||
echo "error: '$1' needs root (run under sudo)" >&2
|
||||
@@ -133,21 +125,8 @@ cmd_up() {
|
||||
echo " $dev host=$host guest=$(guest_ip "$i") $own_desc"
|
||||
done
|
||||
|
||||
# The orchestrator/gateway VM's dedicated link. Same rootless-open
|
||||
# ownership as the pool, but NAT'd to the internet (below) — it is
|
||||
# trusted infra, not an isolated agent slot.
|
||||
ip link show "$ORCH_IFACE" >/dev/null 2>&1 \
|
||||
|| ip tuntap add dev "$ORCH_IFACE" mode tap "${own_args[@]}"
|
||||
ip addr replace "$(orch_host)/31" dev "$ORCH_IFACE"
|
||||
ip link set "$ORCH_IFACE" up
|
||||
echo " $ORCH_IFACE host=$(orch_host) guest=$(orch_guest) (NAT'd egress) $own_desc"
|
||||
|
||||
_install_nft
|
||||
echo "nftables table inet $TABLE installed (fail-closed boundary)"
|
||||
_install_orch_egress
|
||||
echo "orchestrator egress installed ($ORCH_IFACE -> NAT out)"
|
||||
_install_gateway_route
|
||||
echo "agent->gateway route installed (${PREFIX}* :$GATEWAY_PORTS -> $(orch_guest))"
|
||||
echo "done."
|
||||
}
|
||||
|
||||
@@ -163,30 +142,12 @@ _install_nft() {
|
||||
# input: a VM never needs host-local delivery (its gateway is
|
||||
# reached via DNAT->forward), so drop all direct input from VMs
|
||||
# -> host services bound on 0.0.0.0 are unreachable from the VM.
|
||||
# Delete-first (create empty, delete, recreate) so a re-applied `up`
|
||||
# lands identical state instead of appending rules / erroring on the
|
||||
# existing base chains — the setup is idempotent regardless of history.
|
||||
|
||||
# Anti-spoof: bind each TAP to its assigned guest IP. The /31 alone
|
||||
# does NOT make the source address unspoofable — root in an agent VM
|
||||
# can source another bottle's guest IP on its own bbfc TAP, and the
|
||||
# gateway attributes egress/policy/tokens by source IP. So drop any
|
||||
# packet whose source isn't the guest address assigned to the exact
|
||||
# TAP it arrived on, before it can be attributed. One rule per slot.
|
||||
local antispoof="" i
|
||||
for i in $(seq 0 $((POOL_SIZE-1))); do
|
||||
antispoof="${antispoof} iifname \"$(iface "$i")\" ip saddr != $(guest_ip "$i") drop
|
||||
"
|
||||
done
|
||||
|
||||
nft -f - <<EOF
|
||||
table inet $TABLE {}
|
||||
delete table inet $TABLE
|
||||
table inet $TABLE {
|
||||
chain forward {
|
||||
type filter hook forward priority -10; policy accept;
|
||||
iifname != "${PREFIX}*" return
|
||||
${antispoof} ct state established,related accept
|
||||
ct state established,related accept
|
||||
ct status dnat accept
|
||||
drop
|
||||
}
|
||||
@@ -200,87 +161,8 @@ ${antispoof} ct state established,related accept
|
||||
EOF
|
||||
}
|
||||
|
||||
# Give the orchestrator/gateway VM real internet egress (agent VMs get
|
||||
# none — that's the isolation table above). Three parts, because the
|
||||
# path must work both during bootstrap (Docker still present) and after
|
||||
# Docker is removed:
|
||||
# * masquerade — SNAT the orch guest /31 out the host uplink so its
|
||||
# RFC-1918 address can reach the internet.
|
||||
# * nft forward — accept the orch link's forward path (load-bearing
|
||||
# on a pure-nft host whose FORWARD policy drops; a
|
||||
# harmless no-op where forwarding is already open).
|
||||
# It never drops, so it can't weaken the isolation
|
||||
# table's agent drops.
|
||||
# * DOCKER-USER — during bootstrap Docker's FORWARD chain policy is
|
||||
# DROP; its sanctioned DOCKER-USER hook is the only
|
||||
# place a user ACCEPT survives. Best-effort + guarded
|
||||
# (skipped once Docker is gone).
|
||||
_install_orch_egress() {
|
||||
nft -f - <<EOF
|
||||
table inet ${TABLE}_nat {}
|
||||
delete table inet ${TABLE}_nat
|
||||
table inet ${TABLE}_nat {
|
||||
chain forward {
|
||||
type filter hook forward priority -10; policy accept;
|
||||
iifname "$ORCH_IFACE" accept
|
||||
oifname "$ORCH_IFACE" ct state established,related accept
|
||||
}
|
||||
chain postrouting {
|
||||
type nat hook postrouting priority 100; policy accept;
|
||||
ip saddr $(orch_guest) oifname != "$ORCH_IFACE" masquerade
|
||||
}
|
||||
}
|
||||
EOF
|
||||
_docker_user_orch add
|
||||
}
|
||||
|
||||
# Insert (add) or delete (del) the DOCKER-USER ACCEPT rules for the
|
||||
# orchestrator link, idempotently, only when the chain exists.
|
||||
_docker_user_orch() {
|
||||
local op="$1" flag
|
||||
command -v iptables >/dev/null 2>&1 || return 0
|
||||
iptables -t filter -L DOCKER-USER >/dev/null 2>&1 || return 0
|
||||
for flag in "-i" "-o"; do
|
||||
if [ "$op" = add ]; then
|
||||
iptables -C DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT 2>/dev/null \
|
||||
|| iptables -I DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT
|
||||
else
|
||||
iptables -D DOCKER-USER "$flag" "$ORCH_IFACE" -j ACCEPT 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
}
|
||||
|
||||
# Agent -> gateway VM routing. The shared gateway (egress / supervise /
|
||||
# git-http) runs in the orchestrator/infra VM at $(orch_guest). Agents keep
|
||||
# addressing their own host-side TAP IP on the gateway ports; a PREROUTING
|
||||
# DNAT redirects that to the infra VM, and the isolation table's
|
||||
# `ct status dnat accept` forward rule lets it through — every other agent
|
||||
# egress stays dropped. Source IP is deliberately NOT masqueraded: the
|
||||
# gateway attributes each request to the originating bottle by its (nft +
|
||||
# /31 unspoofable) guest IP.
|
||||
_install_gateway_route() {
|
||||
nft -f - <<EOF
|
||||
table ip ${TABLE}_gw {}
|
||||
delete table ip ${TABLE}_gw
|
||||
table ip ${TABLE}_gw {
|
||||
chain prerouting {
|
||||
type nat hook prerouting priority -100; policy accept;
|
||||
iifname "${PREFIX}*" tcp dport { $GATEWAY_PORTS } dnat to $(orch_guest)
|
||||
}
|
||||
}
|
||||
EOF
|
||||
}
|
||||
|
||||
cmd_down() {
|
||||
require_root down
|
||||
_docker_user_orch del
|
||||
nft delete table ip "${TABLE}_gw" 2>/dev/null || true
|
||||
nft delete table inet "${TABLE}_nat" 2>/dev/null || true
|
||||
if ip link show "$ORCH_IFACE" >/dev/null 2>&1; then
|
||||
ip link set "$ORCH_IFACE" down 2>/dev/null || true
|
||||
ip tuntap del dev "$ORCH_IFACE" mode tap 2>/dev/null || true
|
||||
echo " removed $ORCH_IFACE"
|
||||
fi
|
||||
nft delete table inet "$TABLE" 2>/dev/null || true
|
||||
for i in $(seq 0 $((POOL_SIZE-1))); do
|
||||
local dev ; dev="$(iface "$i")"
|
||||
@@ -296,10 +178,6 @@ cmd_down() {
|
||||
cmd_status() {
|
||||
echo "table inet $TABLE:"
|
||||
nft list table inet "$TABLE" 2>/dev/null || echo " (absent)"
|
||||
echo "table inet ${TABLE}_nat (orchestrator egress):"
|
||||
nft list table inet "${TABLE}_nat" 2>/dev/null || echo " (absent)"
|
||||
echo "table ip ${TABLE}_gw (agent->gateway route):"
|
||||
nft list table ip "${TABLE}_gw" 2>/dev/null || echo " (absent)"
|
||||
echo "taps:"
|
||||
for i in $(seq 0 $((POOL_SIZE-1))); do
|
||||
local dev ; dev="$(iface "$i")"
|
||||
@@ -307,9 +185,6 @@ cmd_status() {
|
||||
ip -brief addr show "$dev" | sed 's/^/ /'
|
||||
fi
|
||||
done
|
||||
if ip -brief addr show "$ORCH_IFACE" >/dev/null 2>&1; then
|
||||
ip -brief addr show "$ORCH_IFACE" | sed 's/^/ /'
|
||||
fi
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
|
||||
@@ -1,155 +0,0 @@
|
||||
"""Integration: the Docker orchestrator's control plane enforces the
|
||||
per-host auth secret against a real container (issue #400).
|
||||
|
||||
Unit tests exercise `dispatch()` in-process, socket-free. This starts the
|
||||
actual orchestrator + gateway as Docker containers and drives the real HTTP
|
||||
server over its published loopback port, the same path an agent sharing the
|
||||
gateway network — or the trusted host CLI — would use.
|
||||
|
||||
Gated on a reachable Docker daemon. Uses unique container/network names,
|
||||
test-only image tags (never the production `:latest` ones, so a rebuild
|
||||
here can't make `_running_image_is_current()` see a real host's running
|
||||
gateway as stale and force-recreate it), and a throwaway `BOT_BOTTLE_ROOT`
|
||||
so a run never touches or collides with a real per-host orchestrator or
|
||||
gateway. The whole stack is brought up once for the class (`setUpClass`),
|
||||
not per test method — every test here is a read-only check against the
|
||||
same running control plane.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import secrets
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
|
||||
from bot_bottle.orchestrator.client import OrchestratorClient
|
||||
from bot_bottle.orchestrator.lifecycle import OrchestratorService
|
||||
from bot_bottle.paths import host_control_plane_token
|
||||
from tests._docker import skip_unless_docker
|
||||
|
||||
# Fixed (not per-run-suffixed) so repeated runs reuse the same layer-cached
|
||||
# image instead of leaking a new dangling tag on every invocation.
|
||||
_TEST_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:itest"
|
||||
_TEST_GATEWAY_IMAGE = "bot-bottle-gateway:itest"
|
||||
|
||||
|
||||
@skip_unless_docker()
|
||||
@unittest.skipIf(
|
||||
os.environ.get("GITEA_ACTIONS") == "true",
|
||||
"skipped under act_runner: the orchestrator container bind-mounts the repo "
|
||||
"path into a container on the socket-shared host daemon, which can't see the "
|
||||
"runner's /workspace — same host-bind-mount constraint as the other "
|
||||
"bottle-bringup integration tests",
|
||||
)
|
||||
class TestDockerControlPlaneAuthIntegration(unittest.TestCase):
|
||||
@classmethod
|
||||
def setUpClass(cls) -> None:
|
||||
suffix = secrets.token_hex(4)
|
||||
cls._tmp = tempfile.TemporaryDirectory() # pylint: disable=consider-using-with
|
||||
cls.addClassCleanup(cls._tmp.cleanup)
|
||||
|
||||
# host_control_plane_token() — both the token read below and the one
|
||||
# OrchestratorService injects into the container's env — resolves its
|
||||
# path via the *ambient* BOT_BOTTLE_ROOT env var, not the host_root
|
||||
# kwarg passed to the constructor (that kwarg only controls the DB
|
||||
# bind-mount destination). Without pointing the env var at the same
|
||||
# throwaway dir, this "isolated" test would read/write the developer's
|
||||
# real ~/.bot-bottle/control-plane-token.
|
||||
previous_root = os.environ.get("BOT_BOTTLE_ROOT")
|
||||
|
||||
def _restore_root() -> None:
|
||||
if previous_root is None:
|
||||
os.environ.pop("BOT_BOTTLE_ROOT", None)
|
||||
else:
|
||||
os.environ["BOT_BOTTLE_ROOT"] = previous_root
|
||||
|
||||
os.environ["BOT_BOTTLE_ROOT"] = cls._tmp.name
|
||||
cls.addClassCleanup(_restore_root)
|
||||
|
||||
orchestrator_name = f"bot-bottle-orch-itest-{suffix}"
|
||||
gateway_name = f"bot-bottle-gw-itest-{suffix}"
|
||||
network = f"bot-bottle-net-itest-{suffix}"
|
||||
host_root = Path(cls._tmp.name)
|
||||
cls.addClassCleanup(
|
||||
cls._teardown_docker, orchestrator_name, gateway_name, network, host_root
|
||||
)
|
||||
|
||||
cls.svc = OrchestratorService(
|
||||
orchestrator_name=orchestrator_name,
|
||||
gateway_name=gateway_name,
|
||||
network=network,
|
||||
image=_TEST_ORCHESTRATOR_IMAGE,
|
||||
gateway_image=_TEST_GATEWAY_IMAGE,
|
||||
port=20000 + secrets.randbelow(10000),
|
||||
host_root=host_root,
|
||||
)
|
||||
cls.svc.ensure_running()
|
||||
cls.token = host_control_plane_token()
|
||||
|
||||
@staticmethod
|
||||
def _teardown_docker(
|
||||
orchestrator_name: str, gateway_name: str, network: str, host_root: Path
|
||||
) -> None:
|
||||
subprocess.run(
|
||||
["docker", "rm", "--force", orchestrator_name, gateway_name],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
|
||||
)
|
||||
subprocess.run(
|
||||
["docker", "network", "rm", network],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
|
||||
)
|
||||
# The orchestrator container (no USER directive) wrote the registry
|
||||
# DB as root into the throwaway host_root; chown it back so the
|
||||
# (non-root) tempdir cleanup can remove it. Same workaround
|
||||
# test_multitenant_isolation.py uses for the identical bind mount.
|
||||
subprocess.run(
|
||||
["docker", "run", "--rm", "-v", f"{host_root}:/r",
|
||||
"--entrypoint", "chown", _TEST_GATEWAY_IMAGE, "-R",
|
||||
f"{os.getuid()}:{os.getgid()}", "/r"],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL, check=False,
|
||||
)
|
||||
|
||||
def _request(
|
||||
self, method: str, path: str, *, token: str = ""
|
||||
) -> tuple[int, dict[str, object]]:
|
||||
# Reuses the real host-side client's request/response handling rather
|
||||
# than hand-rolling urllib here; _request (not one of the named
|
||||
# wrapper methods) is what exposes raw status codes for arbitrary
|
||||
# paths/tokens, which is exactly what these auth-boundary tests need.
|
||||
client = OrchestratorClient(self.svc.url, auth_token=token)
|
||||
return client._request(method, path) # pylint: disable=protected-access
|
||||
|
||||
def test_health_is_open_without_a_token(self) -> None:
|
||||
status, payload = self._request("GET", "/health")
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual("ok", payload["status"])
|
||||
|
||||
def test_bottles_rejects_a_caller_with_no_token(self) -> None:
|
||||
"""The enumeration attack from issue #400: an agent sharing the
|
||||
gateway network could list every bottle + its policy with no
|
||||
credential at all."""
|
||||
status, _ = self._request("GET", "/bottles")
|
||||
self.assertEqual(401, status)
|
||||
|
||||
def test_bottles_rejects_a_wrong_token(self) -> None:
|
||||
status, _ = self._request("GET", "/bottles", token="not-the-real-secret")
|
||||
self.assertEqual(401, status)
|
||||
|
||||
def test_bottles_accepts_the_real_token(self) -> None:
|
||||
status, payload = self._request("GET", "/bottles", token=self.token)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual([], payload["bottles"])
|
||||
|
||||
def test_resolve_rejects_a_caller_with_no_token(self) -> None:
|
||||
"""The credential-lift attack from issue #400: an agent could POST
|
||||
/resolve directly and read back the upstream tokens it's never meant
|
||||
to see."""
|
||||
status, _ = self._request("POST", "/resolve")
|
||||
self.assertEqual(401, status)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -205,29 +205,25 @@ class TestFirecrackerFreezer(_FakeHomeMixin, unittest.TestCase):
|
||||
)
|
||||
|
||||
def test_snapshots_running_vm_without_stopping(self):
|
||||
"""Commit should tar the running guest rootfs over SSH into the
|
||||
committed-rootfs artifact (no Docker), not stop the VM."""
|
||||
"""Commit should tar the running guest rootfs over SSH, not stop it."""
|
||||
slug = "dev-abc12"
|
||||
self._write_meta(slug)
|
||||
self._stage_run_dir(slug)
|
||||
freezer = FirecrackerFreezer()
|
||||
agent = _make_agent(slug, "firecracker")
|
||||
|
||||
commit_fn = "bot_bottle.backend.firecracker.freezer._commit_rootfs_via_ssh"
|
||||
with patch(commit_fn) as mock_commit, \
|
||||
with patch("bot_bottle.backend.firecracker.freezer._commit_via_ssh") as mock_commit, \
|
||||
patch("bot_bottle.backend.freeze.info"), \
|
||||
patch("bot_bottle.backend.firecracker.freezer.info"):
|
||||
freezer.commit(agent)
|
||||
|
||||
tar_path = bottle_state.committed_rootfs_path(slug)
|
||||
image_tag = f"bot-bottle-committed-{slug}:latest"
|
||||
self.assertEqual(1, mock_commit.call_count)
|
||||
# (private_key, guest_ip, tar_path) — guest_ip parsed from config.
|
||||
# (private_key, guest_ip, image_tag) — guest_ip parsed from config.
|
||||
args = mock_commit.call_args.args
|
||||
self.assertEqual("100.64.0.1", args[1])
|
||||
self.assertEqual(tar_path, args[2])
|
||||
# The committed-image state records the artifact path; resume boots
|
||||
# from the tar rather than a Docker image.
|
||||
self.assertEqual(str(tar_path), bottle_state.read_committed_image(slug))
|
||||
self.assertEqual(image_tag, args[2])
|
||||
self.assertEqual(image_tag, bottle_state.read_committed_image(slug))
|
||||
self.assertTrue(bottle_state.is_preserved(slug))
|
||||
|
||||
|
||||
|
||||
@@ -244,45 +244,5 @@ class TestHasBackend(unittest.TestCase):
|
||||
self.assertFalse(has_backend("nonexistent"))
|
||||
|
||||
|
||||
class TestEnsureOrchestrator(unittest.TestCase):
|
||||
"""The backend-agnostic orchestrator bring-up entry point. Docker starts
|
||||
the orchestrator + gateway containers; firecracker boots the infra VM;
|
||||
macos-container starts the infra container."""
|
||||
|
||||
def test_docker_delegates_to_orchestrator_service(self):
|
||||
b = get_bottle_backend("docker")
|
||||
with patch(
|
||||
"bot_bottle.orchestrator.lifecycle.OrchestratorService"
|
||||
) as service_cls:
|
||||
service_cls.return_value.ensure_running.return_value = (
|
||||
"http://127.0.0.1:8099"
|
||||
)
|
||||
url = b.ensure_orchestrator()
|
||||
self.assertEqual(url, "http://127.0.0.1:8099")
|
||||
service_cls.return_value.ensure_running.assert_called_once_with()
|
||||
|
||||
def test_firecracker_delegates_to_infra_vm(self):
|
||||
b = get_bottle_backend("firecracker")
|
||||
with patch(
|
||||
"bot_bottle.backend.firecracker.infra_vm.ensure_running"
|
||||
) as ensure_running:
|
||||
ensure_running.return_value.control_plane_url = (
|
||||
"http://10.243.255.1:8099"
|
||||
)
|
||||
url = b.ensure_orchestrator()
|
||||
self.assertEqual(url, "http://10.243.255.1:8099")
|
||||
|
||||
def test_macos_delegates_to_infra_container(self):
|
||||
b = get_bottle_backend("macos-container")
|
||||
with patch(
|
||||
"bot_bottle.backend.macos_container.infra.MacosInfraService"
|
||||
) as service_cls:
|
||||
service_cls.return_value.ensure_running.return_value.control_plane_url = (
|
||||
"http://192.168.128.2:8099"
|
||||
)
|
||||
url = b.ensure_orchestrator()
|
||||
self.assertEqual(url, "http://192.168.128.2:8099")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -54,7 +54,6 @@ def _plan(
|
||||
skills: list[str] | None = None,
|
||||
agent_provision: AgentProvisionPlan | None = None,
|
||||
supervise: bool = False,
|
||||
identity_token: str = "",
|
||||
) -> DockerBottlePlan:
|
||||
bottle_json: dict = {"agent_provider": {"template": "codex"}} # type: ignore
|
||||
if supervise:
|
||||
@@ -101,7 +100,6 @@ def _plan(
|
||||
),
|
||||
supervise_plan=supervise_plan,
|
||||
use_runsc=False,
|
||||
identity_token=identity_token,
|
||||
agent_provision=agent_provision or AgentProvisionPlan(
|
||||
template="codex", command="codex", prompt_mode="read_prompt_file",
|
||||
image="bot-bottle-codex:latest", dockerfile="",
|
||||
@@ -316,31 +314,6 @@ class TestCodexDockerfile(unittest.TestCase):
|
||||
self.assertIn("procps", dockerfile)
|
||||
|
||||
|
||||
# Codex-supported streamable-HTTP MCP config keys (RawMcpServerConfig in
|
||||
# codex-rs/config/src/mcp_types.rs). config.toml uses deny_unknown_fields,
|
||||
# so an entry that names anything outside this set is rejected by the CLI.
|
||||
_CODEX_HTTP_MCP_KEYS = frozenset({
|
||||
"url", "http_headers", "env_http_headers", "bearer_token_env_var",
|
||||
# shared (transport-agnostic) keys
|
||||
"environment_id", "auth", "startup_timeout_sec", "startup_timeout_ms",
|
||||
"tool_timeout_sec", "enabled", "required", "supports_parallel_tool_calls",
|
||||
"default_tools_approval_mode", "enabled_tools", "disabled_tools",
|
||||
"scopes", "oauth", "oauth_resource", "name", "tools",
|
||||
})
|
||||
|
||||
|
||||
def _append_target(bottle: MagicMock) -> tuple[str, str]:
|
||||
"""Reconstruct (config_path, appended_toml) from the base64 append
|
||||
script the provider ran."""
|
||||
import base64 as _b64
|
||||
|
||||
script = bottle.exec.call_args.args[0]
|
||||
# printf %s '<b64>' | base64 -d >> '<path>'
|
||||
b64 = script.split("printf %s ", 1)[1].split(" |", 1)[0].strip("'")
|
||||
path = script.rsplit(">> ", 1)[1].strip().strip("'")
|
||||
return path, _b64.b64decode(b64).decode()
|
||||
|
||||
|
||||
class TestCodexSuperviseMcp(unittest.TestCase):
|
||||
def test_noop_when_supervise_disabled(self):
|
||||
bottle = _make_bottle()
|
||||
@@ -349,68 +322,27 @@ class TestCodexSuperviseMcp(unittest.TestCase):
|
||||
)
|
||||
bottle.exec.assert_not_called()
|
||||
|
||||
def test_appends_streamable_http_entry_as_node(self):
|
||||
import tomllib
|
||||
|
||||
bottle = _make_bottle()
|
||||
plan = _plan(supervise=True, identity_token="tok-abc123")
|
||||
CodexAgentProvider().provision_supervise_mcp(plan, bottle, _URL)
|
||||
bottle.exec.assert_called_once()
|
||||
self.assertEqual("node", bottle.exec.call_args.kwargs.get("user"))
|
||||
|
||||
config_path, appended = _append_target(bottle)
|
||||
self.assertEqual("/home/node/.codex/config.toml", config_path)
|
||||
# The appended block must be valid TOML and parse to a streamable
|
||||
# HTTP server carrying the identity token as a static http header.
|
||||
parsed = tomllib.loads(appended)
|
||||
server = parsed["mcp_servers"]["supervise"]
|
||||
self.assertEqual(_URL, server["url"])
|
||||
self.assertEqual(
|
||||
"tok-abc123", server["http_headers"]["x-bot-bottle-identity"],
|
||||
)
|
||||
# Never emit an unsupported key (config.toml is deny_unknown_fields);
|
||||
# in particular there is no `--header` / `header` surface.
|
||||
self.assertTrue(
|
||||
set(server).issubset(_CODEX_HTTP_MCP_KEYS),
|
||||
f"unsupported codex mcp keys: {set(server) - _CODEX_HTTP_MCP_KEYS}",
|
||||
)
|
||||
self.assertNotIn("mcp add", bottle.exec.call_args.args[0])
|
||||
|
||||
def test_appends_to_custom_codex_home(self):
|
||||
bottle = _make_bottle()
|
||||
provision = AgentProvisionPlan(
|
||||
template="codex", command="codex", prompt_mode="read_prompt_file",
|
||||
image="", dockerfile="", guest_home="/home/node",
|
||||
instance_name="bot-bottle-demo-abc12",
|
||||
prompt_file=Path("/tmp/prompt.txt"),
|
||||
guest_env={"CODEX_HOME": "/home/node/alt-codex"},
|
||||
)
|
||||
plan = _plan(
|
||||
supervise=True, agent_provision=provision, identity_token="tok",
|
||||
)
|
||||
CodexAgentProvider().provision_supervise_mcp(plan, bottle, _URL)
|
||||
config_path, _ = _append_target(bottle)
|
||||
self.assertEqual("/home/node/alt-codex/config.toml", config_path)
|
||||
|
||||
def test_omits_http_headers_when_no_token(self):
|
||||
import tomllib
|
||||
|
||||
def test_runs_codex_mcp_add_as_node(self):
|
||||
bottle = _make_bottle()
|
||||
CodexAgentProvider().provision_supervise_mcp(
|
||||
_plan(supervise=True), bottle, _URL,
|
||||
)
|
||||
_, appended = _append_target(bottle)
|
||||
server = tomllib.loads(appended)["mcp_servers"]["supervise"]
|
||||
self.assertNotIn("http_headers", server)
|
||||
bottle.exec.assert_called_once()
|
||||
script = bottle.exec.call_args.args[0]
|
||||
self.assertEqual("node", bottle.exec.call_args.kwargs.get("user"))
|
||||
self.assertEqual(
|
||||
"/home/node/.codex/packages/standalone/current/bin/codex "
|
||||
f"mcp add supervise --url {_URL}",
|
||||
script,
|
||||
)
|
||||
|
||||
def test_registration_failure_is_fatal(self):
|
||||
def test_logs_warning_on_failure_but_does_not_raise(self):
|
||||
bottle = _make_bottle(
|
||||
exec_result=ExecResult(returncode=1, stdout="", stderr="boom"),
|
||||
)
|
||||
with self.assertRaises(SystemExit):
|
||||
CodexAgentProvider().provision_supervise_mcp(
|
||||
_plan(supervise=True), bottle, _URL,
|
||||
)
|
||||
CodexAgentProvider().provision_supervise_mcp(
|
||||
_plan(supervise=True), bottle, _URL,
|
||||
)
|
||||
|
||||
|
||||
class TestCodexHeadlessPrompt(unittest.TestCase):
|
||||
|
||||
@@ -8,7 +8,6 @@ real mitmproxy package."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import types
|
||||
import unittest
|
||||
@@ -108,14 +107,14 @@ class _Flow:
|
||||
def _log_request(addon: EgressAddon, flow: _Flow) -> dict[str, Any]:
|
||||
buf = StringIO()
|
||||
with patch("sys.stderr", buf):
|
||||
addon._log_request(flow, os.environ) # type: ignore[arg-type]
|
||||
addon._log_request(flow) # type: ignore[arg-type]
|
||||
return json.loads(buf.getvalue())
|
||||
|
||||
|
||||
def _log_response(addon: EgressAddon, flow: _Flow) -> dict[str, Any]:
|
||||
buf = StringIO()
|
||||
with patch("sys.stderr", buf):
|
||||
addon._log_response(flow, os.environ) # type: ignore[arg-type]
|
||||
addon._log_response(flow) # type: ignore[arg-type]
|
||||
return json.loads(buf.getvalue())
|
||||
|
||||
|
||||
|
||||
@@ -141,10 +141,6 @@ class _Flow:
|
||||
self.response = response
|
||||
self.websocket: Any = None
|
||||
self.killed = False
|
||||
# mitmproxy flows carry a per-flow `metadata` dict for addon use; the
|
||||
# egress addon stashes the resolved (config, slug, env) there in
|
||||
# request() so the response/websocket hooks reuse it.
|
||||
self.metadata: dict[str, Any] = {}
|
||||
|
||||
def kill(self) -> None:
|
||||
self.killed = True
|
||||
@@ -863,91 +859,5 @@ class TestSuperviseMultiTenant(unittest.TestCase):
|
||||
self.assertNotIn(_OPENAI_KEY, addon._safe_tokens_for(""))
|
||||
|
||||
|
||||
class TestMultiTenantInboundDlp(unittest.TestCase):
|
||||
"""Consolidated gateway: the response + websocket DLP hooks must scan
|
||||
against the *calling bottle's* config, resolved by source IP in request()
|
||||
and reused here. The static `self.config` is empty in this mode, so before
|
||||
the flow-context stash these hooks silently skipped every scan (fail-open).
|
||||
"""
|
||||
|
||||
def _consolidated_addon(self) -> EgressAddon:
|
||||
addon = _addon(Config(routes=())) # empty static config, as in prod
|
||||
addon._resolver = cast(Any, _CtxResolver({"10.0.0.1": "bottle-a"}))
|
||||
return addon
|
||||
|
||||
def test_response_injection_blocked_after_request_resolves(self) -> None:
|
||||
addon = self._consolidated_addon()
|
||||
flow = _with_client_ip(_Flow(_Request(host="api.example.com")), "10.0.0.1")
|
||||
_run_request(addon, flow) # resolves + stashes bottle-a's allowlist
|
||||
self.assertIsNone(flow.response) # request forwarded
|
||||
flow.response = _Response(200, content=_INJECTION_BLOCK)
|
||||
addon.response(flow) # type: ignore[arg-type]
|
||||
assert flow.response is not None
|
||||
# Empty static config would have found no route and left this 200.
|
||||
self.assertEqual(403, flow.response.status_code)
|
||||
|
||||
def test_websocket_outbound_token_killed_after_request_resolves(self) -> None:
|
||||
addon = self._consolidated_addon()
|
||||
flow = _with_client_ip(_Flow(_Request(host="api.example.com")), "10.0.0.1")
|
||||
_run_request(addon, flow) # the ws upgrade resolves + stashes the config
|
||||
flow.websocket = _WebSocketData(
|
||||
[_Message(f"k={_OPENAI_KEY}".encode(), from_client=True)]
|
||||
)
|
||||
addon.websocket_message(flow) # type: ignore[arg-type]
|
||||
self.assertTrue(flow.killed) # scanned against bottle-a's route now
|
||||
|
||||
def test_websocket_inbound_injection_killed_after_request_resolves(self) -> None:
|
||||
addon = self._consolidated_addon()
|
||||
flow = _with_client_ip(_Flow(_Request(host="api.example.com")), "10.0.0.1")
|
||||
_run_request(addon, flow)
|
||||
flow.websocket = _WebSocketData(
|
||||
[_Message(_INJECTION_BLOCK.encode(), from_client=False)]
|
||||
)
|
||||
addon.websocket_message(flow) # type: ignore[arg-type]
|
||||
self.assertTrue(flow.killed)
|
||||
|
||||
def test_response_log_redacts_per_bottle_resolve_token(self) -> None:
|
||||
# LOG_FULL response logging now runs in multi-tenant mode, so it must
|
||||
# scrub the calling bottle's /resolve token — which lives only in the
|
||||
# resolved env overlay, never in the gateway's os.environ. A
|
||||
# non-token-shaped secret is caught only via that env, so os.environ
|
||||
# redaction (the pre-fix behaviour) would leak it into the log.
|
||||
secret = "bottle-a-provisioned-secret-value"
|
||||
policy = "log: 2\nroutes:\n - host: api.example.com\n"
|
||||
|
||||
class _TokenResolver:
|
||||
def resolve_policy_and_bottle_id(
|
||||
self, ip: str, identity_token: str = "",
|
||||
) -> tuple[str | None, str | None, dict[str, str]]:
|
||||
del ip, identity_token
|
||||
return policy, "bottle-a", {"EGRESS_TOKEN_0": secret}
|
||||
|
||||
addon = _addon(Config(routes=()))
|
||||
addon._resolver = cast(Any, _TokenResolver())
|
||||
flow = _with_client_ip(_Flow(_Request(host="api.example.com")), "10.0.0.1")
|
||||
_run_request(addon, flow)
|
||||
flow.response = _Response(200, content=f"echo {secret} back")
|
||||
buf = StringIO()
|
||||
with patch("sys.stderr", buf):
|
||||
addon.response(flow) # type: ignore[arg-type]
|
||||
logged = buf.getvalue()
|
||||
self.assertIn("egress_response", logged) # LOG_FULL logged the response
|
||||
self.assertNotIn(secret, logged) # redacted via the resolved env overlay
|
||||
|
||||
def test_unattributed_flow_has_no_route_so_frame_passes(self) -> None:
|
||||
# An unattributed source resolves to a deny-all (empty) config, so the
|
||||
# request is blocked at the upgrade and any later frame has no route to
|
||||
# scan against — it passes rather than being attributed to a bottle.
|
||||
addon = self._consolidated_addon()
|
||||
flow = _with_client_ip(_Flow(_Request(host="api.example.com")), "10.9.9.9")
|
||||
_run_request(addon, flow)
|
||||
self.assertIsNotNone(flow.response) # blocked at the upgrade
|
||||
flow.websocket = _WebSocketData(
|
||||
[_Message(f"k={_OPENAI_KEY}".encode(), from_client=True)]
|
||||
)
|
||||
addon.websocket_message(flow) # type: ignore[arg-type]
|
||||
self.assertFalse(flow.killed)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -255,30 +255,6 @@ class TestBottleAgentArgv(unittest.TestCase):
|
||||
argv[idx:],
|
||||
)
|
||||
|
||||
def test_codex_multiword_prompt_survives_ssh_reparse(self):
|
||||
# codex's read_prompt_file mode passes a single positional with
|
||||
# spaces ("Read and follow the instructions in <path>."). ssh
|
||||
# space-joins the remote argv and the guest shell re-splits it, so
|
||||
# the token MUST be quoted or codex sees "and" as a subcommand
|
||||
# (regression). Each remote token is shlex.quote'd; round-tripping
|
||||
# the joined remote command back through shlex.split must recover
|
||||
# the prompt as ONE argument.
|
||||
import shlex
|
||||
|
||||
argv = _bottle(
|
||||
agent_command="codex",
|
||||
agent_prompt_mode="read_prompt_file",
|
||||
agent_provider_template="codex",
|
||||
prompt_path_in_guest="/home/node/.bot-bottle-prompt.txt",
|
||||
).agent_argv([], tty=False)
|
||||
idx = argv.index("--")
|
||||
remote_line = " ".join(argv[idx + 1:]) # what ssh sends to the guest
|
||||
reparsed = shlex.split(remote_line) # what the guest shell sees
|
||||
prompt = "Read and follow the instructions in /home/node/.bot-bottle-prompt.txt."
|
||||
self.assertIn(prompt, reparsed)
|
||||
# codex is the last simple token before the (single) prompt arg.
|
||||
self.assertEqual([*reparsed[reparsed.index("codex"):]], ["codex", prompt])
|
||||
|
||||
def test_workdir_sets_chdir(self):
|
||||
# The agent runs from its workdir via `env --chdir` (ssh-safe;
|
||||
# not a `sh -c 'cd …'` wrapper, which the ssh arg-join mangles).
|
||||
@@ -358,18 +334,6 @@ class TestBootArgs(unittest.TestCase):
|
||||
self.assertEqual("/run/rootfs.ext4", cfg["drives"][0]["path_on_host"])
|
||||
self.assertFalse(cfg["drives"][0]["is_read_only"])
|
||||
self.assertEqual("bbfc0", cfg["network-interfaces"][0]["host_dev_name"])
|
||||
self.assertEqual(1, len(cfg["drives"])) # no data drive by default
|
||||
|
||||
def test_config_adds_data_drive(self):
|
||||
cfg = cast(Any, firecracker_vm._config(
|
||||
rootfs=Path("/run/rootfs.ext4"), tap="bbfc0",
|
||||
guest_ip="100.64.0.1", host_ip="100.64.0.0", pubkey="k",
|
||||
vcpus=2, mem_mib=2048, guest_mac="06:00:AC:10:00:02",
|
||||
data_drive=Path("/run/registry.ext4"),
|
||||
))
|
||||
self.assertEqual(2, len(cfg["drives"]))
|
||||
self.assertFalse(cfg["drives"][1]["is_root_device"])
|
||||
self.assertEqual("/run/registry.ext4", cfg["drives"][1]["path_on_host"])
|
||||
|
||||
|
||||
class TestBottleExecClose(unittest.TestCase):
|
||||
|
||||
@@ -6,13 +6,10 @@ branches. Mock subprocess/os so nothing needs KVM or a live VM.
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import stat
|
||||
import subprocess
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from unittest.mock import patch
|
||||
|
||||
from bot_bottle.backend.firecracker import firecracker_vm, freezer, netpool, util
|
||||
@@ -67,21 +64,6 @@ class TestNetpoolProbes(unittest.TestCase):
|
||||
patch.object(netpool, "tap_present", side_effect=[True, False]):
|
||||
self.assertEqual(["bbfc1"], netpool.missing_taps())
|
||||
|
||||
def test_orch_slot_is_top_of_ip_base_16(self):
|
||||
# Dedicated orchestrator link: /31 at the top of the IP_BASE /16,
|
||||
# clear of the pool (bottom of the block). Must match the shell
|
||||
# script's orch_host()/orch_guest() and be a non-pool index.
|
||||
with patch.dict("os.environ", {"BOT_BOTTLE_FC_IP_BASE": "10.243.0.0"}):
|
||||
s = netpool.orch_slot()
|
||||
self.assertEqual(netpool.ORCH_IFACE, s.iface)
|
||||
self.assertEqual("10.243.255.0", s.host_ip)
|
||||
self.assertEqual("10.243.255.1", s.guest_ip)
|
||||
self.assertEqual(-1, s.index)
|
||||
# Never collides with a pool slot's guest address.
|
||||
with patch.dict("os.environ", {"BOT_BOTTLE_FC_IP_BASE": "10.243.0.0"}):
|
||||
pool_guests = {sl.guest_ip for sl in netpool.all_slots()}
|
||||
self.assertNotIn(s.guest_ip, pool_guests)
|
||||
|
||||
|
||||
class TestConsoleTail(unittest.TestCase):
|
||||
def test_reads_tail(self):
|
||||
@@ -131,172 +113,5 @@ class TestRequireFirecracker(unittest.TestCase):
|
||||
util.require_firecracker()
|
||||
|
||||
|
||||
class TestBuildCommittedRootfsDir(unittest.TestCase):
|
||||
"""Resume prepares the base rootfs dir from the freezer's snapshot tar
|
||||
with no Docker: extract, recreate the excluded mount points, inject the
|
||||
guest boot bits."""
|
||||
|
||||
def _make_tar(self, tmp: Path) -> Path:
|
||||
import tarfile
|
||||
|
||||
src = tmp / "src"
|
||||
(src / "home" / "node").mkdir(parents=True)
|
||||
(src / "home" / "node" / "hello").write_text("hi")
|
||||
tar_path = tmp / "rootfs.tar"
|
||||
with tarfile.open(tar_path, "w") as tar:
|
||||
tar.add(src, arcname=".")
|
||||
return tar_path
|
||||
|
||||
def test_extracts_recreates_mountpoints_and_injects_boot(self):
|
||||
with tempfile.TemporaryDirectory(prefix="fc-committed.") as d:
|
||||
tmp = Path(d)
|
||||
tar_path = self._make_tar(tmp)
|
||||
# Stand in for the static dropbear that inject_guest_boot copies.
|
||||
dropbear = tmp / "dropbear"
|
||||
dropbear.write_text("#!/bin/true\n")
|
||||
cache = tmp / "cache"
|
||||
cache.mkdir()
|
||||
|
||||
with patch.object(util, "cache_dir", return_value=cache), \
|
||||
patch.object(util, "dropbear_path", return_value=dropbear), \
|
||||
patch.object(util, "info"):
|
||||
base = util.build_committed_rootfs_dir(tar_path)
|
||||
|
||||
self.assertEqual("hi", (base / "home" / "node" / "hello").read_text())
|
||||
for mount_point in ("proc", "sys", "dev", "run"):
|
||||
self.assertTrue((base / mount_point).is_dir(),
|
||||
f"missing recreated mount point /{mount_point}")
|
||||
self.assertTrue((base / "bb-dropbear").is_file())
|
||||
self.assertTrue((base / "bb-init").is_file())
|
||||
self.assertTrue((base / ".bb-ready").is_file())
|
||||
|
||||
def test_caches_on_repeat_and_reextracts_after_refreeze(self):
|
||||
with tempfile.TemporaryDirectory(prefix="fc-committed.") as d:
|
||||
tmp = Path(d)
|
||||
tar_path = self._make_tar(tmp)
|
||||
dropbear = tmp / "dropbear"
|
||||
dropbear.write_text("#!/bin/true\n")
|
||||
cache = tmp / "cache"
|
||||
cache.mkdir()
|
||||
|
||||
ctx = [
|
||||
patch.object(util, "cache_dir", return_value=cache),
|
||||
patch.object(util, "dropbear_path", return_value=dropbear),
|
||||
patch.object(util, "info"),
|
||||
]
|
||||
for c in ctx:
|
||||
c.start()
|
||||
self.addCleanup(lambda: [c.stop() for c in ctx])
|
||||
|
||||
real_run = subprocess.run
|
||||
calls = {"n": 0}
|
||||
|
||||
def counting_run(argv: list[str], *a: Any, **k: Any) -> Any:
|
||||
if argv and argv[0] == "tar":
|
||||
calls["n"] += 1
|
||||
return real_run(argv, *a, **k)
|
||||
|
||||
with patch.object(util.subprocess, "run", side_effect=counting_run):
|
||||
first = util.build_committed_rootfs_dir(tar_path)
|
||||
second = util.build_committed_rootfs_dir(tar_path)
|
||||
self.assertEqual(first, second)
|
||||
self.assertEqual(1, calls["n"]) # cached — no re-extract
|
||||
|
||||
# A re-freeze rewrites the tar; a new size/mtime -> new cache
|
||||
# key -> re-extract. Force a distinct mtime so the test isn't
|
||||
# at the mercy of filesystem timestamp granularity.
|
||||
import tarfile
|
||||
extra = tmp / "extra"
|
||||
extra.mkdir()
|
||||
(extra / "note").write_text("v2")
|
||||
with tarfile.open(tar_path, "w") as tar:
|
||||
tar.add(extra, arcname=".")
|
||||
st = tar_path.stat()
|
||||
os.utime(tar_path, ns=(st.st_atime_ns, st.st_mtime_ns + 1_000_000_000))
|
||||
|
||||
third = util.build_committed_rootfs_dir(tar_path)
|
||||
self.assertNotEqual(first, third)
|
||||
self.assertEqual(2, calls["n"])
|
||||
|
||||
|
||||
class TestInjectGuestBootSymlinkSafe(unittest.TestCase):
|
||||
"""A committed snapshot is guest-controlled: inject_guest_boot must not
|
||||
follow a planted symlink and overwrite a host file during resume."""
|
||||
|
||||
def test_planted_symlink_does_not_escape_staging_tree(self):
|
||||
with tempfile.TemporaryDirectory(prefix="fc-inject.") as d:
|
||||
tmp = Path(d)
|
||||
dropbear = tmp / "dropbear"
|
||||
dropbear.write_bytes(b"DROPBEAR")
|
||||
# A host file the malicious snapshot tries to clobber.
|
||||
victim = tmp / "victim"
|
||||
victim.write_text("original")
|
||||
|
||||
rootfs = tmp / "rootfs"
|
||||
rootfs.mkdir()
|
||||
# The snapshot planted bb-init/bb-dropbear as symlinks to it.
|
||||
(rootfs / "bb-init").symlink_to(victim)
|
||||
(rootfs / "bb-dropbear").symlink_to(victim)
|
||||
|
||||
with patch.object(util, "dropbear_path", return_value=dropbear):
|
||||
util.inject_guest_boot(rootfs, init_script="#!/bin/sh\nreal\n")
|
||||
|
||||
# Host file untouched; the staged paths are fresh regular files.
|
||||
self.assertEqual("original", victim.read_text())
|
||||
self.assertFalse((rootfs / "bb-init").is_symlink())
|
||||
self.assertFalse((rootfs / "bb-dropbear").is_symlink())
|
||||
self.assertEqual("#!/bin/sh\nreal\n", (rootfs / "bb-init").read_text())
|
||||
self.assertEqual(b"DROPBEAR", (rootfs / "bb-dropbear").read_bytes())
|
||||
|
||||
|
||||
class TestCommitRootfsPermissions(unittest.TestCase):
|
||||
"""The snapshot tar can hold the bottle's private workspace, so the
|
||||
freezer must write it owner-only (0600)."""
|
||||
|
||||
def _commit(self, tar_path: Path) -> int:
|
||||
"""Run _commit_rootfs_via_ssh with a stubbed ssh|tar pipe; return the
|
||||
mode of the open partial observed mid-stream (from subprocess.run)."""
|
||||
key = tar_path.parent.parent / "key"
|
||||
key.write_text("K")
|
||||
seen: dict[str, int] = {}
|
||||
|
||||
def fake_run(argv: list[str], *a: Any, **k: Any) -> Any:
|
||||
out = k["stdout"]
|
||||
seen["mode"] = stat.S_IMODE(os.fstat(out.fileno()).st_mode)
|
||||
out.write(b"TARDATA")
|
||||
return subprocess.CompletedProcess(argv, 0, b"", b"")
|
||||
|
||||
with patch.object(freezer.util, "ssh_base_argv", return_value=["ssh"]), \
|
||||
patch.object(freezer.subprocess, "run", side_effect=fake_run):
|
||||
freezer._commit_rootfs_via_ssh(key, "10.0.0.1", tar_path)
|
||||
return seen["mode"]
|
||||
|
||||
def test_snapshot_created_owner_only(self):
|
||||
with tempfile.TemporaryDirectory(prefix="fc-freeze.") as d:
|
||||
tar_path = Path(d) / "state" / "committed-rootfs.tar"
|
||||
tar_path.parent.mkdir()
|
||||
stream_mode = self._commit(tar_path)
|
||||
|
||||
self.assertEqual(0o600, stream_mode) # private during the stream
|
||||
self.assertEqual(b"TARDATA", tar_path.read_bytes())
|
||||
self.assertEqual(0o600, stat.S_IMODE(tar_path.stat().st_mode))
|
||||
|
||||
def test_leftover_world_readable_partial_is_recreated_private(self):
|
||||
"""A partial left 0644 by an interrupted prior run must not keep the
|
||||
new snapshot world-readable while it streams."""
|
||||
with tempfile.TemporaryDirectory(prefix="fc-freeze.") as d:
|
||||
tar_path = Path(d) / "state" / "committed-rootfs.tar"
|
||||
tar_path.parent.mkdir()
|
||||
partial = tar_path.with_name(tar_path.name + ".partial")
|
||||
partial.write_bytes(b"stale")
|
||||
os.chmod(partial, 0o644)
|
||||
|
||||
stream_mode = self._commit(tar_path)
|
||||
|
||||
self.assertEqual(0o600, stream_mode)
|
||||
self.assertEqual(b"TARDATA", tar_path.read_bytes())
|
||||
self.assertEqual(0o600, stat.S_IMODE(tar_path.stat().st_mode))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,74 +0,0 @@
|
||||
"""Unit tests for the docker-free Firecracker agent-image builder.
|
||||
|
||||
The VM boot / SSH / buildah plumbing (`_build_in_infra`) is integration-tested
|
||||
on a KVM host; here we cover the cache decision, the boot-bit injection, and
|
||||
the smoke-test no-op — the logic that must hold without a VM.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from bot_bottle.backend.firecracker import image_builder
|
||||
|
||||
|
||||
class TestBuildAgentRootfsDir(unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.cache = Path(self._tmp.name)
|
||||
self.dockerfile = self.cache / "Dockerfile"
|
||||
self.dockerfile.write_text("FROM node:22-slim\n")
|
||||
self.addCleanup(self._tmp.cleanup)
|
||||
|
||||
def test_cache_hit_skips_rebuild(self):
|
||||
digest = image_builder._dockerfile_hash(self.dockerfile)
|
||||
base = self.cache / "rootfs" / f"agent-{digest}"
|
||||
base.mkdir(parents=True)
|
||||
(base / ".bb-ready").write_text("ok\n")
|
||||
with patch.object(image_builder.util, "cache_dir", return_value=self.cache), \
|
||||
patch.object(image_builder, "_build_in_infra") as build:
|
||||
out = image_builder.build_agent_rootfs_dir(
|
||||
self.dockerfile, image_tag="t:latest")
|
||||
build.assert_not_called()
|
||||
self.assertEqual(base, out)
|
||||
|
||||
def test_cache_miss_builds_injects_and_marks_ready(self):
|
||||
with patch.object(image_builder.util, "cache_dir", return_value=self.cache), \
|
||||
patch.object(image_builder, "_build_in_infra") as build, \
|
||||
patch.object(image_builder.util, "inject_guest_boot") as inject:
|
||||
out = image_builder.build_agent_rootfs_dir(
|
||||
self.dockerfile, image_tag="t:latest", smoke_test=("claude", "--version"))
|
||||
build.assert_called_once()
|
||||
# smoke_test threads through to the VM build.
|
||||
self.assertEqual(("claude", "--version"), build.call_args.args[2])
|
||||
inject.assert_called_once()
|
||||
self.assertTrue((out / ".bb-ready").is_file())
|
||||
|
||||
def test_content_addressed_cache_key(self):
|
||||
other = self.cache / "Dockerfile2"
|
||||
other.write_text("FROM python:3.12-slim\n")
|
||||
self.assertNotEqual(
|
||||
image_builder._dockerfile_hash(self.dockerfile),
|
||||
image_builder._dockerfile_hash(other),
|
||||
)
|
||||
|
||||
|
||||
class TestSmokeTest(unittest.TestCase):
|
||||
def test_empty_argv_is_noop(self):
|
||||
with patch.object(image_builder, "_ssh") as ssh:
|
||||
image_builder._smoke_test(Path("/k"), "10.0.0.1", "tag", "ctr", ())
|
||||
ssh.assert_not_called()
|
||||
|
||||
def test_failed_smoke_dies(self):
|
||||
import subprocess
|
||||
result = subprocess.CompletedProcess([], 1, stdout="broken", stderr="")
|
||||
with patch.object(image_builder, "_ssh", return_value=result), \
|
||||
self.assertRaises(SystemExit):
|
||||
image_builder._smoke_test(Path("/k"), "10.0.0.1", "tag", "ctr", ("claude", "--version"))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,189 +0,0 @@
|
||||
"""Unit tests for the Firecracker infra VM (control-plane VM boot).
|
||||
|
||||
The KVM boot / HTTP reachability is integration-tested on a KVM host; here
|
||||
we cover the URL shape, the rootfs-variant wiring, and the health-poll
|
||||
decisions that must hold without a VM.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
from bot_bottle.backend.firecracker import infra_vm
|
||||
|
||||
|
||||
class TestControlPlaneUrl(unittest.TestCase):
|
||||
def test_url_uses_guest_ip_and_port(self):
|
||||
infra = infra_vm.InfraVm(
|
||||
vm=MagicMock(), guest_ip="10.243.255.1", private_key=Path("/k"))
|
||||
self.assertEqual(
|
||||
f"http://10.243.255.1:{infra_vm.CONTROL_PLANE_PORT}",
|
||||
infra.control_plane_url,
|
||||
)
|
||||
|
||||
|
||||
class TestBuildInfraRootfs(unittest.TestCase):
|
||||
def test_uses_infra_variant_and_init(self):
|
||||
with patch.object(infra_vm.util, "build_base_rootfs_dir") as build:
|
||||
build.return_value = Path("/cache/rootfs/x-infra")
|
||||
infra_vm.build_infra_rootfs_dir()
|
||||
build.assert_called_once()
|
||||
self.assertEqual(infra_vm._INFRA_IMAGE, build.call_args.args[0])
|
||||
# variant is "-infra-<init-hash>" so an init change rebuilds the rootfs.
|
||||
self.assertTrue(build.call_args.kwargs["variant"].startswith("-infra-"))
|
||||
# The init runs BOTH the control plane and the gateway data plane,
|
||||
# and exports PATH so gateway_init's subprocess daemons find python3.
|
||||
init = build.call_args.kwargs["init_script"]
|
||||
self.assertIn("bot_bottle.orchestrator", init)
|
||||
self.assertIn("gateway_init.py", init)
|
||||
self.assertIn("export PATH=", init)
|
||||
# Persistent registry volume mounted at the DB dir before the CP starts.
|
||||
self.assertIn("/dev/vdb", init)
|
||||
# VM backend uses git-http (9420); the git:// daemon is left out.
|
||||
self.assertIn("BOT_BOTTLE_GATEWAY_DAEMONS=egress,git-http,supervise", init)
|
||||
|
||||
|
||||
class TestSshGatewayTransport(unittest.TestCase):
|
||||
def test_cp_into_preserves_source_mode(self):
|
||||
import os
|
||||
import tempfile
|
||||
from subprocess import CompletedProcess
|
||||
with tempfile.NamedTemporaryFile() as f:
|
||||
os.chmod(f.name, 0o700) # like the staged access-hook
|
||||
t = infra_vm.SshGatewayTransport(Path("/k"), "10.0.0.1")
|
||||
with patch.object(infra_vm.subprocess, "run",
|
||||
return_value=CompletedProcess([], 0)) as run:
|
||||
t.cp_into(f.name, "/etc/git-gate/access-hook")
|
||||
remote_cmd = run.call_args.args[0][-1]
|
||||
self.assertIn("chmod 700", remote_cmd) # exec bit preserved over SSH
|
||||
|
||||
def test_exec_raises_on_failure(self):
|
||||
from subprocess import CompletedProcess
|
||||
t = infra_vm.SshGatewayTransport(Path("/k"), "10.0.0.1")
|
||||
with patch.object(infra_vm.subprocess, "run",
|
||||
return_value=CompletedProcess([], 1, stderr="nope")), \
|
||||
self.assertRaises(infra_vm.GatewayProvisionError):
|
||||
t.exec(["mkdir", "-p", "/git-gate"])
|
||||
|
||||
|
||||
class TestRegistryVolume(unittest.TestCase):
|
||||
def test_reuses_existing_volume(self):
|
||||
import tempfile
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
vol = Path(td) / "registry.ext4"
|
||||
vol.write_bytes(b"") # already present
|
||||
with patch.object(infra_vm, "registry_volume_path", return_value=vol), \
|
||||
patch.object(infra_vm.subprocess, "run") as run:
|
||||
out = infra_vm._ensure_registry_volume()
|
||||
run.assert_not_called() # no mke2fs when it exists
|
||||
self.assertEqual(vol, out)
|
||||
|
||||
def test_creates_volume_when_missing(self):
|
||||
import tempfile
|
||||
from subprocess import CompletedProcess
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
vol = Path(td) / "registry.ext4"
|
||||
with patch.object(infra_vm, "registry_volume_path", return_value=vol), \
|
||||
patch.object(infra_vm.subprocess, "run",
|
||||
return_value=CompletedProcess([], 0)) as run:
|
||||
infra_vm._ensure_registry_volume()
|
||||
argv = run.call_args.args[0]
|
||||
self.assertIn("mke2fs", argv)
|
||||
self.assertIn(str(vol), argv)
|
||||
|
||||
|
||||
class TestEnsureBuilt(unittest.TestCase):
|
||||
def test_default_pulls_artifact_without_docker(self):
|
||||
# PRD 0069 Stage 2: the launch host pulls the prebuilt rootfs; no Docker.
|
||||
with patch.object(infra_vm.docker_mod, "build_image") as build, \
|
||||
patch.object(infra_vm.infra_artifact, "ensure_artifact_gz") as pull:
|
||||
infra_vm.ensure_built()
|
||||
build.assert_not_called()
|
||||
pull.assert_called_once()
|
||||
|
||||
def test_local_mode_builds_deps_before_infra(self):
|
||||
with patch.dict(os.environ, {"BOT_BOTTLE_INFRA_BUILD": "local"}), \
|
||||
patch.object(infra_vm.docker_mod, "build_image") as build:
|
||||
infra_vm.ensure_built()
|
||||
tags = [c.args[0] for c in build.call_args_list]
|
||||
# infra is FROM gateway and COPY --from orchestrator, so both first.
|
||||
self.assertEqual(infra_vm._INFRA_IMAGE, tags[-1])
|
||||
self.assertIn(infra_vm._ORCHESTRATOR_IMAGE, tags[:-1])
|
||||
self.assertIn(infra_vm._GATEWAY_IMAGE, tags[:-1])
|
||||
|
||||
|
||||
class TestWaitForHealth(unittest.TestCase):
|
||||
def _infra(self, alive: bool = True) -> infra_vm.InfraVm:
|
||||
vm = MagicMock()
|
||||
vm.is_alive.return_value = alive
|
||||
return infra_vm.InfraVm(vm=vm, guest_ip="10.0.0.1", private_key=Path("/k"))
|
||||
|
||||
def test_returns_on_200(self):
|
||||
infra = self._infra()
|
||||
cm = MagicMock()
|
||||
cm.__enter__.return_value.status = 200
|
||||
with patch.object(infra_vm.urllib.request, "urlopen", return_value=cm):
|
||||
infra_vm.wait_for_health(infra, timeout=5) # must not raise
|
||||
|
||||
def test_dies_when_vm_exits(self):
|
||||
infra = self._infra(alive=False)
|
||||
assert infra.vm is not None # narrow for the type checker (it's a mock)
|
||||
infra.vm.process.returncode = 1
|
||||
with patch.object(infra_vm.firecracker_vm, "_console_tail", return_value=""), \
|
||||
self.assertRaises(SystemExit):
|
||||
infra_vm.wait_for_health(infra, timeout=5)
|
||||
|
||||
|
||||
class TestEnsureRunningSingleton(unittest.TestCase):
|
||||
def test_adopts_when_healthy(self):
|
||||
# A healthy control plane + existing key -> adopt (no boot), vm=None.
|
||||
with patch.object(infra_vm, "_health_ok", return_value=True), \
|
||||
patch.object(infra_vm, "_infra_dir") as d, \
|
||||
patch.object(infra_vm, "boot") as boot:
|
||||
keydir = MagicMock()
|
||||
(keydir / "id_ed25519").exists.return_value = True
|
||||
d.return_value = keydir
|
||||
infra = infra_vm.ensure_running()
|
||||
boot.assert_not_called()
|
||||
self.assertIsNone(infra.vm)
|
||||
|
||||
def test_boots_when_unhealthy(self):
|
||||
with patch.object(infra_vm, "_health_ok", return_value=False), \
|
||||
patch.object(infra_vm, "stop") as stop, \
|
||||
patch.object(infra_vm, "ensure_built") as built, \
|
||||
patch.object(infra_vm, "boot") as boot, \
|
||||
patch.object(infra_vm, "wait_for_health") as wait:
|
||||
boot.return_value = infra_vm.InfraVm(
|
||||
guest_ip="10.243.255.1", private_key=Path("/k"), vm=MagicMock())
|
||||
infra_vm.ensure_running()
|
||||
stop.assert_called_once() # clear a stale VM first
|
||||
built.assert_called_once()
|
||||
boot.assert_called_once()
|
||||
wait.assert_called_once()
|
||||
|
||||
|
||||
class TestKillPidfile(unittest.TestCase):
|
||||
def test_noop_when_no_pidfile(self):
|
||||
import tempfile
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
with patch.object(infra_vm, "_pid_file", return_value=Path(td) / "vm.pid"), \
|
||||
patch.object(infra_vm.os, "kill") as kill:
|
||||
infra_vm._kill_pidfile() # must not raise
|
||||
kill.assert_not_called()
|
||||
|
||||
def test_skips_dead_or_recycled_pid(self):
|
||||
import tempfile
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
pidf = Path(td) / "vm.pid"
|
||||
pidf.write_text("999999") # a PID that isn't a live firecracker
|
||||
with patch.object(infra_vm, "_pid_file", return_value=pidf), \
|
||||
patch.object(infra_vm.os, "kill") as kill:
|
||||
infra_vm._kill_pidfile()
|
||||
kill.assert_not_called()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -8,7 +8,6 @@ from unittest.mock import Mock, patch
|
||||
|
||||
from bot_bottle.backend.docker.gateway_provision import (
|
||||
GatewayProvisionError,
|
||||
DockerGatewayTransport,
|
||||
deprovision_git_gate,
|
||||
provision_git_gate,
|
||||
)
|
||||
@@ -55,7 +54,7 @@ class TestProvisionGitGate(unittest.TestCase):
|
||||
def test_copies_creds_and_runs_namespaced_init(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
with patch(_RUN, side_effect=_recorder(calls)):
|
||||
provision_git_gate(DockerGatewayTransport("gw"), "bottle1", _plan(_up("foo", known_hosts="/host/kh")))
|
||||
provision_git_gate("gw", "bottle1", _plan(_up("foo", known_hosts="/host/kh")))
|
||||
|
||||
cps = [c for c in calls if c[:2] == ["docker", "cp"]]
|
||||
self.assertIn(["docker", "cp", "/host/keys/id", "gw:/git-gate/creds/bottle1/foo-key"], cps)
|
||||
@@ -72,32 +71,32 @@ class TestProvisionGitGate(unittest.TestCase):
|
||||
def test_omits_known_hosts_copy_when_absent(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
with patch(_RUN, side_effect=_recorder(calls)):
|
||||
provision_git_gate(DockerGatewayTransport("gw"), "b1", _plan(_up("foo"))) # no known_hosts
|
||||
provision_git_gate("gw", "b1", _plan(_up("foo"))) # no known_hosts
|
||||
creds_cps = [c for c in calls if c[:2] == ["docker", "cp"] and "/git-gate/creds/" in c[3]]
|
||||
self.assertEqual(1, len(creds_cps)) # only the key, not known_hosts
|
||||
self.assertTrue(creds_cps[0][3].endswith("/foo-key"))
|
||||
|
||||
def test_no_upstreams_is_noop(self) -> None:
|
||||
with patch(_RUN) as m:
|
||||
provision_git_gate(DockerGatewayTransport("gw"), "b1", _plan())
|
||||
provision_git_gate("gw", "b1", _plan())
|
||||
m.assert_not_called()
|
||||
|
||||
def test_raises_on_docker_failure(self) -> None:
|
||||
with patch(_RUN, return_value=_proc(returncode=1, stderr="boom")):
|
||||
with self.assertRaises(GatewayProvisionError):
|
||||
provision_git_gate(DockerGatewayTransport("gw"), "b1", _plan(_up("foo")))
|
||||
provision_git_gate("gw", "b1", _plan(_up("foo")))
|
||||
|
||||
def test_rejects_unsafe_bottle_id_before_any_docker(self) -> None:
|
||||
with patch(_RUN) as m:
|
||||
with self.assertRaises(GatewayProvisionError):
|
||||
provision_git_gate(DockerGatewayTransport("gw"), "../etc", _plan(_up("foo")))
|
||||
provision_git_gate("gw", "../etc", _plan(_up("foo")))
|
||||
m.assert_not_called() # rejected before a single docker call
|
||||
|
||||
|
||||
class TestDeprovision(unittest.TestCase):
|
||||
def test_removes_repo_and_creds(self) -> None:
|
||||
with patch(_RUN, return_value=_proc()) as m:
|
||||
deprovision_git_gate(DockerGatewayTransport("gw"), "b1")
|
||||
deprovision_git_gate("gw", "b1")
|
||||
argv = m.call_args.args[0]
|
||||
self.assertEqual(["docker", "exec", "gw", "rm", "-rf"], argv[:5])
|
||||
self.assertIn("/git/b1", argv)
|
||||
@@ -106,7 +105,7 @@ class TestDeprovision(unittest.TestCase):
|
||||
def test_rejects_unsafe_bottle_id(self) -> None:
|
||||
with patch(_RUN) as m:
|
||||
with self.assertRaises(GatewayProvisionError):
|
||||
deprovision_git_gate(DockerGatewayTransport("gw"), "a/b")
|
||||
deprovision_git_gate("gw", "a/b")
|
||||
m.assert_not_called()
|
||||
|
||||
|
||||
|
||||
@@ -6,7 +6,6 @@ Covers the pure `git_gate_render_gitconfig` renderer and the dynamic
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import socket
|
||||
import tempfile
|
||||
import types
|
||||
import unittest
|
||||
@@ -75,19 +74,6 @@ class TestRenderGitconfig(unittest.TestCase):
|
||||
out = git_gate_render_gitconfig((_entry(),), "1.2.3.4:9418", scheme="http")
|
||||
self.assertIn('[url "http://1.2.3.4:9418/repo.git"]', out)
|
||||
|
||||
def test_identity_token_extraheader_over_http(self) -> None:
|
||||
# Delivered as a URL-scoped http.extraHeader so git-http can enforce
|
||||
# the mandatory (source_ip, token) pair; only over the http transport.
|
||||
out = git_gate_render_gitconfig(
|
||||
(_entry(),), "1.2.3.4:9420", scheme="http", identity_token="TOK123")
|
||||
self.assertIn('[http "http://1.2.3.4:9420/"]', out)
|
||||
self.assertIn("extraHeader = x-bot-bottle-identity: TOK123", out)
|
||||
|
||||
def test_identity_token_omitted_over_git_scheme(self) -> None:
|
||||
out = git_gate_render_gitconfig(
|
||||
(_entry(),), "git-gate", scheme="git", identity_token="TOK123")
|
||||
self.assertNotIn("extraHeader", out)
|
||||
|
||||
def test_remote_key_alias_with_nondefault_port(self) -> None:
|
||||
out = git_gate_render_gitconfig(
|
||||
(_entry(RemoteKey="10.0.0.5", UpstreamPort="2222"),), "git-gate",
|
||||
@@ -127,9 +113,8 @@ class TestProvisionDynamicKey(unittest.TestCase):
|
||||
self.assertEqual(b"PRIVATE-KEY-BYTES", key_file.read_bytes())
|
||||
id_file = Path(d) / "repo-deploy-key-id"
|
||||
self.assertEqual("kid123", id_file.read_text())
|
||||
# owner_repo had .git stripped; title carries globalize_slug(slug) + name
|
||||
hostname = socket.gethostname()
|
||||
self.assertEqual([("o/r", f"bot-bottle:{hostname}-myslug:repo")], fake.created)
|
||||
# owner_repo had .git stripped; title carries slug + name
|
||||
self.assertEqual([("o/r", "bot-bottle:myslug:repo")], fake.created)
|
||||
|
||||
def test_missing_token_raises(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as d, \
|
||||
|
||||
@@ -1,182 +0,0 @@
|
||||
"""Unit: the prebuilt infra-rootfs artifact pull (PRD 0069 Stage 2).
|
||||
|
||||
The launch-host half — version hashing and download/verify/decompress — is
|
||||
what keeps a docker-free host from booting a stale or corrupted rootfs, so the
|
||||
checksum + fail-closed paths are locked here. Network is mocked; no Docker.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import gzip
|
||||
import hashlib
|
||||
import io
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from bot_bottle.backend.firecracker import infra_artifact as ia
|
||||
from bot_bottle.log import Die
|
||||
|
||||
|
||||
def _gz(data: bytes) -> bytes:
|
||||
return gzip.compress(data)
|
||||
|
||||
|
||||
class _FakeNet:
|
||||
"""Map artifact URLs to bytes (or an HTTPError) for urlopen."""
|
||||
|
||||
def __init__(self, responses: "dict[str, bytes | Exception]") -> None:
|
||||
self._responses = responses
|
||||
self.calls: list[str] = []
|
||||
|
||||
def urlopen(self, req: urllib.request.Request, *a: object, **k: object) -> io.BytesIO:
|
||||
url = req.full_url
|
||||
self.calls.append(url)
|
||||
val = self._responses.get(url)
|
||||
if isinstance(val, Exception):
|
||||
raise val
|
||||
if val is None:
|
||||
raise urllib.error.HTTPError(url, 404, "not found", {}, None) # type: ignore[arg-type]
|
||||
return io.BytesIO(val)
|
||||
|
||||
|
||||
class _CacheMixin(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self._env = mock.patch.dict(
|
||||
os.environ,
|
||||
{"BOT_BOTTLE_FC_CACHE": self._tmp.name,
|
||||
"BOT_BOTTLE_INFRA_ARTIFACT_TOKEN": ""},
|
||||
clear=False,
|
||||
)
|
||||
self._env.start()
|
||||
self.addCleanup(self._env.stop)
|
||||
self.addCleanup(self._tmp.cleanup)
|
||||
|
||||
def _serve(self, version: str, gz_bytes: bytes, sha_text: str | None = None):
|
||||
if sha_text is None:
|
||||
sha_text = f"{hashlib.sha256(gz_bytes).hexdigest()} rootfs.ext4.gz\n"
|
||||
net = _FakeNet({
|
||||
ia.artifact_url(version, "rootfs.ext4.gz"): gz_bytes,
|
||||
ia.artifact_url(version, "rootfs.ext4.gz.sha256"): sha_text.encode(),
|
||||
})
|
||||
return mock.patch.object(ia.urllib.request, "urlopen", net.urlopen), net
|
||||
|
||||
|
||||
class TestVersion(unittest.TestCase):
|
||||
def test_deterministic_16_hex(self) -> None:
|
||||
v = ia.infra_artifact_version("#!/bin/sh\ntrue\n")
|
||||
self.assertEqual(v, ia.infra_artifact_version("#!/bin/sh\ntrue\n"))
|
||||
self.assertEqual(16, len(v))
|
||||
int(v, 16) # hex
|
||||
|
||||
def test_init_change_bumps_version(self) -> None:
|
||||
self.assertNotEqual(
|
||||
ia.infra_artifact_version("a"), ia.infra_artifact_version("b"))
|
||||
|
||||
|
||||
class TestVersionInputs(unittest.TestCase):
|
||||
"""The hash must cover *every* file baked into the rootfs, not just `*.py`
|
||||
(`COPY bot_bottle` is wholesale) — else a non-Python change (e.g. the egress
|
||||
entrypoint shell script) leaves the version unchanged and a launch host
|
||||
boots a rootfs whose code differs from its checkout."""
|
||||
|
||||
def _fake_repo(self, root: Path) -> None:
|
||||
pkg = root / "bot_bottle"
|
||||
pkg.mkdir()
|
||||
(pkg / "app.py").write_text("print('hi')\n")
|
||||
(pkg / "egress_entrypoint.sh").write_text("#!/bin/sh\nexec mitmdump\n")
|
||||
(pkg / "netpool.defaults.env").write_text("FOO=1\n")
|
||||
for name in ("Dockerfile.orchestrator", "Dockerfile.gateway", "Dockerfile.infra"):
|
||||
(root / name).write_text(f"FROM scratch # {name}\n")
|
||||
|
||||
def test_non_python_file_change_bumps_version(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root = Path(d)
|
||||
self._fake_repo(root)
|
||||
before = ia.infra_artifact_version("init", repo_root=root)
|
||||
(root / "bot_bottle" / "egress_entrypoint.sh").write_text(
|
||||
"#!/bin/sh\nexec mitmdump --different\n")
|
||||
after = ia.infra_artifact_version("init", repo_root=root)
|
||||
self.assertNotEqual(before, after)
|
||||
|
||||
def test_pyc_and_pycache_ignored(self) -> None:
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
root = Path(d)
|
||||
self._fake_repo(root)
|
||||
before = ia.infra_artifact_version("init", repo_root=root)
|
||||
cache = root / "bot_bottle" / "__pycache__"
|
||||
cache.mkdir()
|
||||
(cache / "app.cpython-312.pyc").write_bytes(b"\x00bytecode")
|
||||
(root / "bot_bottle" / "app.pyc").write_bytes(b"\x00bytecode")
|
||||
after = ia.infra_artifact_version("init", repo_root=root)
|
||||
self.assertEqual(before, after)
|
||||
|
||||
|
||||
class TestEnsureArtifact(_CacheMixin):
|
||||
def test_downloads_verifies_and_caches(self) -> None:
|
||||
version = "deadbeef00000000"
|
||||
gz = _gz(b"fake ext4 bytes")
|
||||
patcher, net = self._serve(version, gz)
|
||||
with patcher:
|
||||
path = ia.ensure_artifact_gz(version)
|
||||
self.assertTrue(path.is_file())
|
||||
self.assertEqual(gz, path.read_bytes())
|
||||
first_calls = len(net.calls)
|
||||
# Second call is a cache hit — no further network.
|
||||
ia.ensure_artifact_gz(version)
|
||||
self.assertEqual(first_calls, len(net.calls))
|
||||
|
||||
def test_checksum_mismatch_fails_closed(self) -> None:
|
||||
version = "beefbeefbeefbeef"
|
||||
gz = _gz(b"payload")
|
||||
patcher, _ = self._serve(version, gz, sha_text="0" * 64 + " rootfs.ext4.gz\n")
|
||||
with patcher:
|
||||
with self.assertRaises(Die) as ctx:
|
||||
ia.ensure_artifact_gz(version)
|
||||
self.assertIn("checksum mismatch", str(ctx.exception.message))
|
||||
# nothing left cached to accidentally boot
|
||||
self.assertFalse((ia._cache_root(version) / "rootfs.ext4.gz").exists())
|
||||
|
||||
def test_missing_artifact_points_at_publish(self) -> None:
|
||||
version = "0000000000000000"
|
||||
net = _FakeNet({}) # everything 404s
|
||||
with mock.patch.object(ia.urllib.request, "urlopen", net.urlopen):
|
||||
with self.assertRaises(Die) as ctx:
|
||||
ia.ensure_artifact_gz(version)
|
||||
self.assertIn("publish_infra", str(ctx.exception.message))
|
||||
|
||||
def test_materialize_gunzips_to_dest(self) -> None:
|
||||
version = "1234123412341234"
|
||||
raw = b"the real rootfs contents" * 100
|
||||
patcher, _ = self._serve(version, _gz(raw))
|
||||
with patcher, tempfile.TemporaryDirectory() as d:
|
||||
dest = Path(d) / "rootfs.ext4"
|
||||
ia.materialize_ext4(version, dest)
|
||||
self.assertEqual(raw, dest.read_bytes())
|
||||
|
||||
|
||||
class TestConfig(unittest.TestCase):
|
||||
def test_base_and_owner_overridable(self) -> None:
|
||||
with mock.patch.dict(os.environ, {
|
||||
"BOT_BOTTLE_INFRA_ARTIFACT_BASE": "https://mirror.example/",
|
||||
"BOT_BOTTLE_INFRA_ARTIFACT_OWNER": "acme",
|
||||
}):
|
||||
url = ia.artifact_url("v1", "rootfs.ext4.gz")
|
||||
self.assertEqual(
|
||||
"https://mirror.example/api/packages/acme/generic/"
|
||||
"bot-bottle-firecracker-infra/v1/rootfs.ext4.gz", url)
|
||||
|
||||
def test_local_build_flag(self) -> None:
|
||||
with mock.patch.dict(os.environ, {"BOT_BOTTLE_INFRA_BUILD": "local"}):
|
||||
self.assertTrue(ia.local_build_requested())
|
||||
with mock.patch.dict(os.environ, {"BOT_BOTTLE_INFRA_BUILD": ""}):
|
||||
self.assertFalse(ia.local_build_requested())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -1,136 +0,0 @@
|
||||
"""Unit: macOS consolidated launch — register-after-start (PRD 0070)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import MagicMock, Mock, patch
|
||||
|
||||
from bot_bottle.backend.macos_container.consolidated_launch import (
|
||||
GatewayEndpoint,
|
||||
ensure_gateway,
|
||||
register_agent,
|
||||
teardown_consolidated,
|
||||
)
|
||||
from bot_bottle.egress import EgressPlan, EgressRoute
|
||||
from bot_bottle.git_gate import GitGatePlan
|
||||
from bot_bottle.orchestrator.client import RegisteredBottle
|
||||
|
||||
_MOD = "bot_bottle.backend.macos_container.consolidated_launch"
|
||||
|
||||
|
||||
def _egress_plan() -> EgressPlan:
|
||||
return EgressPlan(
|
||||
slug="demo", routes_path=Path("/x"),
|
||||
routes=(EgressRoute(host="api.example.com"),), token_env_map={},
|
||||
)
|
||||
|
||||
|
||||
def _git_plan() -> GitGatePlan:
|
||||
return GitGatePlan(
|
||||
slug="demo", entrypoint_script=Path(), hook_script=Path(),
|
||||
access_hook_script=Path(), upstreams=(),
|
||||
)
|
||||
|
||||
|
||||
def _endpoint() -> GatewayEndpoint:
|
||||
return GatewayEndpoint(
|
||||
orchestrator_url="http://192.168.128.2:8099",
|
||||
gateway_ip="192.168.128.3",
|
||||
gateway_ca_pem="-----BEGIN CERTIFICATE-----\n",
|
||||
network="bot-bottle-mac-gateway",
|
||||
)
|
||||
|
||||
|
||||
def _client() -> Mock:
|
||||
c = Mock()
|
||||
c.register_bottle.return_value = RegisteredBottle("b1", "tok")
|
||||
return c
|
||||
|
||||
|
||||
class TestEnsureGateway(unittest.TestCase):
|
||||
def _run(self, service: MagicMock) -> GatewayEndpoint:
|
||||
with patch(f"{_MOD}.MacosInfraService", return_value=service):
|
||||
return ensure_gateway()
|
||||
|
||||
def _service(self) -> MagicMock:
|
||||
from bot_bottle.backend.macos_container.infra import InfraEndpoint
|
||||
service = MagicMock()
|
||||
service.ensure_running.return_value = InfraEndpoint(
|
||||
control_plane_url="http://192.168.128.2:8099",
|
||||
gateway_ip="192.168.128.2",
|
||||
)
|
||||
service.network = "bot-bottle-mac-gateway"
|
||||
service.ca_cert_pem.return_value = "PEM"
|
||||
return service
|
||||
|
||||
def test_reports_gateway_endpoint(self) -> None:
|
||||
endpoint = self._run(self._service())
|
||||
self.assertEqual("http://192.168.128.2:8099", endpoint.orchestrator_url)
|
||||
self.assertEqual("192.168.128.2", endpoint.gateway_ip)
|
||||
self.assertEqual("PEM", endpoint.gateway_ca_pem)
|
||||
self.assertEqual("bot-bottle-mac-gateway", endpoint.network)
|
||||
|
||||
def test_control_plane_and_gateway_share_one_address(self) -> None:
|
||||
"""One infra container hosts both, so the gateway IP and the
|
||||
control-plane host are the same."""
|
||||
endpoint = self._run(self._service())
|
||||
self.assertEqual(
|
||||
endpoint.gateway_ip,
|
||||
endpoint.orchestrator_url.split("://")[1].split(":")[0],
|
||||
)
|
||||
|
||||
|
||||
class TestRegisterAgent(unittest.TestCase):
|
||||
def _run(
|
||||
self, client: Mock, provision: Mock | None = None,
|
||||
*, source_ip: str = "192.168.128.9",
|
||||
):
|
||||
with patch(f"{_MOD}.OrchestratorClient", return_value=client), \
|
||||
patch(f"{_MOD}.provision_git_gate", provision or Mock()):
|
||||
return register_agent(
|
||||
_egress_plan(), _git_plan(),
|
||||
source_ip=source_ip, endpoint=_endpoint(), image_ref="img:1",
|
||||
)
|
||||
|
||||
def test_registers_by_the_address_read_from_the_live_container(self) -> None:
|
||||
"""The attribution key is the DHCP-assigned address the caller read
|
||||
back — there is no --ip to pin it up front."""
|
||||
client = _client()
|
||||
ctx = self._run(client, source_ip="192.168.128.9")
|
||||
self.assertEqual("192.168.128.9", ctx.source_ip)
|
||||
self.assertEqual("192.168.128.9", client.register_bottle.call_args.args[0])
|
||||
|
||||
def test_returns_identity_token_and_bottle_id(self) -> None:
|
||||
ctx = self._run(_client())
|
||||
self.assertEqual("b1", ctx.bottle_id)
|
||||
self.assertEqual("tok", ctx.identity_token)
|
||||
|
||||
def test_provisions_git_gate_for_the_registered_bottle(self) -> None:
|
||||
provision = Mock()
|
||||
self._run(_client(), provision)
|
||||
self.assertEqual("b1", provision.call_args.args[1])
|
||||
|
||||
def test_rolls_registration_back_when_provisioning_fails(self) -> None:
|
||||
"""A provisioning failure must not leave an orphan registration
|
||||
holding the source IP."""
|
||||
client = _client()
|
||||
provision = Mock(side_effect=RuntimeError("boom"))
|
||||
with self.assertRaises(RuntimeError):
|
||||
self._run(client, provision)
|
||||
client.teardown_bottle.assert_called_once_with("b1")
|
||||
|
||||
|
||||
class TestTeardown(unittest.TestCase):
|
||||
def test_deregisters_and_deprovisions(self) -> None:
|
||||
client = Mock()
|
||||
deprovision = Mock()
|
||||
with patch(f"{_MOD}.OrchestratorClient", return_value=client), \
|
||||
patch(f"{_MOD}.deprovision_git_gate", deprovision):
|
||||
teardown_consolidated("b1", orchestrator_url="http://o:8099")
|
||||
client.teardown_bottle.assert_called_once_with("b1")
|
||||
self.assertEqual("b1", deprovision.call_args.args[1])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -43,31 +43,10 @@ class TestMacosContainerCleanup(unittest.TestCase):
|
||||
|
||||
|
||||
class TestMacosContainerEnumerate(unittest.TestCase):
|
||||
"""The backend launches bottles again (PRD 0070), so enumeration is real
|
||||
rather than the disabled-era stub. These must not shell out: `container`
|
||||
does not exist on the Linux CI host."""
|
||||
|
||||
def _enumerate(self, stdout: str, returncode: int = 0):
|
||||
completed = enum_mod.subprocess.CompletedProcess(
|
||||
args=[], returncode=returncode, stdout=stdout, stderr="",
|
||||
)
|
||||
with patch.object(enum_mod.subprocess, "run", return_value=completed), \
|
||||
patch.object(enum_mod, "read_metadata", return_value=None):
|
||||
return enum_mod.enumerate_active()
|
||||
|
||||
def test_lists_agent_containers_by_slug(self):
|
||||
agents = self._enumerate("bot-bottle-dev-abc\nunrelated\n")
|
||||
self.assertEqual(["dev-abc"], [a.slug for a in agents])
|
||||
self.assertEqual(["macos-container"], [a.backend_name for a in agents])
|
||||
|
||||
def test_excludes_the_infra_singleton(self):
|
||||
"""The infra container shares the bot-bottle- prefix but is
|
||||
infrastructure — listing it would invent an agent per host."""
|
||||
agents = self._enumerate("bot-bottle-mac-infra\nbot-bottle-dev-abc\n")
|
||||
self.assertEqual(["dev-abc"], [a.slug for a in agents])
|
||||
|
||||
def test_empty_when_the_cli_fails(self):
|
||||
self.assertEqual([], self._enumerate("", returncode=1))
|
||||
def test_enumerate_active_is_empty_while_disabled(self):
|
||||
# The macOS backend is disabled during the companion-container removal cleanup
|
||||
# (#385); it launches nothing, so there is nothing to enumerate.
|
||||
self.assertEqual([], enum_mod.enumerate_active())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
||||
@@ -1,210 +0,0 @@
|
||||
"""Unit: macOS agent run wiring — the attribution invariant + token delivery
|
||||
(PRD 0070).
|
||||
|
||||
Replaces the argv coverage from the per-bottle companion-container era
|
||||
(`test_macos_container_launch.py`, removed with that architecture in #385).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from types import SimpleNamespace
|
||||
from typing import cast
|
||||
from unittest.mock import patch
|
||||
|
||||
from bot_bottle.backend.macos_container.bottle import MacosContainerBottle
|
||||
from bot_bottle.backend.macos_container.bottle_plan import MacosContainerBottlePlan
|
||||
from bot_bottle.backend.macos_container.consolidated_launch import GatewayEndpoint
|
||||
from bot_bottle.backend.macos_container.launch import (
|
||||
_agent_run_argv,
|
||||
_identity_proxy_env,
|
||||
_proxy_url,
|
||||
)
|
||||
from bot_bottle.manifest import ManifestIndex
|
||||
|
||||
_BOTTLE = "bot_bottle.backend.macos_container.bottle"
|
||||
|
||||
_MANIFEST = ManifestIndex.from_json_obj({
|
||||
"bottles": {"dev": {}},
|
||||
"agents": {"demo": {"skills": [], "prompt": "", "bottle": "dev"}},
|
||||
}).load_for_agent("demo")
|
||||
|
||||
|
||||
def _endpoint() -> GatewayEndpoint:
|
||||
return GatewayEndpoint(
|
||||
orchestrator_url="http://192.168.128.2:8099",
|
||||
gateway_ip="192.168.128.3",
|
||||
gateway_ca_pem="PEM",
|
||||
network="bot-bottle-mac-gateway",
|
||||
)
|
||||
|
||||
|
||||
def _plan(
|
||||
stage_dir: Path,
|
||||
*,
|
||||
agent_git_gate_url: str = "",
|
||||
agent_supervise_url: str = "",
|
||||
) -> MacosContainerBottlePlan:
|
||||
routes_path = stage_dir / "routes.yaml"
|
||||
routes_path.write_text("routes: []\n", encoding="utf-8")
|
||||
ca_path = stage_dir / "gateway-ca.pem"
|
||||
ca_path.write_text("ca\n", encoding="utf-8")
|
||||
egress_plan = SimpleNamespace(
|
||||
mitmproxy_ca_host_path=ca_path,
|
||||
routes_path=routes_path,
|
||||
routes=("route",),
|
||||
token_env_map={"EGRESS_TOKEN_0": "HOST_TOKEN"},
|
||||
canary="",
|
||||
canary_env="",
|
||||
)
|
||||
return cast(MacosContainerBottlePlan, SimpleNamespace(
|
||||
spec=SimpleNamespace(),
|
||||
manifest=_MANIFEST,
|
||||
stage_dir=stage_dir,
|
||||
slug="dev-abc",
|
||||
container_name="bot-bottle-dev-abc",
|
||||
image="bot-bottle-agent:latest",
|
||||
forwarded_env={"OAUTH_TOKEN": "host-value"},
|
||||
egress_plan=egress_plan,
|
||||
git_gate_plan=SimpleNamespace(upstreams=()),
|
||||
supervise_plan=None,
|
||||
agent_provision=SimpleNamespace(
|
||||
guest_env={"LITERAL": "value"},
|
||||
provisioned_env={},
|
||||
),
|
||||
agent_git_gate_url=agent_git_gate_url,
|
||||
agent_supervise_url=agent_supervise_url,
|
||||
))
|
||||
|
||||
|
||||
class TestAgentRunArgv(unittest.TestCase):
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.argv = _agent_run_argv(_plan(Path(self._tmp.name)), _endpoint())
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self._tmp.cleanup()
|
||||
|
||||
def test_drops_net_raw(self) -> None:
|
||||
"""The attribution invariant: Apple grants CAP_NET_RAW by default,
|
||||
which would let an agent forge a neighbour's source address with a raw
|
||||
socket and be attributed as that bottle."""
|
||||
self.assertIn("--cap-drop", self.argv)
|
||||
self.assertEqual("CAP_NET_RAW", self.argv[self.argv.index("--cap-drop") + 1])
|
||||
|
||||
def test_attaches_to_the_shared_gateway_network(self) -> None:
|
||||
self.assertEqual(
|
||||
"bot-bottle-mac-gateway", self.argv[self.argv.index("--network") + 1],
|
||||
)
|
||||
|
||||
def test_never_pins_an_ip(self) -> None:
|
||||
"""Apple Container 1.0.0 has no --ip: the address is DHCP-assigned and
|
||||
read back after start."""
|
||||
self.assertNotIn("--ip", self.argv)
|
||||
|
||||
def test_run_time_proxy_carries_no_identity_token(self) -> None:
|
||||
"""The token is minted by registration, which happens after this run —
|
||||
so it cannot be here. `/resolve` denies the token-less pair (#366),
|
||||
which is the safe direction; the real value arrives at exec time."""
|
||||
joined = " ".join(self.argv)
|
||||
self.assertIn(f"HTTP_PROXY={_proxy_url('192.168.128.3')}", joined)
|
||||
self.assertNotIn("bottle:", joined)
|
||||
|
||||
def test_gateway_bypasses_the_proxy(self) -> None:
|
||||
"""git-http + supervise live on the gateway and must be reached
|
||||
directly, not through its own egress proxy."""
|
||||
entry = next(a for a in self.argv if a.startswith("NO_PROXY="))
|
||||
self.assertIn("192.168.128.3", entry)
|
||||
|
||||
def test_forwarded_secrets_stay_off_argv(self) -> None:
|
||||
"""Bare name → inherited from the run process env, so the value never
|
||||
lands on the command line."""
|
||||
self.assertIn("OAUTH_TOKEN", self.argv)
|
||||
self.assertNotIn("host-value", " ".join(self.argv))
|
||||
|
||||
def test_agent_init_is_a_no_op(self) -> None:
|
||||
"""Every agent command arrives via `container exec`; the init process
|
||||
just holds the container open."""
|
||||
self.assertEqual("sleep", self.argv[-2])
|
||||
|
||||
|
||||
class TestIdentityTokenDelivery(unittest.TestCase):
|
||||
def test_exec_env_carries_the_token_as_proxy_credentials(self) -> None:
|
||||
env = _identity_proxy_env(_endpoint(), "s3cret")
|
||||
self.assertEqual(
|
||||
"http://bottle:s3cret@192.168.128.3:9099", env["HTTP_PROXY"],
|
||||
)
|
||||
self.assertEqual(env["HTTP_PROXY"], env["https_proxy"])
|
||||
|
||||
def test_no_token_means_no_override(self) -> None:
|
||||
self.assertEqual({}, _identity_proxy_env(_endpoint(), ""))
|
||||
|
||||
def test_token_value_never_reaches_argv(self) -> None:
|
||||
"""`ps` is world-readable: the token rides the child env behind a bare
|
||||
`--env` name, never the command line."""
|
||||
bottle = MacosContainerBottle(
|
||||
"bot-bottle-demo", lambda: None, None,
|
||||
exec_env=_identity_proxy_env(_endpoint(), "s3cret"),
|
||||
)
|
||||
argv = bottle.agent_argv(["--help"], tty=False)
|
||||
self.assertNotIn("s3cret", " ".join(argv))
|
||||
self.assertIn("HTTP_PROXY", argv)
|
||||
self.assertEqual("--env", argv[argv.index("HTTP_PROXY") - 1])
|
||||
|
||||
def test_bottle_without_exec_env_is_unchanged(self) -> None:
|
||||
bottle = MacosContainerBottle("bot-bottle-demo", lambda: None, None)
|
||||
argv = bottle.agent_argv(["--help"], tty=False)
|
||||
self.assertNotIn("--env", argv)
|
||||
|
||||
|
||||
class TestPlanIdentityToken(unittest.TestCase):
|
||||
"""git-gate's gitconfig extraHeader and the supervise MCP --header read
|
||||
`getattr(plan, "identity_token", "")` at provision time and both bypass the
|
||||
egress proxy (NO_PROXY), so the exec-time proxy token never reaches them —
|
||||
the plan must carry the token or /resolve fail-closes and git + supervise
|
||||
are denied on macOS."""
|
||||
|
||||
def test_macos_plan_has_the_identity_token_field(self) -> None:
|
||||
from dataclasses import fields
|
||||
|
||||
from bot_bottle.backend.macos_container.bottle_plan import (
|
||||
MacosContainerBottlePlan,
|
||||
)
|
||||
names = {f.name for f in fields(MacosContainerBottlePlan)}
|
||||
self.assertIn("identity_token", names)
|
||||
|
||||
def test_matches_the_docker_plan(self) -> None:
|
||||
"""Both consolidated backends must expose identity_token so a shared
|
||||
provision-time consumer degrades to neither backend silently."""
|
||||
from dataclasses import fields
|
||||
|
||||
from bot_bottle.backend.docker.bottle_plan import DockerBottlePlan
|
||||
from bot_bottle.backend.macos_container.bottle_plan import (
|
||||
MacosContainerBottlePlan,
|
||||
)
|
||||
docker = {f.name for f in fields(DockerBottlePlan)}
|
||||
macos = {f.name for f in fields(MacosContainerBottlePlan)}
|
||||
self.assertIn("identity_token", docker & macos)
|
||||
|
||||
def test_provisioning_exec_also_carries_the_token(self) -> None:
|
||||
"""`provision` runs through `exec`; a provider whose provision step
|
||||
fetches anything would otherwise egress token-less and be denied."""
|
||||
bottle = MacosContainerBottle(
|
||||
"bot-bottle-demo", lambda: None, None,
|
||||
exec_env=_identity_proxy_env(_endpoint(), "s3cret"),
|
||||
)
|
||||
with patch(f"{_BOTTLE}.subprocess.run") as run:
|
||||
run.return_value = SimpleNamespace(returncode=0, stdout="", stderr="")
|
||||
bottle.exec("echo hi")
|
||||
argv, kwargs = run.call_args.args[0], run.call_args.kwargs
|
||||
self.assertIn("HTTP_PROXY", argv)
|
||||
self.assertNotIn("s3cret", " ".join(argv))
|
||||
self.assertEqual(
|
||||
"http://bottle:s3cret@192.168.128.3:9099", kwargs["env"]["HTTP_PROXY"],
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -273,55 +273,5 @@ resolver #2
|
||||
)
|
||||
|
||||
|
||||
def _completed(stdout: str, returncode: int = 0):
|
||||
return util.subprocess.CompletedProcess(args=[], returncode=returncode, stdout=stdout, stderr="")
|
||||
|
||||
|
||||
class TestInspectDigests(unittest.TestCase):
|
||||
"""image_digest and container_image_digest must read the SAME field shape
|
||||
(a `descriptor.digest`) so a running container and its image are
|
||||
comparable. An asymmetry recreates the gateway on every launch."""
|
||||
|
||||
_IMAGE = '{"configuration": {"descriptor": {"digest": "sha256:abc123"}}, "id": "abc123"}'
|
||||
_CONTAINER = '[{"configuration": {"image": {"descriptor": {"digest": "sha256:abc123"}}}}]'
|
||||
|
||||
def test_image_and_container_digests_agree_for_the_same_image(self):
|
||||
with patch.object(util, "run_container_argv") as run:
|
||||
run.return_value = _completed(self._IMAGE)
|
||||
img = util.image_digest("bot-bottle-gateway:latest")
|
||||
run.return_value = _completed(self._CONTAINER)
|
||||
ctr = util.container_image_digest("bot-bottle-mac-gateway")
|
||||
self.assertEqual("abc123", img)
|
||||
self.assertEqual(img, ctr)
|
||||
|
||||
def test_image_digest_never_falls_back_to_a_tag_or_id(self):
|
||||
"""The old `id` fallback could yield a value container_image_digest
|
||||
can't produce, permanently mismatching. Missing descriptor → '' (don't
|
||||
churn), never a stray id/tag."""
|
||||
with patch.object(util, "run_container_argv") as run:
|
||||
run.return_value = _completed('{"id": "bot-bottle-gateway:latest"}')
|
||||
self.assertEqual("", util.image_digest("bot-bottle-gateway:latest"))
|
||||
|
||||
def test_unreadable_inspect_returns_empty(self):
|
||||
with patch.object(util, "run_container_argv") as run:
|
||||
run.return_value = _completed("", returncode=1)
|
||||
self.assertEqual("", util.image_digest("x"))
|
||||
self.assertEqual("", util.container_image_digest("x"))
|
||||
self.assertEqual({}, util.container_env("x"))
|
||||
|
||||
|
||||
class TestWaitContainerIpv4(unittest.TestCase):
|
||||
def test_returns_address_once_dhcp_assigns_it(self):
|
||||
with patch.object(util, "try_container_ipv4_on_network", side_effect=["", "", "192.168.128.4"]), \
|
||||
patch.object(util.time, "sleep"):
|
||||
ip = util.wait_container_ipv4_on_network("c", "net", timeout=5, poll=0)
|
||||
self.assertEqual("192.168.128.4", ip)
|
||||
|
||||
def test_returns_empty_on_timeout(self):
|
||||
with patch.object(util, "try_container_ipv4_on_network", return_value=""), \
|
||||
patch.object(util.time, "sleep"):
|
||||
self.assertEqual("", util.wait_container_ipv4_on_network("c", "net", timeout=-1))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,168 +0,0 @@
|
||||
"""Unit: the single macOS infra container (control plane + gateway, PRD 0070)."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import Mock, patch
|
||||
|
||||
from bot_bottle.backend.macos_container.infra import (
|
||||
INFRA_DB_VOLUME,
|
||||
MacosInfraService,
|
||||
OrchestratorStartError,
|
||||
probe_control_plane_url,
|
||||
)
|
||||
|
||||
_INFRA = "bot_bottle.backend.macos_container.infra"
|
||||
|
||||
|
||||
def _ok(stdout: str = "") -> Mock:
|
||||
return Mock(returncode=0, stdout=stdout, stderr="")
|
||||
|
||||
|
||||
def _fail(stderr: str = "boom") -> Mock:
|
||||
return Mock(returncode=1, stdout="", stderr=stderr)
|
||||
|
||||
|
||||
class TestInfraRun(unittest.TestCase):
|
||||
def _run_container(self, svc: MacosInfraService) -> list[str]:
|
||||
run = Mock(return_value=_ok())
|
||||
|
||||
def _spec(src: str, tgt: str, readonly: bool = False) -> str:
|
||||
return f"type=bind,source={src},target={tgt}" + (
|
||||
",readonly" if readonly else "")
|
||||
|
||||
with patch(f"{_INFRA}.container_mod") as mod, \
|
||||
patch(f"{_INFRA}.ensure_networks"):
|
||||
mod.dns_server.return_value = "1.1.1.1"
|
||||
mod.bind_mount_spec.side_effect = _spec
|
||||
mod.run_container_argv = run
|
||||
svc._run_container("h1")
|
||||
return run.call_args.args[0]
|
||||
|
||||
def test_single_container_runs_both_processes(self) -> None:
|
||||
"""The whole point: one container starts the control plane AND the
|
||||
gateway daemons, so one kernel owns the DB."""
|
||||
argv = self._run_container(MacosInfraService(repo_root=Path("/r")))
|
||||
script = argv[-1]
|
||||
self.assertIn("bot_bottle.orchestrator", script)
|
||||
self.assertIn("gateway_init.py", script)
|
||||
self.assertIn("127.0.0.1", script) # they reach each other on loopback
|
||||
|
||||
def test_db_is_a_container_only_volume(self) -> None:
|
||||
"""No host bind-mount of the DB — a named volume only this container
|
||||
mounts, so the DB is never written by two kernels."""
|
||||
argv = self._run_container(MacosInfraService(repo_root=Path("/r")))
|
||||
vols = [argv[i + 1] for i, a in enumerate(argv) if a == "--volume"]
|
||||
self.assertTrue(any(v.startswith(f"{INFRA_DB_VOLUME}:") for v in vols))
|
||||
# The repo source is bind-mounted read-only; the DB is not a bind mount.
|
||||
mounts = [argv[i + 1] for i, a in enumerate(argv) if a == "--mount"]
|
||||
self.assertTrue(all("bot-bottle.db" not in m for m in mounts))
|
||||
|
||||
def test_nat_network_precedes_the_host_only_network(self) -> None:
|
||||
argv = self._run_container(MacosInfraService(repo_root=Path("/r")))
|
||||
nets = [argv[i + 1] for i, a in enumerate(argv) if a == "--network"]
|
||||
self.assertEqual(["bot-bottle-mac-egress", "bot-bottle-mac-gateway"], nets)
|
||||
|
||||
def test_source_hash_is_labelled_for_recreate(self) -> None:
|
||||
argv = self._run_container(MacosInfraService(repo_root=Path("/r")))
|
||||
self.assertIn("BOT_BOTTLE_SOURCE_HASH=h1", argv)
|
||||
|
||||
def test_start_failure_raises(self) -> None:
|
||||
svc = MacosInfraService(repo_root=Path("/r"))
|
||||
with patch(f"{_INFRA}.container_mod") as mod, \
|
||||
patch(f"{_INFRA}.ensure_networks"):
|
||||
mod.dns_server.return_value = "1.1.1.1"
|
||||
mod.bind_mount_spec.return_value = "m"
|
||||
mod.run_container_argv = Mock(return_value=_fail())
|
||||
with self.assertRaises(OrchestratorStartError):
|
||||
svc._run_container("h1")
|
||||
|
||||
|
||||
class TestInfraEnsureRunning(unittest.TestCase):
|
||||
def test_current_healthy_container_is_left_alone(self) -> None:
|
||||
"""Idempotent singleton: N launches must not churn the infra container
|
||||
and drop every live bottle's control plane."""
|
||||
svc = MacosInfraService(repo_root=Path("/r"))
|
||||
run = Mock()
|
||||
with patch(f"{_INFRA}.container_mod") as mod, \
|
||||
patch(f"{_INFRA}.source_hash", return_value="h1"), \
|
||||
patch.object(svc, "_run_container", run), \
|
||||
patch.object(svc, "is_healthy", return_value=True):
|
||||
mod.container_is_running.return_value = True
|
||||
mod.container_env.return_value = {"BOT_BOTTLE_SOURCE_HASH": "h1"}
|
||||
mod.try_container_ipv4_on_network.return_value = "192.168.128.2"
|
||||
endpoint = svc.ensure_running()
|
||||
run.assert_not_called()
|
||||
self.assertEqual("http://192.168.128.2:8099", endpoint.control_plane_url)
|
||||
self.assertEqual("192.168.128.2", endpoint.gateway_ip)
|
||||
|
||||
def test_changed_source_recreates(self) -> None:
|
||||
svc = MacosInfraService(repo_root=Path("/r"))
|
||||
run = Mock()
|
||||
with patch(f"{_INFRA}.container_mod") as mod, \
|
||||
patch(f"{_INFRA}.source_hash", return_value="h2"), \
|
||||
patch.object(svc, "ensure_built"), \
|
||||
patch.object(svc, "_run_container", run), \
|
||||
patch.object(svc, "is_healthy", return_value=True):
|
||||
mod.container_is_running.return_value = True
|
||||
mod.container_env.return_value = {"BOT_BOTTLE_SOURCE_HASH": "h1"}
|
||||
mod.try_container_ipv4_on_network.return_value = "192.168.128.2"
|
||||
svc.ensure_running()
|
||||
run.assert_called_once()
|
||||
|
||||
def test_wedged_but_current_container_is_recreated(self) -> None:
|
||||
"""Current source but a dead HTTP server must be recreated, not polled
|
||||
to death forever — health, not just the source label, gates reuse."""
|
||||
svc = MacosInfraService(repo_root=Path("/r"))
|
||||
run = Mock()
|
||||
health = Mock(side_effect=[False, True])
|
||||
with patch(f"{_INFRA}.container_mod") as mod, \
|
||||
patch(f"{_INFRA}.source_hash", return_value="h1"), \
|
||||
patch.object(svc, "ensure_built"), \
|
||||
patch.object(svc, "_run_container", run), \
|
||||
patch.object(svc, "is_healthy", health):
|
||||
mod.container_is_running.return_value = True
|
||||
mod.container_env.return_value = {"BOT_BOTTLE_SOURCE_HASH": "h1"}
|
||||
mod.try_container_ipv4_on_network.return_value = "192.168.128.2"
|
||||
svc.ensure_running()
|
||||
run.assert_called_once()
|
||||
|
||||
def test_never_healthy_raises(self) -> None:
|
||||
svc = MacosInfraService(repo_root=Path("/r"))
|
||||
with patch(f"{_INFRA}.container_mod") as mod, \
|
||||
patch(f"{_INFRA}.source_hash", return_value="h1"), \
|
||||
patch.object(svc, "ensure_built"), \
|
||||
patch.object(svc, "_run_container"), \
|
||||
patch.object(svc, "is_healthy", return_value=False):
|
||||
mod.container_is_running.return_value = False
|
||||
mod.try_container_ipv4_on_network.return_value = "192.168.128.2"
|
||||
with self.assertRaises(OrchestratorStartError):
|
||||
svc.ensure_running(startup_timeout=0.01)
|
||||
|
||||
|
||||
class TestCaCertPem(unittest.TestCase):
|
||||
def test_reads_ca_out_of_the_container(self) -> None:
|
||||
svc = MacosInfraService(repo_root=Path("/r"))
|
||||
with patch(f"{_INFRA}.container_mod") as mod:
|
||||
mod.run_container_argv.return_value = _ok("-----BEGIN CERTIFICATE-----\n")
|
||||
pem = svc.ca_cert_pem()
|
||||
self.assertTrue(pem.startswith("-----BEGIN CERTIFICATE-----"))
|
||||
argv = mod.run_container_argv.call_args.args[0]
|
||||
self.assertEqual(["container", "exec", "bot-bottle-mac-infra", "cat"], argv[:4])
|
||||
|
||||
|
||||
class TestProbeControlPlane(unittest.TestCase):
|
||||
def test_returns_url_when_running(self) -> None:
|
||||
with patch(f"{_INFRA}.container_mod") as mod:
|
||||
mod.try_container_ipv4_on_network.return_value = "192.168.128.2"
|
||||
self.assertEqual("http://192.168.128.2:8099", probe_control_plane_url())
|
||||
|
||||
def test_empty_when_absent(self) -> None:
|
||||
with patch(f"{_INFRA}.container_mod") as mod:
|
||||
mod.try_container_ipv4_on_network.return_value = ""
|
||||
self.assertEqual("", probe_control_plane_url())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -11,22 +11,13 @@ import secrets
|
||||
import tempfile
|
||||
import threading
|
||||
import unittest
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from bot_bottle.orchestrator.broker import StubBroker
|
||||
from bot_bottle.orchestrator.control_plane import dispatch, make_server
|
||||
from bot_bottle.orchestrator.registry import RegistryStore
|
||||
from bot_bottle.orchestrator.service import Orchestrator
|
||||
from bot_bottle.store_manager import StoreManager
|
||||
from bot_bottle.supervise import (
|
||||
Proposal,
|
||||
TOOL_EGRESS_ALLOW,
|
||||
sha256_hex,
|
||||
write_proposal,
|
||||
)
|
||||
|
||||
|
||||
def _body(obj: object) -> bytes:
|
||||
@@ -180,25 +171,13 @@ class TestDispatch(unittest.TestCase):
|
||||
)
|
||||
self.assertEqual(400, status)
|
||||
|
||||
def test_resolve_without_token_denies(self) -> None:
|
||||
# Mandatory token: source-IP alone no longer resolves (fail-closed 403).
|
||||
dispatch(
|
||||
def test_resolve_without_token_by_source_ip(self) -> None:
|
||||
_, reg = dispatch(
|
||||
self.orch, "POST", "/bottles",
|
||||
_body({"source_ip": "10.243.0.5", "policy": "P"}),
|
||||
)
|
||||
status, _ = dispatch(
|
||||
self.orch, "POST", "/resolve", _body({"source_ip": "10.243.0.5"})
|
||||
)
|
||||
self.assertEqual(403, status)
|
||||
|
||||
def test_resolve_with_matching_token(self) -> None:
|
||||
_, reg = dispatch(
|
||||
self.orch, "POST", "/bottles",
|
||||
_body({"source_ip": "10.243.0.6", "policy": "P"}),
|
||||
)
|
||||
status, payload = dispatch(
|
||||
self.orch, "POST", "/resolve",
|
||||
_body({"source_ip": "10.243.0.6", "identity_token": reg["identity_token"]}),
|
||||
self.orch, "POST", "/resolve", _body({"source_ip": "10.243.0.5"})
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertEqual(reg["bottle_id"], payload["bottle_id"])
|
||||
@@ -244,146 +223,5 @@ class TestServerRoundTrip(unittest.TestCase):
|
||||
self.assertEqual(reg["bottle_id"], attr["bottle_id"])
|
||||
|
||||
|
||||
class TestControlPlaneAuth(unittest.TestCase):
|
||||
"""The per-host control-plane secret (issue #400): every route but /health
|
||||
is a trusted-caller op an agent must not be able to drive just because it
|
||||
can reach the port."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
self.addCleanup(self._tmp.cleanup)
|
||||
self.orch = _orchestrator(Path(self._tmp.name) / "r.db")
|
||||
|
||||
def test_health_is_public_even_unauthorized(self) -> None:
|
||||
status, _ = dispatch(self.orch, "GET", "/health", b"", authorized=False)
|
||||
self.assertEqual(200, status)
|
||||
|
||||
def test_unauthorized_denies_every_other_route(self) -> None:
|
||||
for method, path, body in [
|
||||
("GET", "/bottles", b""),
|
||||
("POST", "/bottles", _body({"source_ip": "10.0.0.1"})),
|
||||
("PUT", "/bottles/x/policy", _body({"policy": "routes: []"})),
|
||||
("DELETE", "/bottles/x", b""),
|
||||
("POST", "/resolve", _body({"source_ip": "10.0.0.1", "identity_token": "t"})),
|
||||
("POST", "/attribute", _body({"source_ip": "10.0.0.1", "identity_token": "t"})),
|
||||
("GET", "/supervise/proposals", b""),
|
||||
("POST", "/supervise/respond", _body({"proposal_id": "p", "bottle_slug": "s", "decision": "approve"})),
|
||||
]:
|
||||
status, _ = dispatch(self.orch, method, path, body, authorized=False)
|
||||
self.assertEqual(401, status, f"{method} {path} should be 401 unauthorized")
|
||||
|
||||
def test_deny_happens_before_the_registry_is_touched(self) -> None:
|
||||
"""An unauthorized DELETE must not tear a bottle down. 401, and the
|
||||
bottle is still there."""
|
||||
rec = self.orch.registry.register("10.0.0.9", policy="", metadata="")
|
||||
status, _ = dispatch(
|
||||
self.orch, "DELETE", f"/bottles/{rec.bottle_id}", b"", authorized=False)
|
||||
self.assertEqual(401, status)
|
||||
self.assertIsNotNone(self.orch.registry.get(rec.bottle_id))
|
||||
|
||||
def _server_with_secret(self, secret: str):
|
||||
with patch.dict("os.environ", {"BOT_BOTTLE_CONTROL_PLANE_TOKEN": secret}):
|
||||
server = make_server(self.orch, "127.0.0.1", 0)
|
||||
self.addCleanup(server.server_close)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
self.addCleanup(server.shutdown)
|
||||
host, port = server.server_address[0], server.server_address[1]
|
||||
return f"http://{host}:{port}"
|
||||
|
||||
def _status(self, url: str, *, header: str | None = None) -> int:
|
||||
req = urllib.request.Request(url)
|
||||
if header is not None:
|
||||
req.add_header("x-bot-bottle-control-auth", header)
|
||||
try:
|
||||
return urllib.request.urlopen(req, timeout=5).status
|
||||
except urllib.error.HTTPError as e:
|
||||
return e.code
|
||||
|
||||
def test_configured_server_enforces_the_header_over_http(self) -> None:
|
||||
base = self._server_with_secret("s3cret-admin")
|
||||
# /health is public — no header needed.
|
||||
self.assertEqual(200, self._status(f"{base}/health"))
|
||||
# /bottles requires the secret.
|
||||
self.assertEqual(401, self._status(f"{base}/bottles"))
|
||||
self.assertEqual(401, self._status(f"{base}/bottles", header="wrong"))
|
||||
self.assertEqual(200, self._status(f"{base}/bottles", header="s3cret-admin"))
|
||||
|
||||
def test_unconfigured_server_runs_open(self) -> None:
|
||||
"""No secret set (tests / nft-protected Firecracker): open mode, so the
|
||||
existing round-trip and unit behavior are unchanged."""
|
||||
with patch.dict("os.environ", {}, clear=False):
|
||||
import os
|
||||
os.environ.pop("BOT_BOTTLE_CONTROL_PLANE_TOKEN", None)
|
||||
server = make_server(self.orch, "127.0.0.1", 0)
|
||||
self.addCleanup(server.server_close)
|
||||
self.assertTrue(server.is_authorized(""))
|
||||
self.assertTrue(server.is_authorized("anything"))
|
||||
|
||||
|
||||
class TestDispatchSupervise(unittest.TestCase):
|
||||
"""The /supervise/* routes over the pure dispatch()."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
root = Path(self._tmp.name)
|
||||
db = root / "db" / "bot-bottle.db"
|
||||
db.parent.mkdir(parents=True)
|
||||
self._env = patch.dict("os.environ", {
|
||||
"BOT_BOTTLE_ROOT": str(root),
|
||||
"SUPERVISE_DB_PATH": str(db),
|
||||
})
|
||||
self._env.start()
|
||||
self.store = RegistryStore(db)
|
||||
self.store.migrate()
|
||||
StoreManager(db).migrate()
|
||||
secret = secrets.token_bytes(16)
|
||||
self.orch = Orchestrator(self.store, StubBroker(secret), secret)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self._env.stop()
|
||||
self._tmp.cleanup()
|
||||
|
||||
def _queue(self, slug: str, proposed: str) -> str:
|
||||
self.store.register(
|
||||
"10.243.0.1", metadata=json.dumps({"slug": slug}), policy="routes: []\n")
|
||||
p = Proposal.new(
|
||||
bottle_slug=slug, tool=TOOL_EGRESS_ALLOW, proposed_file=proposed,
|
||||
justification="need it", current_file_hash=sha256_hex(proposed))
|
||||
write_proposal(p)
|
||||
return p.id
|
||||
|
||||
def test_list_pending(self) -> None:
|
||||
pid = self._queue("demo", "routes:\n - host: google.com\n")
|
||||
status, payload = dispatch(self.orch, "GET", "/supervise/proposals", b"")
|
||||
self.assertEqual(200, status)
|
||||
proposals = payload["proposals"]
|
||||
assert isinstance(proposals, list)
|
||||
self.assertEqual(pid, proposals[0]["id"])
|
||||
|
||||
def test_respond_approve_applies_and_clears(self) -> None:
|
||||
pid = self._queue("demo", "routes:\n - host: google.com\n")
|
||||
status, payload = dispatch(
|
||||
self.orch, "POST", "/supervise/respond",
|
||||
_body({"proposal_id": pid, "bottle_slug": "demo", "decision": "approve"}),
|
||||
)
|
||||
self.assertEqual(200, status)
|
||||
self.assertTrue(payload["responded"])
|
||||
_, listing = dispatch(self.orch, "GET", "/supervise/proposals", b"")
|
||||
self.assertEqual([], listing["proposals"])
|
||||
|
||||
def test_respond_requires_fields(self) -> None:
|
||||
status, _ = dispatch(
|
||||
self.orch, "POST", "/supervise/respond", _body({"decision": "approve"}))
|
||||
self.assertEqual(400, status)
|
||||
|
||||
def test_respond_unknown_proposal_conflicts(self) -> None:
|
||||
status, payload = dispatch(
|
||||
self.orch, "POST", "/supervise/respond",
|
||||
_body({"proposal_id": "ghost", "bottle_slug": "demo", "decision": "approve"}),
|
||||
)
|
||||
self.assertEqual(409, status)
|
||||
self.assertIn("no such proposal", str(payload["error"]))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -38,7 +38,7 @@ class TestDockerGateway(unittest.TestCase):
|
||||
def test_ensure_running_noop_when_up_and_image_current(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout=self.sc.name) # running
|
||||
@@ -58,7 +58,7 @@ class TestDockerGateway(unittest.TestCase):
|
||||
# a rebuild's new flat daemons take effect.
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout=self.sc.name)
|
||||
@@ -76,7 +76,7 @@ class TestDockerGateway(unittest.TestCase):
|
||||
def test_ensure_running_starts_the_singleton_when_absent(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
return _proc(stdout="") if argv[:2] == ["docker", "ps"] else _proc()
|
||||
|
||||
@@ -91,18 +91,11 @@ class TestDockerGateway(unittest.TestCase):
|
||||
self.assertEqual(self.sc.network, runs[0][runs[0].index("--network") + 1])
|
||||
# Persists its CA on a named volume so agents keep trusting it.
|
||||
self.assertTrue(any("mitmproxy" in a for a in runs[0]))
|
||||
# Shares the ONE host DB: the supervise daemon queues into the same
|
||||
# file the orchestrator + operator (over HTTP) use.
|
||||
self.assertTrue(any(
|
||||
a.startswith("SUPERVISE_DB_PATH=") and a.endswith("/run/supervise/bot-bottle.db")
|
||||
for a in runs[0]))
|
||||
self.assertTrue(any(
|
||||
a.endswith(":/run/supervise") for a in runs[0]))
|
||||
|
||||
def test_ensure_running_creates_network_when_missing(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:3] == ["docker", "network", "inspect"]:
|
||||
return _proc(returncode=1, stderr="No such network")
|
||||
@@ -135,7 +128,7 @@ class TestDockerGateway(unittest.TestCase):
|
||||
def test_ensure_running_reuses_existing_network(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
return _proc(stdout="") if argv[:2] == ["docker", "ps"] else _proc()
|
||||
|
||||
@@ -144,7 +137,7 @@ class TestDockerGateway(unittest.TestCase):
|
||||
self.assertEqual([], [c for c in calls if c[:3] == ["docker", "network", "create"]])
|
||||
|
||||
def test_ensure_running_raises_on_docker_failure(self) -> None:
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout="")
|
||||
if argv[:2] == ["docker", "run"]:
|
||||
@@ -181,7 +174,7 @@ class TestDockerGatewayBuild(unittest.TestCase):
|
||||
# build-if-missing silently ran a stale single-tenant image.
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def rec(argv: list[str], **_kw: object) -> Mock:
|
||||
def rec(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
return _proc() # image present, build succeeds
|
||||
|
||||
@@ -196,7 +189,7 @@ class TestDockerGatewayBuild(unittest.TestCase):
|
||||
def test_ensure_built_no_cache_env_forces_full_rebuild(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def rec(argv: list[str], **_kw: object) -> Mock:
|
||||
def rec(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
return _proc()
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@ from bot_bottle.orchestrator.lifecycle import (
|
||||
ORCHESTRATOR_SOURCE_HASH_LABEL,
|
||||
OrchestratorService,
|
||||
OrchestratorStartError,
|
||||
source_hash,
|
||||
_source_hash,
|
||||
)
|
||||
from tests.unit import use_bottle_root
|
||||
|
||||
@@ -57,10 +57,10 @@ class TestOrchestratorService(unittest.TestCase):
|
||||
# A healthy control plane already running the *current* bind-mounted
|
||||
# source is left alone — recreating it on every launch would drop
|
||||
# every other active bottle's in-memory egress tokens (#381).
|
||||
current = source_hash(self.svc._repo_root)
|
||||
current = _source_hash(self.svc._repo_root)
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout=ORCHESTRATOR_NAME)
|
||||
@@ -83,7 +83,7 @@ class TestOrchestratorService(unittest.TestCase):
|
||||
# effect, same as the gateway's image-staleness check.
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout=ORCHESTRATOR_NAME)
|
||||
@@ -98,13 +98,13 @@ class TestOrchestratorService(unittest.TestCase):
|
||||
self.assertEqual(1, len(runs))
|
||||
self.assertIn(ORCHESTRATOR_NAME, runs[0])
|
||||
# the fresh container is labeled with the current hash, not the stale one
|
||||
current = source_hash(self.svc._repo_root)
|
||||
current = _source_hash(self.svc._repo_root)
|
||||
self.assertIn(f"{ORCHESTRATOR_SOURCE_HASH_LABEL}={current}", runs[0])
|
||||
|
||||
def test_ensure_running_starts_orchestrator_container_when_absent(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout="") # not running
|
||||
@@ -127,7 +127,7 @@ class TestOrchestratorService(unittest.TestCase):
|
||||
# gateway data plane — built from Dockerfile.orchestrator when absent.
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout="") # orchestrator not running
|
||||
@@ -148,7 +148,7 @@ class TestOrchestratorService(unittest.TestCase):
|
||||
def test_ensure_running_skips_orchestrator_image_build_when_present(self) -> None:
|
||||
calls: list[list[str]] = []
|
||||
|
||||
def fake(argv: list[str], **_kw: object) -> Mock:
|
||||
def fake(argv: list[str]) -> Mock:
|
||||
calls.append(argv)
|
||||
if argv[:2] == ["docker", "ps"]:
|
||||
return _proc(stdout="")
|
||||
|
||||
@@ -2,26 +2,15 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import secrets
|
||||
import tempfile
|
||||
import unittest
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from bot_bottle.orchestrator.broker import LaunchBroker, LaunchRequest, StubBroker
|
||||
from bot_bottle.orchestrator.registry import RegistryStore
|
||||
from bot_bottle.orchestrator.service import Orchestrator
|
||||
from bot_bottle.orchestrator.gateway import Gateway
|
||||
from bot_bottle.store_manager import StoreManager
|
||||
from bot_bottle.supervise import (
|
||||
Proposal,
|
||||
STATUS_APPROVED,
|
||||
TOOL_EGRESS_ALLOW,
|
||||
read_response,
|
||||
sha256_hex,
|
||||
write_proposal,
|
||||
)
|
||||
|
||||
|
||||
class _FailingBroker(LaunchBroker):
|
||||
@@ -125,12 +114,9 @@ class TestOrchestrator(unittest.TestCase):
|
||||
def test_set_policy_unknown_is_false(self) -> None:
|
||||
self.assertFalse(self.orch.set_policy("ghost", "{}"))
|
||||
|
||||
def test_resolve_requires_matching_token(self) -> None:
|
||||
# Mandatory (source_ip, token) pair — no source-IP-only fallback.
|
||||
def test_resolve_by_source_ip_without_token(self) -> None:
|
||||
rec = self.orch.launch_bottle("10.243.0.1", policy="P")
|
||||
self.assertIsNone(self.orch.resolve("10.243.0.1", "")) # empty token denies
|
||||
self.assertIsNone(self.orch.resolve("10.243.0.1", "wrong")) # mismatch denies
|
||||
got = self.orch.resolve("10.243.0.1", rec.identity_token) # exact pair
|
||||
got = self.orch.resolve("10.243.0.1") # network-layer, no token
|
||||
assert got is not None
|
||||
self.assertEqual(rec.bottle_id, got.bottle_id)
|
||||
self.assertEqual("P", got.policy)
|
||||
@@ -166,141 +152,5 @@ class TestOrchestrator(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestOrchestratorSupervise(unittest.TestCase):
|
||||
"""Operator-approval flow: the orchestrator applies the decision
|
||||
server-side against the single DB (queue + policy + audit)."""
|
||||
|
||||
def setUp(self) -> None:
|
||||
self._tmp = tempfile.TemporaryDirectory()
|
||||
root = Path(self._tmp.name)
|
||||
db = root / "db" / "bot-bottle.db"
|
||||
db.parent.mkdir(parents=True)
|
||||
# One DB for registry + supervise queue + audit (as in the VM).
|
||||
self._env = patch.dict("os.environ", {
|
||||
"BOT_BOTTLE_ROOT": str(root),
|
||||
"SUPERVISE_DB_PATH": str(db),
|
||||
})
|
||||
self._env.start()
|
||||
self.store = RegistryStore(db)
|
||||
self.store.migrate()
|
||||
StoreManager(db).migrate()
|
||||
secret = secrets.token_bytes(16)
|
||||
self.orch = Orchestrator(self.store, StubBroker(secret), secret)
|
||||
|
||||
def tearDown(self) -> None:
|
||||
self._env.stop()
|
||||
self._tmp.cleanup()
|
||||
|
||||
def _register(self, slug: str, policy: str) -> str:
|
||||
rec = self.store.register(
|
||||
"10.243.0.1", metadata=json.dumps({"slug": slug}), policy=policy)
|
||||
return rec.bottle_id
|
||||
|
||||
def _queue(self, slug: str, proposed: str) -> str:
|
||||
p = Proposal.new(
|
||||
bottle_slug=slug, tool=TOOL_EGRESS_ALLOW, proposed_file=proposed,
|
||||
justification="need it", current_file_hash=sha256_hex(proposed))
|
||||
write_proposal(p)
|
||||
return p.id
|
||||
|
||||
def test_pending_lists_queued_proposal(self) -> None:
|
||||
self._register("demo", "routes: []\n")
|
||||
pid = self._queue("demo", "routes:\n - host: google.com\n")
|
||||
pending = self.orch.supervise_pending()
|
||||
self.assertEqual(1, len(pending))
|
||||
self.assertEqual(pid, pending[0]["id"])
|
||||
self.assertEqual("demo", pending[0]["bottle_slug"])
|
||||
|
||||
def test_approve_applies_policy_writes_response_and_clears_pending(self) -> None:
|
||||
bottle_id = self._register("demo", "routes:\n - host: existing.com\n")
|
||||
new_routes = "routes:\n - host: google.com\n"
|
||||
pid = self._queue("demo", new_routes)
|
||||
ok, err = self.orch.supervise_respond(
|
||||
pid, bottle_slug="demo", decision="approve")
|
||||
self.assertTrue(ok, err)
|
||||
# policy live-applied so /resolve serves the new routes
|
||||
rec = self.store.get(bottle_id)
|
||||
assert rec is not None
|
||||
self.assertEqual(new_routes, rec.policy)
|
||||
# response written -> agent unblocks, proposal no longer pending
|
||||
self.assertEqual(STATUS_APPROVED, read_response("demo", pid).status)
|
||||
self.assertEqual([], self.orch.supervise_pending())
|
||||
|
||||
def test_pending_carries_human_label(self) -> None:
|
||||
# The proposal is keyed by bottle_id, but pending dicts also expose the
|
||||
# bottle's human slug so the operator sees a name, not a hex id.
|
||||
bottle_id = self._register("codex-dev-a1b2c", "routes: []\n")
|
||||
self._queue(bottle_id, "routes:\n - host: google.com\n")
|
||||
pending = self.orch.supervise_pending()
|
||||
self.assertEqual(bottle_id, pending[0]["bottle_slug"])
|
||||
self.assertEqual("codex-dev-a1b2c", pending[0]["bottle_label"])
|
||||
|
||||
def test_pending_label_falls_back_to_slug_when_bottle_gone(self) -> None:
|
||||
# No registry record (torn down): label is the id, never empty.
|
||||
self._queue("ghost-id", "routes:\n - host: google.com\n")
|
||||
pending = self.orch.supervise_pending()
|
||||
self.assertEqual("ghost-id", pending[0]["bottle_label"])
|
||||
|
||||
def test_approve_by_bottle_id_applies_policy(self) -> None:
|
||||
# Consolidated reality: the supervise server keys each proposal by the
|
||||
# orchestrator-assigned bottle_id, not the human slug. Approval must
|
||||
# resolve the record by that id and apply the policy (regression for
|
||||
# the "bottle <id> is no longer registered" 409).
|
||||
bottle_id = self._register("codex-dev-a1b2c", "routes: []\n")
|
||||
new_routes = "routes:\n - host: google.com\n"
|
||||
pid = self._queue(bottle_id, new_routes)
|
||||
ok, err = self.orch.supervise_respond(
|
||||
pid, bottle_slug=bottle_id, decision="approve")
|
||||
self.assertTrue(ok, err)
|
||||
rec = self.store.get(bottle_id)
|
||||
assert rec is not None
|
||||
self.assertEqual(new_routes, rec.policy)
|
||||
|
||||
def test_modify_applies_final_file_not_proposed(self) -> None:
|
||||
bottle_id = self._register("demo", "routes: []\n")
|
||||
pid = self._queue("demo", "routes:\n - host: google.com\n")
|
||||
edited = "routes:\n - host: example.com\n"
|
||||
ok, _ = self.orch.supervise_respond(
|
||||
pid, bottle_slug="demo", decision="modify", final_file=edited)
|
||||
self.assertTrue(ok)
|
||||
rec = self.store.get(bottle_id)
|
||||
assert rec is not None
|
||||
self.assertEqual(edited, rec.policy)
|
||||
|
||||
def test_reject_leaves_policy_unchanged(self) -> None:
|
||||
bottle_id = self._register("demo", "routes:\n - host: existing.com\n")
|
||||
pid = self._queue("demo", "routes:\n - host: google.com\n")
|
||||
ok, _ = self.orch.supervise_respond(
|
||||
pid, bottle_slug="demo", decision="reject", notes="no")
|
||||
self.assertTrue(ok)
|
||||
rec = self.store.get(bottle_id)
|
||||
assert rec is not None
|
||||
self.assertEqual("routes:\n - host: existing.com\n", rec.policy)
|
||||
self.assertEqual("rejected", read_response("demo", pid).status)
|
||||
|
||||
def test_unknown_proposal_is_error(self) -> None:
|
||||
ok, err = self.orch.supervise_respond(
|
||||
"ghost", bottle_slug="demo", decision="approve")
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("no such proposal", err)
|
||||
|
||||
def test_unknown_decision_is_error(self) -> None:
|
||||
self._register("demo", "routes: []\n")
|
||||
pid = self._queue("demo", "routes:\n - host: google.com\n")
|
||||
ok, err = self.orch.supervise_respond(
|
||||
pid, bottle_slug="demo", decision="bogus")
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("unknown decision", err)
|
||||
|
||||
def test_approve_when_bottle_gone_cannot_apply(self) -> None:
|
||||
# Proposal queued but the bottle was torn down before the operator
|
||||
# acted: an egress apply has no target, so respond fails closed.
|
||||
pid = self._queue("ghost-bottle", "routes:\n - host: google.com\n")
|
||||
ok, err = self.orch.supervise_respond(
|
||||
pid, bottle_slug="ghost-bottle", decision="approve")
|
||||
self.assertFalse(ok)
|
||||
self.assertIn("no longer registered", err)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1,62 +0,0 @@
|
||||
"""Unit: the infra-artifact publisher's upload path (PRD 0069 Stage 2).
|
||||
|
||||
The rootfs is hundreds of MB, so `_put` must stream it from disk rather than
|
||||
read it into memory. Network is mocked; no Docker, no real build.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import tempfile
|
||||
import unittest
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from unittest import mock
|
||||
|
||||
from bot_bottle.backend.firecracker import publish_infra as pub
|
||||
|
||||
|
||||
class _Resp:
|
||||
status = 201
|
||||
|
||||
def __enter__(self) -> "_Resp":
|
||||
return self
|
||||
|
||||
def __exit__(self, *a: object) -> bool:
|
||||
return False
|
||||
|
||||
|
||||
class TestPut(unittest.TestCase):
|
||||
def test_streams_file_body_with_content_length(self) -> None:
|
||||
captured: list[urllib.request.Request] = []
|
||||
|
||||
def fake_urlopen(req: urllib.request.Request, *a: object, **k: object) -> _Resp:
|
||||
captured.append(req)
|
||||
return _Resp()
|
||||
|
||||
with tempfile.TemporaryDirectory() as d:
|
||||
f = Path(d) / "rootfs.ext4.gz"
|
||||
payload = b"x" * 4096
|
||||
f.write_bytes(payload)
|
||||
with mock.patch.object(pub.urllib.request, "urlopen", fake_urlopen):
|
||||
pub._put("https://reg/pkg", f, token="t")
|
||||
|
||||
req = captured[0]
|
||||
# Body is the open file object (streamed), never the bytes in memory.
|
||||
self.assertTrue(hasattr(req.data, "read"))
|
||||
self.assertNotIsInstance(req.data, (bytes, bytearray))
|
||||
self.assertEqual(str(len(payload)), req.get_header("Content-length"))
|
||||
|
||||
def test_small_bytes_body_still_works(self) -> None:
|
||||
captured: list[urllib.request.Request] = []
|
||||
|
||||
def fake_urlopen(req: urllib.request.Request, *a: object, **k: object) -> _Resp:
|
||||
captured.append(req)
|
||||
return _Resp()
|
||||
|
||||
with mock.patch.object(pub.urllib.request, "urlopen", fake_urlopen):
|
||||
pub._put("https://reg/sha", b"abc123 rootfs\n", token="")
|
||||
self.assertEqual(b"abc123 rootfs\n", captured[0].data)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
+170
-122
@@ -1,24 +1,31 @@
|
||||
"""Unit: supervise headless paths — the discovery + approve/reject that the
|
||||
TUI key handlers call into.
|
||||
"""Unit: supervise headless paths (PRD 0013 phase 4, PRD 0016).
|
||||
|
||||
These go through the orchestrator HTTP client now (the operator never
|
||||
touches the DB directly), so the client is mocked here; the server-side
|
||||
apply / response / audit is covered in test_orchestrator_service.
|
||||
The curses TUI itself isn't exercised here — these tests cover the
|
||||
discovery + approve/reject paths that the TUI's key handlers call into.
|
||||
"""
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
import unittest
|
||||
from datetime import datetime, timezone
|
||||
from unittest.mock import MagicMock, patch
|
||||
from pathlib import Path
|
||||
from unittest.mock import patch
|
||||
|
||||
from bot_bottle import supervise
|
||||
from tests.unit import use_bottle_root
|
||||
from bot_bottle.audit_store import AuditStore
|
||||
from bot_bottle.cli import supervise as supervise_cli
|
||||
from bot_bottle.queue_store import QueueStore
|
||||
from bot_bottle.supervise import (
|
||||
Proposal,
|
||||
STATUS_APPROVED,
|
||||
STATUS_MODIFIED,
|
||||
STATUS_REJECTED,
|
||||
TOOL_EGRESS_ALLOW,
|
||||
TOOL_EGRESS_BLOCK,
|
||||
TOOL_GITLEAKS_ALLOW,
|
||||
TOOL_EGRESS_TOKEN_ALLOW,
|
||||
read_audit_entries,
|
||||
read_response,
|
||||
sha256_hex,
|
||||
)
|
||||
|
||||
@@ -26,131 +33,198 @@ from bot_bottle.supervise import (
|
||||
FIXED = datetime(2026, 5, 25, 12, 0, 0, tzinfo=timezone.utc)
|
||||
|
||||
|
||||
def _proposal(slug: str = "dev", tool: str = TOOL_EGRESS_ALLOW,
|
||||
*, now: datetime = FIXED) -> Proposal:
|
||||
def _proposal(slug: str = "dev", tool: str = TOOL_EGRESS_ALLOW) -> Proposal:
|
||||
payloads = {
|
||||
TOOL_EGRESS_ALLOW: "routes:\n - host: example.com\n",
|
||||
TOOL_EGRESS_BLOCK: "routes:\n - host: example.com\n",
|
||||
supervise.TOOL_EGRESS_ALLOW: "routes:\n - host: example.com\n",
|
||||
supervise.TOOL_EGRESS_BLOCK: "routes:\n - host: example.com\n",
|
||||
TOOL_GITLEAKS_ALLOW: "file: tests/test_fixture.py\nline: 3\n",
|
||||
TOOL_EGRESS_TOKEN_ALLOW: "host: api.example.com\ndetector: token\n",
|
||||
}
|
||||
payload = payloads.get(tool, "")
|
||||
return Proposal.new(
|
||||
bottle_slug=slug, tool=tool, proposed_file=payload,
|
||||
justification=f"needed for {slug}", current_file_hash=sha256_hex(payload),
|
||||
now=now,
|
||||
bottle_slug=slug, tool=tool,
|
||||
proposed_file=payload,
|
||||
justification=f"needed for {slug}",
|
||||
current_file_hash=sha256_hex(payload),
|
||||
now=FIXED,
|
||||
)
|
||||
|
||||
|
||||
class _ClientMixin:
|
||||
"""Install a mock orchestrator client as the CLI-session singleton."""
|
||||
class _FakeHomeMixin:
|
||||
"""Point bot_bottle_root at a temp dir (via BOT_BOTTLE_ROOT) for the test."""
|
||||
|
||||
def _install_client(self, pending: "list[Proposal] | None" = None) -> MagicMock:
|
||||
client = MagicMock()
|
||||
client.supervise_pending.return_value = [
|
||||
p.to_dict() for p in (pending or [])
|
||||
]
|
||||
patcher = patch.object(supervise_cli, "_client", return_value=client)
|
||||
patcher.start()
|
||||
self.addCleanup(patcher.stop) # type: ignore[attr-defined]
|
||||
self.addCleanup( # type: ignore[attr-defined]
|
||||
lambda: setattr(supervise_cli, "_client_instance", None))
|
||||
return client
|
||||
def _setup_fake_home(self):
|
||||
self._tmp = tempfile.TemporaryDirectory(prefix="supervise-test.")
|
||||
self._restore_home = use_bottle_root(Path(self._tmp.name) / ".bot-bottle")
|
||||
QueueStore("").migrate()
|
||||
AuditStore().migrate()
|
||||
|
||||
def _teardown_fake_home(self):
|
||||
self._restore_home()
|
||||
self._tmp.cleanup()
|
||||
|
||||
|
||||
class TestDiscoverPending(_ClientMixin, unittest.TestCase):
|
||||
def test_empty(self) -> None:
|
||||
self._install_client([])
|
||||
class TestDiscoverPending(_FakeHomeMixin, unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._setup_fake_home()
|
||||
|
||||
def tearDown(self):
|
||||
self._teardown_fake_home()
|
||||
|
||||
def test_empty_when_no_queues(self):
|
||||
self.assertEqual([], supervise_cli.discover_pending())
|
||||
|
||||
def test_lists_all_bottles(self) -> None:
|
||||
self._install_client([_proposal("dev"), _proposal("api")])
|
||||
def test_walks_all_slug_subdirs(self):
|
||||
for slug in ("dev", "api"):
|
||||
supervise.write_proposal(_proposal(slug=slug))
|
||||
pending = supervise_cli.discover_pending()
|
||||
self.assertEqual(
|
||||
{"dev", "api"}, {qp.proposal.bottle_slug for qp in pending})
|
||||
self.assertEqual({"dev", "api"}, {qp.proposal.bottle_slug for qp in pending})
|
||||
|
||||
def test_sorted_by_arrival(self) -> None:
|
||||
early = _proposal(
|
||||
"api", now=datetime(2026, 5, 25, 10, 0, 0, tzinfo=timezone.utc))
|
||||
late = _proposal(
|
||||
"dev", now=datetime(2026, 5, 25, 14, 0, 0, tzinfo=timezone.utc))
|
||||
self._install_client([late, early])
|
||||
def test_sorted_by_arrival_across_bottles(self):
|
||||
early = Proposal.new(
|
||||
bottle_slug="api", tool=TOOL_EGRESS_ALLOW,
|
||||
proposed_file="routes:\n - host: early.example.com\n", justification="early",
|
||||
current_file_hash="h",
|
||||
now=datetime(2026, 5, 25, 10, 0, 0, tzinfo=timezone.utc),
|
||||
)
|
||||
late = Proposal.new(
|
||||
bottle_slug="dev", tool=TOOL_EGRESS_ALLOW,
|
||||
proposed_file="routes:\n - host: late.example.com\n", justification="late",
|
||||
current_file_hash="h",
|
||||
now=datetime(2026, 5, 25, 14, 0, 0, tzinfo=timezone.utc),
|
||||
)
|
||||
for p in (late, early):
|
||||
supervise.write_proposal(p)
|
||||
pending = supervise_cli.discover_pending()
|
||||
self.assertEqual([early.id, late.id], [qp.proposal.id for qp in pending])
|
||||
|
||||
def test_label_comes_from_bottle_label(self) -> None:
|
||||
# The server tags each dict with the human slug; the CLI displays it
|
||||
# while the proposal stays keyed by the opaque bottle_id.
|
||||
client = MagicMock()
|
||||
d = _proposal("3601cbe883c2786d").to_dict()
|
||||
d["bottle_label"] = "codex-dev-a1b2c"
|
||||
client.supervise_pending.return_value = [d]
|
||||
with patch.object(supervise_cli, "_client", return_value=client):
|
||||
pending = supervise_cli.discover_pending()
|
||||
self.assertEqual("codex-dev-a1b2c", pending[0].label)
|
||||
self.assertEqual("3601cbe883c2786d", pending[0].proposal.bottle_slug)
|
||||
|
||||
def test_label_falls_back_to_slug_when_absent(self) -> None:
|
||||
# Legacy dicts without bottle_label (e.g. an older orchestrator).
|
||||
self._install_client([_proposal("dev")])
|
||||
self.assertEqual("dev", supervise_cli.discover_pending()[0].label)
|
||||
def test_excludes_already_responded(self):
|
||||
p = _proposal()
|
||||
supervise.write_proposal(p)
|
||||
supervise.write_response("dev", supervise.Response(
|
||||
proposal_id=p.id, status=STATUS_APPROVED, notes="",
|
||||
))
|
||||
self.assertEqual([], supervise_cli.discover_pending())
|
||||
|
||||
|
||||
class TestApproveReject(_ClientMixin, unittest.TestCase):
|
||||
def _qp(self, tool: str = TOOL_EGRESS_ALLOW) -> "supervise_cli.QueuedProposal":
|
||||
return supervise_cli.QueuedProposal(proposal=_proposal(tool=tool))
|
||||
class TestApproveReject(_FakeHomeMixin, unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._setup_fake_home()
|
||||
|
||||
def test_approve_calls_respond(self) -> None:
|
||||
client = self._install_client()
|
||||
qp = self._qp()
|
||||
supervise_cli.approve(qp)
|
||||
client.supervise_respond.assert_called_once_with(
|
||||
qp.proposal.id, bottle_slug="dev", decision="approve",
|
||||
notes="", final_file=None,
|
||||
)
|
||||
def tearDown(self):
|
||||
self._teardown_fake_home()
|
||||
|
||||
def test_modify_sets_decision_and_final_file(self) -> None:
|
||||
client = self._install_client()
|
||||
qp = self._qp()
|
||||
edited = "routes:\n - host: edited.example.com\n"
|
||||
supervise_cli.approve(qp, final_file=edited, notes="tweaked")
|
||||
client.supervise_respond.assert_called_once_with(
|
||||
qp.proposal.id, bottle_slug="dev", decision="modify",
|
||||
notes="tweaked", final_file=edited,
|
||||
)
|
||||
def _enqueue(self, tool: str = TOOL_EGRESS_ALLOW):
|
||||
p = _proposal(tool=tool)
|
||||
supervise.write_proposal(p)
|
||||
return supervise_cli.QueuedProposal(proposal=p)
|
||||
|
||||
def test_reject_calls_respond(self) -> None:
|
||||
client = self._install_client()
|
||||
qp = self._qp()
|
||||
def test_approve_writes_response(self):
|
||||
qp = self._enqueue()
|
||||
with patch(
|
||||
"bot_bottle.cli.supervise.apply_routes_change",
|
||||
return_value=("routes: []\n", "routes:\n - host: example.com\n"),
|
||||
):
|
||||
supervise_cli.approve(qp)
|
||||
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
|
||||
self.assertEqual(STATUS_APPROVED, resp.status)
|
||||
self.assertIsNone(resp.final_file)
|
||||
|
||||
def test_approve_with_final_file_marks_modified(self):
|
||||
qp = self._enqueue()
|
||||
with patch(
|
||||
"bot_bottle.cli.supervise.apply_routes_change",
|
||||
return_value=("routes: []\n", "routes:\n - host: edited.example.com\n"),
|
||||
):
|
||||
supervise_cli.approve(
|
||||
qp,
|
||||
final_file="routes:\n - host: edited.example.com\n",
|
||||
notes="tweaked",
|
||||
)
|
||||
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
|
||||
self.assertEqual(STATUS_MODIFIED, resp.status)
|
||||
self.assertEqual("routes:\n - host: edited.example.com\n", resp.final_file)
|
||||
self.assertEqual("tweaked", resp.notes)
|
||||
|
||||
def test_reject_writes_rejection(self):
|
||||
qp = self._enqueue()
|
||||
supervise_cli.reject(qp, reason="nope")
|
||||
client.supervise_respond.assert_called_once_with(
|
||||
qp.proposal.id, bottle_slug="dev", decision="reject", notes="nope",
|
||||
)
|
||||
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
|
||||
self.assertEqual(STATUS_REJECTED, resp.status)
|
||||
self.assertEqual("nope", resp.notes)
|
||||
|
||||
def test_tui_report_only_requires_reason(self) -> None:
|
||||
self._install_client()
|
||||
qp = self._qp(tool=TOOL_GITLEAKS_ALLOW)
|
||||
def test_approve_egress_block_writes_audit_log(self):
|
||||
qp = self._enqueue(tool=supervise.TOOL_EGRESS_BLOCK)
|
||||
with patch(
|
||||
"bot_bottle.cli.supervise.apply_routes_change",
|
||||
return_value=("routes: []\n", "routes:\n - host: example.com\n"),
|
||||
) as apply_routes_change:
|
||||
supervise_cli.approve(qp)
|
||||
apply_routes_change.assert_called_once_with(
|
||||
"dev",
|
||||
"routes:\n - host: example.com\n",
|
||||
)
|
||||
entries = read_audit_entries("egress", "dev")
|
||||
self.assertEqual(1, len(entries))
|
||||
self.assertEqual(STATUS_APPROVED, entries[0].operator_action)
|
||||
self.assertEqual("needed for dev", entries[0].justification)
|
||||
|
||||
def test_approve_gitleaks_allow_leaves_response_for_gate(self):
|
||||
qp = self._enqueue(tool=TOOL_GITLEAKS_ALLOW)
|
||||
supervise_cli.approve(qp, notes="dummy fixture")
|
||||
# Gate polls the DB for the response; TUI must not archive it.
|
||||
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
|
||||
self.assertEqual(STATUS_APPROVED, resp.status)
|
||||
self.assertEqual("dummy fixture", resp.notes)
|
||||
|
||||
def test_tui_gitleaks_allow_requires_reason(self):
|
||||
qp = self._enqueue(tool=TOOL_GITLEAKS_ALLOW)
|
||||
with patch.object(supervise_cli, "_prompt", return_value=""):
|
||||
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
|
||||
self.assertEqual("approve aborted (empty reason)", status)
|
||||
|
||||
def test_tui_report_only_writes_reason(self) -> None:
|
||||
client = self._install_client()
|
||||
qp = self._qp(tool=TOOL_GITLEAKS_ALLOW)
|
||||
def test_tui_gitleaks_allow_writes_reason(self):
|
||||
qp = self._enqueue(tool=TOOL_GITLEAKS_ALLOW)
|
||||
with patch.object(supervise_cli, "_prompt", return_value="test fixture"):
|
||||
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
|
||||
self.assertIn("approved gitleaks-allow", status)
|
||||
client.supervise_respond.assert_called_once()
|
||||
self.assertEqual(
|
||||
"test fixture", client.supervise_respond.call_args.kwargs["notes"])
|
||||
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
|
||||
self.assertEqual("test fixture", resp.notes)
|
||||
|
||||
def test_suffix_for_token_allow_is_txt(self) -> None:
|
||||
self.assertEqual(
|
||||
".txt", supervise_cli._suffix_for_tool(TOOL_EGRESS_TOKEN_ALLOW))
|
||||
def test_approve_token_allow_leaves_response_for_egress(self):
|
||||
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
|
||||
supervise_cli.approve(qp, notes="false positive")
|
||||
# The egress addon polls the DB for the response; the TUI must
|
||||
# not archive it (the addon archives after reading).
|
||||
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
|
||||
self.assertEqual(STATUS_APPROVED, resp.status)
|
||||
self.assertEqual("false positive", resp.notes)
|
||||
|
||||
def test_token_allow_writes_no_audit_log(self):
|
||||
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
|
||||
supervise_cli.approve(qp, notes="false positive")
|
||||
self.assertEqual([], read_audit_entries("egress", "dev"))
|
||||
|
||||
def test_tui_token_allow_requires_reason(self):
|
||||
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
|
||||
with patch.object(supervise_cli, "_prompt", return_value=""):
|
||||
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
|
||||
self.assertEqual("approve aborted (empty reason)", status)
|
||||
|
||||
def test_tui_token_allow_writes_reason(self):
|
||||
qp = self._enqueue(tool=TOOL_EGRESS_TOKEN_ALLOW)
|
||||
with patch.object(supervise_cli, "_prompt", return_value="legit"):
|
||||
status = supervise_cli._approve_from_tui(None, qp) # type: ignore[arg-type]
|
||||
self.assertIn("approved egress-token-allow", status)
|
||||
resp = read_response(qp.proposal.bottle_slug, qp.proposal.id)
|
||||
self.assertEqual("legit", resp.notes)
|
||||
|
||||
def test_suffix_for_token_allow_is_txt(self):
|
||||
self.assertEqual(".txt", supervise_cli._suffix_for_tool(TOOL_EGRESS_TOKEN_ALLOW))
|
||||
|
||||
|
||||
class TestEditInEditor(unittest.TestCase):
|
||||
def test_runs_editor_returns_edited_content(self) -> None:
|
||||
def test_runs_editor_returns_edited_content(self):
|
||||
original_editor = os.environ.get("EDITOR")
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
@@ -171,7 +245,7 @@ class TestEditInEditor(unittest.TestCase):
|
||||
else:
|
||||
os.environ["EDITOR"] = original_editor
|
||||
|
||||
def test_returns_none_when_unchanged(self) -> None:
|
||||
def test_returns_none_when_unchanged(self):
|
||||
original_editor = os.environ.get("EDITOR")
|
||||
try:
|
||||
with tempfile.NamedTemporaryFile(
|
||||
@@ -193,31 +267,5 @@ class TestEditInEditor(unittest.TestCase):
|
||||
os.environ["EDITOR"] = original_editor
|
||||
|
||||
|
||||
class TestResolveOrchestratorUrl(unittest.TestCase):
|
||||
"""`_resolve_orchestrator_url` starts the backend orchestrator on demand
|
||||
when discovery finds nothing — supervise is often the first thing run."""
|
||||
|
||||
def test_returns_discovered_url_without_starting(self) -> None:
|
||||
with patch.object(
|
||||
supervise_cli, "discover_orchestrator_url",
|
||||
return_value="http://127.0.0.1:8099",
|
||||
), patch("bot_bottle.backend.get_bottle_backend") as get_backend:
|
||||
url = supervise_cli._resolve_orchestrator_url()
|
||||
self.assertEqual(url, "http://127.0.0.1:8099")
|
||||
get_backend.assert_not_called() # nothing to start; discovery won
|
||||
|
||||
def test_starts_backend_orchestrator_when_none_running(self) -> None:
|
||||
backend = MagicMock()
|
||||
backend.name = "firecracker"
|
||||
backend.ensure_orchestrator.return_value = "http://10.243.255.1:8099"
|
||||
with patch.object(
|
||||
supervise_cli, "discover_orchestrator_url",
|
||||
side_effect=supervise_cli.OrchestratorClientError("none"),
|
||||
), patch("bot_bottle.backend.get_bottle_backend", return_value=backend):
|
||||
url = supervise_cli._resolve_orchestrator_url()
|
||||
self.assertEqual(url, "http://10.243.255.1:8099")
|
||||
backend.ensure_orchestrator.assert_called_once_with()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -56,15 +56,6 @@ class _FakeHomeMixin:
|
||||
class TestCmdSuperviseErrorPaths(_FakeHomeMixin, unittest.TestCase):
|
||||
def setUp(self):
|
||||
self._setup_fake_home()
|
||||
# `cmd_supervise` establishes the orchestrator client up front; these
|
||||
# tests exercise the curses / crash-logging paths that run *after*
|
||||
# that, so stub the client. Otherwise the outcome depends on whether a
|
||||
# live orchestrator happens to be reachable (CI has none, so the
|
||||
# up-front connect would error and short-circuit before curses).
|
||||
client_patch = mock.patch.object(
|
||||
supervise_cli, "_client", return_value=mock.MagicMock())
|
||||
client_patch.start()
|
||||
self.addCleanup(client_patch.stop)
|
||||
|
||||
def tearDown(self):
|
||||
self._teardown_fake_home()
|
||||
|
||||
@@ -631,15 +631,9 @@ class TestHttpEndToEnd(unittest.TestCase):
|
||||
|
||||
|
||||
class _FakeResolver:
|
||||
def __init__(
|
||||
self,
|
||||
bottle_id: str | None = None,
|
||||
raises: bool = False,
|
||||
policy: str = "",
|
||||
) -> None:
|
||||
def __init__(self, bottle_id: str | None = None, raises: bool = False) -> None:
|
||||
self._bottle_id = bottle_id
|
||||
self._raises = raises
|
||||
self._policy = policy
|
||||
self.calls: list[str] = []
|
||||
|
||||
def resolve_bottle_id(self, source_ip: str, identity_token: str = "") -> str | None:
|
||||
@@ -651,15 +645,6 @@ class _FakeResolver:
|
||||
raise supervise_server.PolicyResolveError("orchestrator down")
|
||||
return self._bottle_id
|
||||
|
||||
def resolve_policy_and_bottle_id(
|
||||
self, source_ip: str, identity_token: str = "",
|
||||
) -> "tuple[str, str | None, dict[str, str]]":
|
||||
del identity_token
|
||||
self.calls.append(source_ip)
|
||||
if self._raises:
|
||||
raise supervise_server.PolicyResolveError("orchestrator down")
|
||||
return self._policy, self._bottle_id, {}
|
||||
|
||||
|
||||
def _handler(resolver: object) -> MCPHandler:
|
||||
"""A bare MCPHandler wired with a server (carrying the resolver) and a
|
||||
@@ -697,41 +682,5 @@ class TestAttributedConfig(unittest.TestCase):
|
||||
)
|
||||
|
||||
|
||||
class TestResolvedRoutesPayload(unittest.TestCase):
|
||||
"""`list-egress-routes` answers from the calling bottle's resolved policy in
|
||||
consolidated mode — not the gateway's empty static table. Regression: an
|
||||
empty list led agents to propose replace-all route files that dropped base
|
||||
hosts like api.anthropic.com on approval."""
|
||||
|
||||
def test_returns_resolved_bottle_routes(self) -> None:
|
||||
policy = (
|
||||
"routes:\n"
|
||||
" - host: api.anthropic.com\n"
|
||||
" - host: www.google.com\n"
|
||||
)
|
||||
payload = _handler(
|
||||
_FakeResolver(bottle_id="b1", policy=policy)
|
||||
)._resolved_routes_payload()
|
||||
assert payload is not None
|
||||
self.assertFalse(payload["isError"]) # type: ignore[index]
|
||||
data = json.loads(payload["content"][0]["text"]) # type: ignore[index]
|
||||
hosts = {r["host"] for r in data["routes"]}
|
||||
self.assertEqual({"api.anthropic.com", "www.google.com"}, hosts)
|
||||
|
||||
def test_orchestrator_error_fails_closed_to_empty(self) -> None:
|
||||
# resolve_client_context swallows resolver errors → deny-all (empty),
|
||||
# never another bottle's routes.
|
||||
payload = _handler(
|
||||
_FakeResolver(raises=True)
|
||||
)._resolved_routes_payload()
|
||||
assert payload is not None
|
||||
data = json.loads(payload["content"][0]["text"]) # type: ignore[index]
|
||||
self.assertEqual([], data["routes"])
|
||||
|
||||
def test_single_tenant_returns_none(self) -> None:
|
||||
# No resolver → caller falls back to the static introspection endpoint.
|
||||
self.assertIsNone(_handler(None)._resolved_routes_payload())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
Reference in New Issue
Block a user