Compare commits

...

7 Commits

Author SHA1 Message Date
didericis-claude 78d72264c8 fix(backend): fix pyright errors in lazy-load implementation
test / integration (pull_request) Successful in 8s
test / unit (pull_request) Successful in 29s
test / coverage (pull_request) Successful in 32s
lint / lint (push) Failing after 7s
- Rename _BACKENDS → _backends: pyright treats uppercase module-level
  names as constants and flags the reassignment in _get_backends() as
  reportConstantRedefinition; lowercase avoids this.
- Add TYPE_CHECKING guard importing CommitCancelled/Freezer/get_freezer
  from .freeze: pyright cannot see module-level __getattr__ bindings, so
  reportUnsupportedDunderAll fired for those three __all__ entries; the
  guard makes them visible to the type checker without running at import
  time.
- Update test_backend_selection.py to patch _backends (lowercase).
2026-07-18 05:02:57 -04:00
didericis-claude 0f62d70b81 fix(backend): silence pylint false positives from lazy-load pattern
`undefined-all-variable` fires on CommitCancelled / Freezer / get_freezer
in __all__ because pylint can't see module-level __getattr__ bindings;
`global-statement` fires on the _BACKENDS singleton setter. Both are
intentional patterns — add inline disables rather than suppress globally.
2026-07-18 05:02:57 -04:00
didericis-claude c8ef5a5638 perf: lazy-load backend modules and consolidate docker subprocess helpers
Importing backend.docker.util previously triggered eager loading of all
three backend packages (~76 modules) because backend/__init__.py imported
DockerBottleBackend, FirecrackerBottleBackend, and MacosContainerBottleBackend
at module scope. This made the module prohibitively expensive to import
from the orchestrator layer and elsewhere.

The three backend imports are now deferred into _get_backends(), which
loads all three on first call and caches the result in the module-level
_BACKENDS variable (initially None). Module-level __getattr__ exposes
backend classes and freeze symbols lazily for existing import/patch sites.

backend/docker/util.py raw subprocess.run(["docker", ...]) calls are
replaced with the shared run_docker primitive from docker_cmd, eliminating
the duplication between the backend and orchestrator implementations.
_silent_run() is removed; image_exists() is inlined directly onto
run_docker. The commit_container test is updated to patch run_docker
instead of subprocess.run.
2026-07-18 05:02:57 -04:00
didericis d3c4fc0fd4 ci(test): drop actions/setup-python; install into the container's system Python
test / integration (pull_request) Successful in 7s
test / unit (pull_request) Successful in 44s
test / coverage (pull_request) Successful in 36s
test / integration (push) Successful in 7s
Update Quality Badges / update-badges (push) Failing after 11s
test / unit (push) Successful in 29s
test / coverage (push) Successful in 35s
lint / lint (push) Successful in 2m24s
The old act_runner engine (v0.2.13 on the delphi-ci runner) mishandles
actions/setup-python's PATH injection: pip installs coverage into the
toolcache interpreter while `python3` in later steps resolves back to the
image's system Python, so unit/coverage jobs failed with "No module named
coverage". Newer runners (TrueNAS's v0.6.1) don't, which is why it only
broke on delphi.

The runner-images/act container already ships Python 3.12, and the job
container is ephemeral, so drop setup-python entirely and install straight
into the system Python with --break-system-packages. Every step now uses
one interpreter consistently, on any runner version. Also removes the
redundant setup-python step from the integration job (stdlib-only).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UoEZHDjv84ChoZbozQERhJ
2026-07-18 04:58:38 -04:00
didericis-claude 232dfdf37a refactor(gateway): replace egress_addon.py copy with a one-line shim
test / integration (pull_request) Successful in 1m15s
test / coverage (pull_request) Failing after 1m13s
test / unit (pull_request) Successful in 1m19s
mitmdump -s requires a file path, not a module. Instead of copying the
full egress_addon.py to /app/, write a one-line shim at image build time
that re-exports addons from the installed package. mitmdump finds the
addons list in the shim's namespace; all real addon code stays in
bot_bottle/egress_addon.py.
2026-07-18 07:59:57 +00:00
didericis-claude 9a0dd821ef refactor(gateway): invoke daemons via python3 -m instead of /app/ file copies
lint / lint (push) Successful in 2m20s
test / unit (pull_request) Successful in 1m12s
test / integration (pull_request) Successful in 26s
test / coverage (pull_request) Successful in 1m29s
supervise_server, git_http_backend, and gateway_init all have __main__
guards, so python3 -m bot_bottle.X replaces the individual COPY lines
to /app/. egress_addon.py stays as a file copy because mitmdump -s
requires a file path rather than a module reference.
2026-07-18 07:55:38 +00:00
didericis-claude 5ad3449e3b refactor(gateway): replace flat-file import shims with installed package
lint / lint (push) Successful in 2m22s
test / unit (pull_request) Successful in 1m12s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m23s
Install bot_bottle via pip in Dockerfile.gateway instead of COPYing
individual .py files flat under /app/. This eliminates the try/except
import shims in egress_addon_core, dlp_detectors, egress_addon,
supervise, supervise_server, and git_http_backend that existed only
to support the flat-bundle layout.

Adds bot_bottle/constants.py as a single source of truth for
IDENTITY_HEADER and GIT_GATE_TIMEOUT_SECS, removing the duplicated
literal definitions in egress_addon.py, supervise_server.py,
git_http_backend.py, and git_gate_render.py.

Test files updated to match: test_supervise_server.py drops the
sys.path.insert hack in favour of direct package imports; the
egress_addon test shims no longer pre-populate sys.modules with a
bare egress_addon_core alias.
2026-07-18 03:01:41 +00:00
20 changed files with 204 additions and 290 deletions
+14 -17
View File
@@ -34,13 +34,13 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
# No actions/setup-python: the runner image already ships Python 3.12,
# and older act_runner engines mishandle setup-python's PATH (coverage
# lands in one interpreter, `python3` resolves to another). Install
# straight into the ephemeral job container's system Python —
# --break-system-packages is safe because the container is disposable.
- name: Install dev requirements
run: python3 -m pip install -r requirements-dev.txt
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
- name: Run unit tests
run: python3 -m coverage run -m unittest discover -t . -s tests/unit -v
@@ -54,11 +54,8 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
# No actions/setup-python (see the note in the `unit` job); the
# container's system Python 3.12 runs the stdlib test suite directly.
- name: Show environment
run: |
python3 --version
@@ -88,13 +85,13 @@ jobs:
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: "3.12"
# No actions/setup-python: the runner image already ships Python 3.12,
# and older act_runner engines mishandle setup-python's PATH (coverage
# lands in one interpreter, `python3` resolves to another). Install
# straight into the ephemeral job container's system Python —
# --break-system-packages is safe because the container is disposable.
- name: Install dev requirements
run: python3 -m pip install -r requirements-dev.txt
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
- name: Combined coverage report (unit + integration)
run: PYTHON=python3 bash scripts/coverage.sh critical
+16 -27
View File
@@ -16,10 +16,12 @@
# Layout:
#
# /usr/bin/gitleaks gitleaks binary
# /app/egress_addon.py + siblings mitmproxy addon (egress)
# /app/egress_addon.py mitmproxy addon entry point
# /app/egress-entrypoint.sh mitmdump launcher
# /app/supervise_server.py + .py supervise MCP server
# /app/gateway_init.py PID 1 supervisor
# /usr/local/lib/python*/bot_bottle/ installed package (all daemons + shared modules)
# /app/egress_addon.py one-line shim: re-exports addons from package
# (mitmdump -s requires a file path, not a module)
# /etc/egress/routes.yaml bind-mounted at run time
# /etc/git-gate/pre-receive docker-cp'd at start time
# /git-gate-entrypoint.sh docker-cp'd at start time
# /git-gate/creds/* docker-cp'd at start time
@@ -87,27 +89,16 @@ RUN arch="${TARGETARCH:-$(dpkg --print-architecture)}" \
&& tar -xzf /tmp/gitleaks.tar.gz -C /usr/bin gitleaks \
&& rm /tmp/gitleaks.tar.gz
# Project Python: addon + server modules + the init supervisor.
# Kept flat under /app/ so mitmdump's loader resolves them as
# top-level siblings (absolute imports), matching the prior
# Dockerfile.egress / Dockerfile.supervise layout.
COPY bot_bottle/egress_addon_core.py /app/egress_addon_core.py
COPY bot_bottle/egress_dlp_config.py /app/egress_dlp_config.py
COPY bot_bottle/egress_addon.py /app/egress_addon.py
COPY bot_bottle/policy_resolver.py /app/policy_resolver.py
COPY bot_bottle/dlp_detectors.py /app/dlp_detectors.py
COPY bot_bottle/yaml_subset.py /app/yaml_subset.py
COPY bot_bottle/paths.py /app/paths.py
COPY bot_bottle/migrations.py /app/migrations.py
COPY bot_bottle/db_store.py /app/db_store.py
COPY bot_bottle/supervise_types.py /app/supervise_types.py
COPY bot_bottle/queue_store.py /app/queue_store.py
COPY bot_bottle/audit_store.py /app/audit_store.py
COPY bot_bottle/store_manager.py /app/store_manager.py
COPY bot_bottle/supervise.py /app/supervise.py
COPY bot_bottle/supervise_server.py /app/supervise_server.py
COPY bot_bottle/gateway_init.py /app/gateway_init.py
COPY bot_bottle/git_http_backend.py /app/git_http_backend.py
# Install bot_bottle as a proper package so entry-point scripts can use
# `from bot_bottle.X import Y` absolute imports. A rename or a missing
# module is caught at pip-install time — not at container runtime.
COPY pyproject.toml /src/
COPY bot_bottle/ /src/bot_bottle/
RUN pip install --no-cache-dir /src/
# mitmdump -s requires a file path, not a module. Write a one-line shim that
# re-exports `addons` from the installed package; mitmdump finds it there.
RUN printf 'from bot_bottle.egress_addon import addons\n' > /app/egress_addon.py
COPY bot_bottle/egress_entrypoint.sh /app/egress-entrypoint.sh
RUN chmod +x /app/egress-entrypoint.sh
@@ -126,10 +117,8 @@ RUN mkdir -p \
# subset the bottle uses.
EXPOSE 8888 9099 9418 9420 9100
# WORKDIR matches Dockerfile.supervise's prior layout so the
# in-app same-dir import in supervise_server.py stays deterministic.
WORKDIR /app
# PID 1 is the supervisor. It owns signal handling and exit-code
# propagation; no `exec` chain in the entrypoint itself.
ENTRYPOINT ["python3", "/app/gateway_init.py"]
ENTRYPOINT ["python3", "-m", "bot_bottle.gateway_init"]
+73 -31
View File
@@ -40,7 +40,7 @@ from abc import ABC, abstractmethod
from contextlib import AbstractContextManager
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Generic, Sequence, TypeVar
from typing import TYPE_CHECKING, Any, Generic, Sequence, TypeVar
from ..agent_provider import AgentProvisionPlan, get_provider, build_agent_provision_plan
from ..egress import EgressPlan
@@ -54,6 +54,9 @@ from ..workspace import WorkspacePlan, workspace_plan
from .print_util import print_multi, visible_agent_env_names
from .util import host_skill_dir
if TYPE_CHECKING:
from .freeze import CommitCancelled, Freezer, get_freezer
@dataclass(frozen=True)
class BottleSpec:
@@ -584,28 +587,63 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
Not called by the launch path or the test suite."""
# Import concrete backend classes AFTER the base types are defined, so
# each backend module can pull BottleSpec / BottlePlan / BottleBackend
# via `from . import ...` without hitting a partially-initialized module.
from .docker import DockerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .firecracker import FirecrackerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
from .macos_container import MacosContainerBottleBackend # noqa: E402 # pylint: disable=wrong-import-position
# Freezer is imported after the backend classes for the same reason:
# Freezer.commit_slug constructs ActiveAgent, which must be fully
# defined first.
from .freeze import CommitCancelled, Freezer, get_freezer # noqa: E402 # pylint: disable=wrong-import-position
# _backends is None until the first call to _get_backends(), at which
# point all three concrete backend classes are imported and instantiated.
# Keeping the imports out of module scope means that importing any
# backend sub-module (e.g. `backend.docker.util`) no longer drags the
# firecracker and macos-container implementations into memory.
#
# Tests may replace _backends with a {name: fake} dict via patch.object;
# _get_backends() returns the current module-level value as-is when it
# is not None, so test fakes take effect without triggering real imports.
_backends: dict[str, BottleBackend[Any, Any]] | None = None
# The dict is heterogeneous: each value is a BottleBackend specialized
# over its own plan type. Concrete plan types are erased here because
# the registry is selected at runtime and the CLI only needs the
# unparameterized methods (prepare → plan → launch(plan), cleanup, etc.).
_BACKENDS: dict[str, BottleBackend[Any, Any]] = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
}
def _get_backends() -> dict[str, BottleBackend[Any, Any]]:
"""Return the registry of all backend instances, loading lazily on first call."""
global _backends # pylint: disable=global-statement
if _backends is None:
from .docker import DockerBottleBackend
from .firecracker import FirecrackerBottleBackend
from .macos_container import MacosContainerBottleBackend
_backends = {
"docker": DockerBottleBackend(),
"firecracker": FirecrackerBottleBackend(),
"macos-container": MacosContainerBottleBackend(),
}
return _backends
def __getattr__(name: str) -> Any:
"""Lazily surface concrete backend classes and freeze symbols at the
package level so existing `from bot_bottle.backend import X` and
`patch.object(backend_mod, X, ...)` call-sites keep working without
forcing an import of every backend at module-init time."""
if name == "DockerBottleBackend":
from .docker import DockerBottleBackend
globals()[name] = DockerBottleBackend
return DockerBottleBackend
if name == "FirecrackerBottleBackend":
from .firecracker import FirecrackerBottleBackend
globals()[name] = FirecrackerBottleBackend
return FirecrackerBottleBackend
if name == "MacosContainerBottleBackend":
from .macos_container import MacosContainerBottleBackend
globals()[name] = MacosContainerBottleBackend
return MacosContainerBottleBackend
if name == "CommitCancelled":
from .freeze import CommitCancelled
globals()[name] = CommitCancelled
return CommitCancelled
if name == "Freezer":
from .freeze import Freezer
globals()[name] = Freezer
return Freezer
if name == "get_freezer":
from .freeze import get_freezer
globals()[name] = get_freezer
return get_freezer
raise AttributeError(f"module {__name__!r} has no attribute {name!r}")
def get_bottle_backend(
@@ -623,10 +661,11 @@ def get_bottle_backend(
Dies with a pointer at the known backends if the chosen name
isn't implemented."""
resolved = name or os.environ.get("BOT_BOTTLE_BACKEND") or _default_backend_name()
if resolved not in _BACKENDS:
known = ", ".join(sorted(_BACKENDS))
backends = _get_backends()
if resolved not in backends:
known = ", ".join(sorted(backends))
die(f"unknown backend {resolved!r}; known backends: {known}")
return _BACKENDS[resolved]
return backends[resolved]
def _default_backend_name() -> str:
@@ -636,16 +675,17 @@ def _default_backend_name() -> str:
# `firecracker` binary isn't installed yet: selecting it here routes
# start through firecracker's preflight, which prints an install
# pointer, instead of silently falling back to docker.
from .firecracker import FirecrackerBottleBackend
if FirecrackerBottleBackend.is_host_capable():
return "firecracker"
return "docker"
def known_backend_names() -> tuple[str, ...]:
"""Sorted tuple of all backend keys in `_BACKENDS`. Used by
"""Sorted tuple of all backend keys in `_get_backends()`. Used by
argparse (`--backend` choices) and the dashboard's backend
picker."""
return tuple(sorted(_BACKENDS))
return tuple(sorted(_get_backends()))
def has_backend(name: str) -> bool:
@@ -657,9 +697,10 @@ def has_backend(name: str) -> bool:
Returns False for unknown names so callers can pass
arbitrary input without separate validation."""
if name not in _BACKENDS:
backends = _get_backends()
if name not in backends:
return False
return _BACKENDS[name].is_available()
return backends[name].is_available()
def enumerate_active_agents() -> list[ActiveAgent]:
@@ -675,10 +716,11 @@ def enumerate_active_agents() -> list[ActiveAgent]:
deterministic tiebreaker. Agents with missing metadata
(`started_at == ""`) sort first."""
out: list[ActiveAgent] = []
for name in known_backend_names():
if not has_backend(name):
backends = _get_backends()
for name in sorted(backends):
if not backends[name].is_available():
continue
out.extend(_BACKENDS[name].enumerate_active())
out.extend(backends[name].enumerate_active())
out.sort(key=lambda a: (a.started_at, a.slug))
return out
+7 -34
View File
@@ -8,7 +8,7 @@ import os
import re
import shutil
import subprocess
from typing import Iterable, Iterator
from typing import Iterator
from ...docker_cmd import run_docker
from ...log import die, info
@@ -32,12 +32,7 @@ def container_name_candidates(base: str) -> Iterator[str]:
def runsc_available() -> bool:
"""Return True if the Docker daemon has the gVisor (`runsc`) runtime
registered. Called once per prepare; the result lives on the plan."""
r = subprocess.run(
["docker", "info", "--format", "{{json .Runtimes}}"],
capture_output=True,
text=True,
check=False,
)
r = run_docker(["docker", "info", "--format", "{{json .Runtimes}}"])
return r.returncode == 0 and "runsc" in r.stdout
@@ -51,20 +46,15 @@ def require_docker() -> None:
def image_exists(ref: str) -> bool:
return _silent_run(["docker", "image", "inspect", ref]) == 0
return run_docker(["docker", "image", "inspect", ref]).returncode == 0
def container_exists(name: str) -> bool:
"""Returns True if a container (running or stopped) with the given
name exists. Uses `docker ps -a -q -f name=^<name>$` so substring
matches don't false-positive."""
result = subprocess.run(
["docker", "ps", "-a", "-q", "-f", f"name=^{name}$"],
capture_output=True,
text=True,
check=True,
)
return bool(result.stdout.strip())
result = run_docker(["docker", "ps", "-a", "-q", "-f", f"name=^{name}$"])
return result.returncode == 0 and bool(result.stdout.strip())
def force_remove_container(name: str) -> None:
@@ -72,12 +62,7 @@ def force_remove_container(name: str) -> None:
doesn't — and the rm itself is best-effort (errors swallowed) so
this is safe to register as a teardown callback."""
if container_exists(name):
subprocess.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
)
run_docker(["docker", "rm", "-f", name])
def docker_exec_root(container: str, argv: list[str]) -> None:
@@ -205,22 +190,10 @@ def verify_agent_image(image: str, argv: tuple[str, ...]) -> None:
def commit_container(container_name: str, image_tag: str) -> None:
"""Run `docker commit <container_name> <image_tag>` to snapshot the
running container's filesystem state as a local Docker image."""
result = subprocess.run(
["docker", "commit", container_name, image_tag],
capture_output=True, text=True, check=False,
)
result = run_docker(["docker", "commit", container_name, image_tag])
if result.returncode != 0:
die(
f"docker commit {container_name!r}{image_tag!r} failed: "
f"{(result.stderr or '').strip() or '<no stderr>'}"
)
info(f"committed {container_name!r}{image_tag!r}")
def _silent_run(cmd: Iterable[str]) -> int:
return subprocess.run(
list(cmd),
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
).returncode
+17
View File
@@ -0,0 +1,17 @@
"""Shared wire-protocol constants for gateway-bundled modules.
Single source of truth for values that appear across the egress addon,
git-http backend, supervise server, and git-gate renderer. Importing
from this module instead of duplicating the literals means a rename is
a one-line change and is caught by the type checker at the import site."""
# App-layer identity token header. Delivered as proxy credentials
# (HTTPS_PROXY=http://<bottle_id>:<token>@gw) by launch; the egress
# addon reads and strips it, the supervise server and git-http backend
# read it for attribution, and none of them forward it upstream.
IDENTITY_HEADER = "x-bot-bottle-identity"
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
# git daemon (--timeout/--init-timeout), the access-hook subprocess in
# git_http_backend, and the git http-backend CGI subprocess.
GIT_GATE_TIMEOUT_SECS = 15
+3 -7
View File
@@ -3,9 +3,8 @@
Pure Python, no mitmproxy dependency. Each detector is a module-level
function returning `ScanResult | None`.
Ships flat into the gateway image alongside
`egress_addon_core.py` — both this file and the package source use
the same try/except import shim pattern.
Available in the gateway via the installed `bot_bottle` package
(see `Dockerfile.gateway`).
"""
from __future__ import annotations
@@ -20,10 +19,7 @@ from math import log2
from collections import Counter
from urllib.parse import quote as url_quote
try:
from egress_addon_core import ScanResult # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
from .egress_addon_core import ScanResult
from .egress_addon_core import ScanResult
# ---------------------------------------------------------------------------
+5 -26
View File
@@ -15,7 +15,9 @@ import typing
from mitmproxy import http # type: ignore[import-not-found] # pylint: disable=import-error
from egress_addon_core import ( # type: ignore[import-not-found] # pylint: disable=import-error
from bot_bottle.constants import IDENTITY_HEADER
from bot_bottle.dlp_detectors import redact_tokens, strip_crlf
from bot_bottle.egress_addon_core import (
LOG_BLOCKS,
LOG_FULL,
DEFAULT_OUTBOUND_ON_MATCH,
@@ -38,24 +40,8 @@ from egress_addon_core import ( # type: ignore[import-not-found] # pylint: dis
scan_inbound,
scan_outbound,
)
try:
from dlp_detectors import redact_tokens, strip_crlf # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
from bot_bottle.dlp_detectors import ( # type: ignore[import-not-found]
redact_tokens,
strip_crlf,
)
try:
import supervise as _sv # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
from bot_bottle import supervise as _sv # type: ignore[import-not-found]
try:
from policy_resolver import PolicyResolver # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
from bot_bottle.policy_resolver import PolicyResolver
from bot_bottle import supervise as _sv
from bot_bottle.policy_resolver import PolicyResolver
INTROSPECT_HOST = "_egress.local"
@@ -66,13 +52,6 @@ INTROSPECT_HOST = "_egress.local"
# back to — so an unset value is a fatal misconfiguration (see __init__).
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
# App-layer identity token. Delivered as proxy credentials
# (`HTTPS_PROXY=http://<bottle_id>:<token>@gw`): clients honor it as part of
# the proxy protocol without app changes, and the addon reads + strips it so
# it never leaks upstream. The legacy `x-bot-bottle-identity` request header
# is still stripped defensively (git-http uses that header on its own port).
IDENTITY_HEADER = "x-bot-bottle-identity"
# Per-flow key under which `request()` stashes the resolved (Config, supervise
# slug, env) so the later `response()` and `websocket_message()` hooks scan
# against the *calling bottle's* policy — the same one the request was decided
+16 -32
View File
@@ -6,9 +6,9 @@ exercise the parse + decision functions without depending on the
`mitmproxy.http.HTTPFlow` API and is loaded inside the gateway
container.
Imports: stdlib + `yaml_subset` (which is itself stdlib-only and
ships flat into the gateway image alongside this file —
see `Dockerfile.gateway`)."""
Imports: stdlib + sibling package modules (`yaml_subset`,
`egress_dlp_config`). Available in the gateway via the installed
`bot_bottle` package (see `Dockerfile.gateway`)."""
from __future__ import annotations
@@ -16,36 +16,20 @@ import re
import typing
from dataclasses import dataclass
try:
from yaml_subset import YamlSubsetError, parse_yaml_subset # type: ignore[import-not-found]
except ImportError: # pragma: no cover - host-side path
from .yaml_subset import YamlSubsetError, parse_yaml_subset
from .yaml_subset import YamlSubsetError, parse_yaml_subset
# DLP detector-config parsing lives in a sibling module (also flat-bundled
# into the gateway — see Dockerfile.gateway). Re-exported below so existing
# `from egress_addon_core import ON_MATCH_*` callers keep working.
try:
from egress_dlp_config import ( # type: ignore[import-not-found]
DEFAULT_OUTBOUND_ON_MATCH,
INBOUND_DETECTOR_NAMES,
ON_MATCH_BLOCK,
ON_MATCH_REDACT,
ON_MATCH_SUPERVISE,
OUTBOUND_DETECTOR_NAMES,
OUTBOUND_ON_MATCH_VALUES,
parse_dlp_block,
)
except ImportError: # pragma: no cover - host-side path
from .egress_dlp_config import (
DEFAULT_OUTBOUND_ON_MATCH,
INBOUND_DETECTOR_NAMES,
ON_MATCH_BLOCK,
ON_MATCH_REDACT,
ON_MATCH_SUPERVISE,
OUTBOUND_DETECTOR_NAMES,
OUTBOUND_ON_MATCH_VALUES,
parse_dlp_block,
)
# DLP detector-config parsing lives in a sibling module. Re-exported below
# so existing `from egress_addon_core import ON_MATCH_*` callers keep working.
from .egress_dlp_config import (
DEFAULT_OUTBOUND_ON_MATCH,
INBOUND_DETECTOR_NAMES,
ON_MATCH_BLOCK,
ON_MATCH_REDACT,
ON_MATCH_SUPERVISE,
OUTBOUND_DETECTOR_NAMES,
OUTBOUND_ON_MATCH_VALUES,
parse_dlp_block,
)
# ---------------------------------------------------------------------------
+2 -2
View File
@@ -78,8 +78,8 @@ def _env_for_daemon(name: str, base_env: dict[str, str]) -> dict[str, str]:
_DAEMONS: tuple[_DaemonSpec, ...] = (
_DaemonSpec("egress", ("/bin/sh", "/app/egress-entrypoint.sh")),
_DaemonSpec("git-gate", ("/bin/sh", "/git-gate-entrypoint.sh")),
_DaemonSpec("git-http", ("python3", "/app/git_http_backend.py")),
_DaemonSpec("supervise", ("python3", "/app/supervise_server.py")),
_DaemonSpec("git-http", ("python3", "-m", "bot_bottle.git_http_backend")),
_DaemonSpec("supervise", ("python3", "-m", "bot_bottle.supervise_server")),
)
+1 -7
View File
@@ -14,18 +14,12 @@ import shlex
from dataclasses import dataclass
from pathlib import Path
from .constants import GIT_GATE_TIMEOUT_SECS, IDENTITY_HEADER
from .manifest import ManifestBottle, ManifestGitEntry
# Short network alias for git-gate inside the gateway. The
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
GIT_GATE_HOSTNAME = "git-gate"
# App-layer identity token header the agent's git sends to git-http and the
# gateway validates (mirrors egress_addon / git_http_backend IDENTITY_HEADER).
IDENTITY_HEADER = "x-bot-bottle-identity"
# Shared timeout (seconds) for all git-gate subprocess and CGI calls:
# git daemon (--timeout/--init-timeout), the access-hook subprocess in
# git_http_backend, and the git http-backend CGI subprocess.
GIT_GATE_TIMEOUT_SECS = 15
@dataclass(frozen=True)
+2 -23
View File
@@ -26,16 +26,8 @@ from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from pathlib import Path
from urllib.parse import urlsplit
# policy_resolver ships flat alongside this file in the gateway
# image (see Dockerfile.gateway); the bot_bottle.* fallback is the
# host-side / test path. Mirrors egress_addon's import shape.
try:
from policy_resolver import ( # type: ignore[import-not-found]
PolicyResolveError,
PolicyResolver,
)
except ImportError: # pragma: no cover - host-side path
from bot_bottle.policy_resolver import PolicyResolveError, PolicyResolver
from bot_bottle.constants import GIT_GATE_TIMEOUT_SECS, IDENTITY_HEADER
from bot_bottle.policy_resolver import PolicyResolveError, PolicyResolver
DEFAULT_PORT = 9420
@@ -46,12 +38,6 @@ DEFAULT_PORT = 9420
# repo-root fallback.
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
# App-layer identity token (defense-in-depth over the source-IP invariant);
# the agent injects it, the backend reads it for attribution and never
# forwards it to `git http-backend`. Mirrors egress_addon.IDENTITY_HEADER
# (duplicated, not imported: egress_addon pulls in mitmproxy).
IDENTITY_HEADER = "x-bot-bottle-identity"
# The base under which each bottle's `<bottle_id>` repo namespace is nested.
DEFAULT_REPO_ROOT = "/git"
@@ -89,13 +75,6 @@ def resolve_sandbox_root(
return None # bottle_id tried to escape the root → deny
return namespace
# Mirrors git_gate_render.GIT_GATE_TIMEOUT_SECS. Duplicated rather than
# imported: this module ships as a flat top-level sibling in the gateway
# bundle image (see Dockerfile.gateway), not as part of the bot_bottle
# package, so `bot_bottle.git_gate` and its dependency chain aren't
# available at runtime.
GIT_GATE_TIMEOUT_SECS = 15
# Bound memory use while still allowing ordinary git push packfiles.
MAX_BODY_BYTES = 100 * 1024 * 1024
+1 -2
View File
@@ -22,8 +22,7 @@ closed too rather than silently serving stale or empty policy.
The resolved value is the policy blob the orchestrator stores verbatim; the
consumer parses it (e.g. the egress addon's `load_config`). This module is
stdlib-only and free of bot-bottle imports so it can be COPYed flat into
the gateway.
stdlib-only and free of bot-bottle imports.
"""
from __future__ import annotations
+16 -34
View File
@@ -37,40 +37,22 @@ from abc import ABC
from dataclasses import dataclass
from pathlib import Path
try:
from .supervise_types import (
ACTION_OPERATOR_EDIT,
AuditEntry,
Proposal,
Response,
STATUSES,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOLS,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_EGRESS_TOKEN_ALLOW,
TOOL_GITLEAKS_ALLOW,
TOOL_LIST_EGRESS_ROUTES,
)
except ImportError:
from supervise_types import ( # type: ignore[import-not-found,no-redef] # pylint: disable=import-error,no-name-in-module
ACTION_OPERATOR_EDIT,
AuditEntry,
Proposal,
Response,
STATUSES,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOLS,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_EGRESS_TOKEN_ALLOW,
TOOL_GITLEAKS_ALLOW,
TOOL_LIST_EGRESS_ROUTES,
)
from .supervise_types import (
ACTION_OPERATOR_EDIT,
AuditEntry,
Proposal,
Response,
STATUSES,
STATUS_APPROVED,
STATUS_MODIFIED,
STATUS_REJECTED,
TOOLS,
TOOL_EGRESS_ALLOW,
TOOL_EGRESS_BLOCK,
TOOL_EGRESS_TOKEN_ALLOW,
TOOL_GITLEAKS_ALLOW,
TOOL_LIST_EGRESS_ROUTES,
)
try:
+8 -20
View File
@@ -26,9 +26,8 @@ Speaks MCP over HTTP+JSON-RPC. Methods handled:
Everything else returns JSON-RPC error -32601 (method not found).
Stdlib-only. The Dockerfile copies this file + bot_bottle/supervise.py
into the image; the server imports `supervise` for the queue / Proposal
plumbing.
The Dockerfile copies this script to /app/supervise_server.py and installs
the bot_bottle package so its `from bot_bottle.*` imports resolve.
"""
from __future__ import annotations
@@ -42,29 +41,18 @@ import time
import typing
from dataclasses import dataclass, replace
try:
# Same-directory imports inside the bundle container; these files are
# COPYed flat under /app by Dockerfile.gateway.
from egress_addon_core import (
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
)
from policy_resolver import PolicyResolveError, PolicyResolver
import supervise as _sv
except ModuleNotFoundError:
# Package imports for host-side tests and tooling.
from .egress_addon_core import (
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
)
from .policy_resolver import PolicyResolveError, PolicyResolver
from . import supervise as _sv
from bot_bottle.constants import IDENTITY_HEADER
from bot_bottle.egress_addon_core import (
LOG_OFF, load_config, resolve_client_context, route_to_yaml_dict,
)
from bot_bottle.policy_resolver import PolicyResolveError, PolicyResolver
from bot_bottle import supervise as _sv
# --- JSON-RPC / MCP plumbing ----------------------------------------------
MCP_PROTOCOL_VERSION = "2024-11-05"
# App-layer identity token header (mirrors egress_addon / git_http_backend).
IDENTITY_HEADER = "x-bot-bottle-identity"
SERVER_NAME = "bot-bottle-supervise"
SERVER_VERSION = "0.1.0"
+8
View File
@@ -0,0 +1,8 @@
[build-system]
requires = ["setuptools>=68"]
build-backend = "setuptools.backends.legacy:build"
[project]
name = "bot-bottle"
version = "0.0.0"
requires-python = ">=3.11"
+8 -8
View File
@@ -40,7 +40,7 @@ class TestGetBottleBackend(unittest.TestCase):
return True
with patch.dict(os.environ, {}, clear=True), \
patch.object(backend_mod, "_BACKENDS", {
patch.object(backend_mod, "_backends", {
"macos-container": _FakeBackend(),
"docker": _FakeBackend(),
}):
@@ -61,7 +61,7 @@ class TestGetBottleBackend(unittest.TestCase):
with patch.dict(os.environ, {}, clear=True), \
patch.object(backend_mod.FirecrackerBottleBackend,
"is_host_capable", classmethod(lambda cls: False)), \
patch.object(backend_mod, "_BACKENDS", {
patch.object(backend_mod, "_backends", {
"macos-container": _FakeBackend("macos-container", False),
"docker": _FakeBackend("docker", True),
}):
@@ -83,7 +83,7 @@ class TestGetBottleBackend(unittest.TestCase):
with patch.dict(os.environ, {}, clear=True), \
patch.object(backend_mod.FirecrackerBottleBackend,
"is_host_capable", classmethod(lambda cls: True)), \
patch.object(backend_mod, "_BACKENDS", {
patch.object(backend_mod, "_backends", {
"macos-container": _FakeBackend("macos-container", False),
"firecracker": _FakeBackend("firecracker", False),
"docker": _FakeBackend("docker", True),
@@ -133,7 +133,7 @@ class TestEnumerateActiveAgents(unittest.TestCase):
return self._items
with patch.object(
backend_mod, "_BACKENDS",
backend_mod, "_backends",
{"docker": _FakeBackend([a]), "firecracker": _FakeBackend([b])},
):
self.assertEqual([a, b], enumerate_active_agents())
@@ -167,7 +167,7 @@ class TestEnumerateActiveAgents(unittest.TestCase):
return self._items
with patch.object(
backend_mod, "_BACKENDS",
backend_mod, "_backends",
{
"docker": _FakeBackend([newer, tie_b]),
"firecracker": _FakeBackend([missing_metadata, tie_a]),
@@ -187,7 +187,7 @@ class TestEnumerateActiveAgents(unittest.TestCase):
return []
with patch.object(
backend_mod, "_BACKENDS",
backend_mod, "_backends",
{"docker": _FakeBackend(), "firecracker": _FakeBackend()},
):
self.assertEqual([], enumerate_active_agents())
@@ -218,7 +218,7 @@ class TestEnumerateActiveAgents(unittest.TestCase):
return self._items
with patch.object(
backend_mod, "_BACKENDS",
backend_mod, "_backends",
{
"docker": _FakeBackend([present], available=True),
"firecracker": _FakeBackend([hidden], available=False),
@@ -234,7 +234,7 @@ class TestHasBackend(unittest.TestCase):
return False
with patch.object(
backend_mod, "_BACKENDS", {"docker": _FakeBackend()},
backend_mod, "_backends", {"docker": _FakeBackend()},
):
from bot_bottle.backend import has_backend
self.assertFalse(has_backend("docker"))
+3 -3
View File
@@ -29,7 +29,7 @@ def _fail(stderr: str = "boom") -> subprocess.CompletedProcess: # type: ignore
class TestCommitContainer(unittest.TestCase):
def test_runs_docker_commit(self):
with patch.object(
docker_mod.subprocess, "run", return_value=_ok(),
docker_mod, "run_docker", return_value=_ok(),
) as run, patch.object(docker_mod, "info"):
docker_mod.commit_container(
"bot-bottle-dev-abc12",
@@ -47,7 +47,7 @@ class TestCommitContainer(unittest.TestCase):
def test_dies_on_docker_commit_failure(self):
with patch.object(
docker_mod.subprocess, "run", return_value=_fail("No such container"),
docker_mod, "run_docker", return_value=_fail("No such container"),
), patch.object(
docker_mod, "die", side_effect=SystemExit("die"),
) as die:
@@ -58,7 +58,7 @@ class TestCommitContainer(unittest.TestCase):
def test_die_message_includes_image_tag(self):
with patch.object(
docker_mod.subprocess, "run", return_value=_fail("boom"),
docker_mod, "run_docker", return_value=_fail("boom"),
), patch.object(
docker_mod, "die", side_effect=SystemExit("die"),
) as die:
@@ -18,7 +18,7 @@ from unittest.mock import patch
# ---------------------------------------------------------------------------
# Gateway-import shims — must run before importing egress_addon
# mitmproxy stub — must run before importing egress_addon
# ---------------------------------------------------------------------------
def _ensure_shims() -> None:
@@ -32,9 +32,6 @@ def _ensure_shims() -> None:
setattr(_mm, "http", _mh)
sys.modules["mitmproxy"] = _mm
sys.modules["mitmproxy.http"] = _mh
if "egress_addon_core" not in sys.modules:
import bot_bottle.egress_addon_core as _core
sys.modules["egress_addon_core"] = _core
_ensure_shims()
@@ -190,9 +190,6 @@ def _ensure_shims() -> None:
setattr(mh, "Response", _Response)
if not hasattr(mh, "HTTPFlow"):
setattr(mh, "HTTPFlow", object)
if "egress_addon_core" not in sys.modules:
import bot_bottle.egress_addon_core as _core
sys.modules["egress_addon_core"] = _core
_ensure_shims()
+3 -10
View File
@@ -2,7 +2,6 @@
import http.client
import json
import sys
import tempfile
import threading
import time
@@ -13,15 +12,9 @@ from unittest.mock import patch
from tests.unit import use_bottle_root
# The server module loads `supervise` via same-directory import inside
# the container (Dockerfile.supervise WORKDIRs into /app). For tests
# we mirror that by injecting bot_bottle/ onto sys.path under the
# bare name `supervise`.
sys.path.insert(0, str(Path(__file__).resolve().parent.parent.parent / "bot_bottle"))
import supervise as _sv # noqa: E402 # type: ignore
import queue_store as _qs # noqa: E402 # type: ignore
import audit_store as _as # noqa: E402 # type: ignore
from bot_bottle import supervise as _sv
from bot_bottle import queue_store as _qs
from bot_bottle import audit_store as _as
from bot_bottle import supervise_server # noqa: E402
from bot_bottle.supervise_server import (