Compare commits

...

5 Commits

Author SHA1 Message Date
didericis d0b35b5506 feat(git-gate+orchestrator): slice 13c(i) — per-bottle git-gate provisioning render
lint / lint (push) Successful in 2m9s
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m14s
The consolidated gateway serves every bottle's repos under /git/<bottle_id>/
(slice 10), so each bottle's bare repos + per-repo credential config must be
provisioned into that namespace. This adds the renderer for that; placing
the creds + running it inside the gateway is the launch-wiring step.

- Extract the credential-wiring `init_repo` shell into a shared
  `_git_gate_init_repo_fn(repo_root, creds_dir)` — one source for the
  security-sensitive logic. The single-tenant daemon entrypoint is
  byte-unchanged (still /git + /git-gate/creds; existing tests green).
- git_gate_render_provision(bottle_id, upstreams): posix-sh that inits ONE
  bottle's bare repos under /git/<bottle_id>/ reading creds from
  /git-gate/creds/<bottle_id>/. Init-only (no `git daemon` — the shared
  gateway already serves). Isolating each bottle's repo root + creds dir by
  id is what keeps one bottle's push credentials out of another's repos.
- bottle_id is embedded unquoted in the script, so it's restricted to a
  shell/path-safe alphabet (registry ids are token_hex; defense in depth).

pyright 0 errors; pylint 9.83/10; unit suite green (1724 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 21:15:21 -04:00
didericis ea0c070cfe feat(orchestrator): slice 13b — consolidated registration inputs (egress policy)
lint / lint (push) Successful in 2m10s
test / unit (pull_request) Successful in 1m12s
test / integration (pull_request) Successful in 29s
test / coverage (pull_request) Successful in 1m20s
Bridges the per-bottle `prepare` output to the consolidated registry:
`registration_inputs(plan)` turns a prepared egress plan into the
backend-neutral inputs `Orchestrator.launch_bottle` takes.

- egress_policy(plan): the policy blob = the exact routes YAML the
  per-bottle sidecar read from a file; the multi-tenant gateway's
  PolicyResolver now fetches it from the registry per request instead.
  Same egress_render_routes, so a bottle's allow-list is byte-identical
  moving onto the shared gateway.
- RegistrationInputs bundles policy + metadata; metadata carries the human
  slug so the shared registry can map a minted bottle id back to a name
  (console / supervise display).

Host-side glue (imports bot_bottle.egress) used by the launch path — not
the lean orchestrator control-plane process. Not yet wired into launch
(that's 13c/13d), consistent with the slice-6/10 pattern of landing the
primitive before the cut-over.

Key test: the policy round-trips through load_config back to the same
routes + log level — the resolver-served policy reconstructs the per-bottle
egress config exactly.

pyright 0 errors; pylint 9.83/10; unit suite green (1718 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 20:40:38 -04:00
didericis 24df322c31 feat(orchestrator): slice 13a — orchestrator process lifecycle (idempotent singleton)
lint / lint (push) Successful in 2m9s
test / unit (pull_request) Successful in 1m4s
test / integration (pull_request) Successful in 20s
test / coverage (pull_request) Successful in 1m14s
First sub-slice of docker launch integration: the foundation the CLI needs
to ensure exactly one orchestrator control plane + shared gateway is up
before registering/launching bottles. Does NOT touch the real launch path
yet — it's the lifecycle primitive the cut-over slices build on.

- OrchestratorProcess.ensure_running(): idempotent singleton — returns the
  control-plane URL if a healthy one already answers /health, else spawns
  `python -m bot_bottle.orchestrator --broker docker --gateway` detached
  (start_new_session, output tee'd to <root>/orchestrator.log) and polls
  /health until healthy or timeout (OrchestratorStartError).
- The control-plane port is the singleton key: a second orchestrator can't
  bind it, so a stray double-start fails fast rather than forking a rival.
- Host-process (not container) per the PRD's dev-harness sequencing — it
  already has the host user's docker access to broker launches; the
  data-plane gateway it manages is the container.

pyright 0 errors; pylint 9.83/10; unit suite green (1714 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 20:33:14 -04:00
didericis 80d75de08a feat(supervise+orchestrator): slice 12 — multi-tenant git-gate hook + supervise server
lint / lint (push) Successful in 2m2s
test / unit (pull_request) Successful in 1m0s
test / integration (pull_request) Successful in 22s
test / coverage (pull_request) Successful in 1m11s
Finishes the supervise data-plane writers for the shared gateway. Both
still keyed off a single SUPERVISE_BOTTLE_SLUG env; now each proposal is
attributed to the calling bottle, keeping slices 10/11's per-bottle model.

- git_http_backend: in consolidated mode stamp SUPERVISE_BOTTLE_SLUG=
  <bottle_id> into the CGI env. The gitleaks-allow pre-receive hook runs as
  a child of the `git http-backend` we spawn, so it inherits the stamp and
  queues under the right bottle — no change to the (fragile) embedded hook.
  The bottle id is the namespaced root's final component (slice 10), the
  same key egress uses. Single-tenant leaves the container-stamped slug.
- supervise_server: resolve the proposal slug per request by source IP when
  BOT_BOTTLE_ORCHESTRATOR_URL is set (`_attributed_config`), fail-closed —
  an unattributed / unreachable source raises rather than queue under the
  wrong or empty slug. `serve`/`main` build + attach the resolver and make
  the env slug optional in consolidated mode. Single-tenant unchanged.

Tests: a real git push proving the slug reaches the hook's env; the
supervise attribution matrix (single-tenant slug kept, source-IP bottle
bound, unattributed + resolver-error fail closed).

Remaining supervise gap (noted): websocket DLP still self.config-only.

pyright 0 errors; pylint 9.83/10; unit suite green (1705 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 20:04:26 -04:00
didericis 8827c64a83 feat(supervise+orchestrator): slice 11 — per-bottle supervise queue + DLP safelist
lint / lint (push) Successful in 2m4s
test / unit (pull_request) Successful in 1m6s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m15s
Consolidated egress ran every bottle through one process but keyed the
supervise proposal queue off a single SUPERVISE_BOTTLE_SLUG env and kept
one *global* DLP safelist — so in the shared gateway an operator's token
approval for bottle A would (a) be attributed to the wrong bottle and
(b) leak into bottle B's DLP scan (A's approved secret passes B's egress).
This slice keys both per bottle, resolved by source IP.

- policy_resolver: add `resolve_policy_and_bottle_id` — policy + bottle id
  in one `/resolve`, so egress keys routing *and* the supervise
  queue/safelist from a single round-trip. Fail-closed (403 -> (None,None)).
- egress_addon_core: add `resolve_client_context` (+ `ContextResolverLike`)
  returning `(Config, bottle_id)`, sharing the fail-closed parse with
  `resolve_client_config` via `_config_from_policy`.
- egress_addon: `_active_config` -> `_resolve_flow` returns `(Config, slug)`;
  `safe_tokens` set -> per-bottle `_safe_tokens_for(slug)`; the token-allow
  write/await/archive + the approved-token add all use the resolved slug.
  Single-tenant (no resolver) unchanged — slug = the env SUPERVISE_BOTTLE_SLUG.

New tests cover the resolver, the fail-closed context matrix, and the
cross-tenant isolation (an approval lands only in the calling bottle's
safelist; the proposal is keyed by the source-IP-attributed bottle;
unattributed IPs can't supervise).

Out of scope (noted): the git-gate gitleaks-allow hook + supervise_server
agent-proposal paths, and websocket DLP (still self.config-only, inert in
consolidated mode) — follow-up slices.

pyright 0 errors; pylint 9.83/10; unit suite green (1700 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 19:48:29 -04:00
18 changed files with 903 additions and 65 deletions
+52 -29
View File
@@ -32,7 +32,7 @@ from egress_addon_core import ( # type: ignore[import-not-found] # pylint: dis
is_git_push_request, is_git_push_request,
load_config, load_config,
match_route, match_route,
resolve_client_config, resolve_client_context,
outbound_scan_headers, outbound_scan_headers,
route_to_yaml_dict, route_to_yaml_dict,
scan_inbound, scan_inbound,
@@ -99,17 +99,29 @@ class EgressAddon:
# Absent → single-tenant (static routes file); behaviour unchanged. # Absent → single-tenant (static routes file); behaviour unchanged.
orch_url = os.environ.get(ORCHESTRATOR_URL_ENV, "").strip() orch_url = os.environ.get(ORCHESTRATOR_URL_ENV, "").strip()
self._resolver = PolicyResolver(orch_url) if orch_url else None self._resolver = PolicyResolver(orch_url) if orch_url else None
# Tokens the operator has approved this session (PRD 0062). In-memory # Tokens the operator has approved this session (PRD 0062), keyed by
# only — a restart re-prompts. Mutated only from the asyncio loop that # bottle so the shared gateway keeps each bottle's safelist separate —
# runs the addon hooks, so no lock is needed. # a global set would let bottle A's approved secret pass bottle B's DLP
self.safe_tokens: set[str] = set() # scan. In-memory only (a restart re-prompts); mutated only from the
# asyncio loop that runs the addon hooks, so no lock is needed.
self._safe_tokens: dict[str, set[str]] = {}
self._supervise_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "").strip() self._supervise_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "").strip()
self._token_allow_timeout = _token_allow_timeout_from_env(os.environ) self._token_allow_timeout = _token_allow_timeout_from_env(os.environ)
self._reload(initial=True) self._reload(initial=True)
self._install_sighup() self._install_sighup()
def _supervise_available(self) -> bool: @staticmethod
return bool(self._supervise_slug) def _supervise_available(slug: str) -> bool:
"""Supervise is reachable for this request iff we resolved a bottle to
attribute its proposals to (single-tenant env slug, or a source-IP
-attributed bottle id). Empty → fail closed (no queue to write to)."""
return bool(slug)
def _safe_tokens_for(self, slug: str) -> set[str]:
"""This bottle's operator-approved DLP safelist (PRD 0062), created on
first use. Keyed by bottle so the shared gateway never leaks one
bottle's approved token into another's scan."""
return self._safe_tokens.setdefault(slug, set())
def _reload(self, *, initial: bool = False) -> None: def _reload(self, *, initial: bool = False) -> None:
try: try:
@@ -218,19 +230,21 @@ class EgressAddon:
+ "\n" + "\n"
) )
def _active_config(self, flow: http.HTTPFlow) -> Config: def _resolve_flow(self, flow: http.HTTPFlow) -> "tuple[Config, str]":
"""The Config to apply to this request. Single-tenant → the static """The `(Config, supervise slug)` to apply to this request. Single-tenant
`self.config`. Consolidated → the calling bottle's Config, resolved → the static `self.config` and the env slug. Consolidated → the calling
by source IP (fail-closed to deny-all if unattributed). The identity bottle's Config and its bottle id, resolved by source IP in one
token, if the agent injected one, is read then stripped so it never round-trip (fail-closed to deny-all + empty slug if unattributed). The
leaks upstream.""" identity token, if the agent injected one, is read then stripped so it
never leaks upstream. The slug keys both the proposal queue and the
per-bottle safelist, so an approval only ever affects its own bottle."""
if self._resolver is None: if self._resolver is None:
return self.config return self.config, self._supervise_slug
conn = flow.client_conn conn = flow.client_conn
client_ip = conn.peername[0] if conn and conn.peername else "" client_ip = conn.peername[0] if conn and conn.peername else ""
token = flow.request.headers.get(IDENTITY_HEADER, "") token = flow.request.headers.get(IDENTITY_HEADER, "")
flow.request.headers.pop(IDENTITY_HEADER, None) flow.request.headers.pop(IDENTITY_HEADER, None)
return resolve_client_config(self._resolver, client_ip, token) return resolve_client_context(self._resolver, client_ip, token)
async def request(self, flow: http.HTTPFlow) -> None: async def request(self, flow: http.HTTPFlow) -> None:
request_path, _, query = flow.request.path.partition("?") request_path, _, query = flow.request.path.partition("?")
@@ -239,14 +253,14 @@ class EgressAddon:
self._serve_introspection(flow, request_path) self._serve_introspection(flow, request_path)
return return
config = self._active_config(flow) config, slug = self._resolve_flow(flow)
# DLP outbound scan BEFORE stripping auth — catches tokens the # DLP outbound scan BEFORE stripping auth — catches tokens the
# agent tried to smuggle in any header, path, query param, or body. # agent tried to smuggle in any header, path, query param, or body.
# Hostname is included to catch DNS-tunnelling exfiltration attempts. # Hostname is included to catch DNS-tunnelling exfiltration attempts.
route = match_route(config.routes, flow.request.pretty_host) route = match_route(config.routes, flow.request.pretty_host)
if route is not None: if route is not None:
if not await self._handle_outbound_dlp(flow, route): if not await self._handle_outbound_dlp(flow, route, slug):
return return
# The redact policy may have rewritten the request line; recompute # The redact policy may have rewritten the request line; recompute
# the path/query the git checks below rely on. # the path/query the git checks below rely on.
@@ -310,6 +324,7 @@ class EgressAddon:
self, self,
flow: http.HTTPFlow, flow: http.HTTPFlow,
route: Route, route: Route,
slug: str,
) -> bool: ) -> bool:
"""Scan the outbound request and apply the route's on-match policy """Scan the outbound request and apply the route's on-match policy
(PRD 0062). Returns True if the request may be forwarded, False if a (PRD 0062). Returns True if the request may be forwarded, False if a
@@ -331,7 +346,7 @@ class EgressAddon:
) )
result = scan_outbound( result = scan_outbound(
route, scan_text, os.environ, route, scan_text, os.environ,
safe_tokens=self.safe_tokens, crlf_text=crlf_text, safe_tokens=self._safe_tokens_for(slug), crlf_text=crlf_text,
) )
if result is None or result.severity != "block": if result is None or result.severity != "block":
return True return True
@@ -366,10 +381,10 @@ class EgressAddon:
# supervise (default): hold the request for operator approval. # supervise (default): hold the request for operator approval.
# Fall back to a hard 403 when supervise isn't wired for the bottle. # Fall back to a hard 403 when supervise isn't wired for the bottle.
if not self._supervise_available(): if not self._supervise_available(slug):
self._block_dlp(flow, result) self._block_dlp(flow, result)
return False return False
approved = await self._supervise_token_block(flow, request_path, result) approved = await self._supervise_token_block(flow, request_path, result, slug)
if not approved: if not approved:
return False # _supervise_token_block wrote the 403 response return False # _supervise_token_block wrote the 403 response
# loop: the approved value is now in safe_tokens; re-scan. # loop: the approved value is now in safe_tokens; re-scan.
@@ -412,12 +427,16 @@ class EgressAddon:
flow: http.HTTPFlow, flow: http.HTTPFlow,
request_path: str, request_path: str,
result: ScanResult, result: ScanResult,
slug: str,
) -> bool: ) -> bool:
"""Route a token DLP block to the operator's supervisor queue and wait. """Route a token DLP block to the operator's supervisor queue and wait.
Returns True if the operator approved (the matched value is added to `slug` attributes the proposal to the calling bottle (its own queue +
`self.safe_tokens` and the caller re-scans); False if the request must safelist) — in the shared gateway this is what keeps one bottle's
be blocked (a 403 response has been written to `flow`).""" approval from unblocking another's request. Returns True if the operator
approved (the matched value is added to that bottle's safelist and the
caller re-scans); False if the request must be blocked (a 403 response
has been written to `flow`)."""
host = flow.request.pretty_host host = flow.request.pretty_host
payload = build_token_allow_payload( payload = build_token_allow_payload(
redact_tokens(host, env=os.environ), redact_tokens(host, env=os.environ),
@@ -426,7 +445,7 @@ class EgressAddon:
result, result,
) )
proposal = _sv.Proposal.new( proposal = _sv.Proposal.new(
bottle_slug=self._supervise_slug, bottle_slug=slug,
tool=_sv.TOOL_EGRESS_TOKEN_ALLOW, tool=_sv.TOOL_EGRESS_TOKEN_ALLOW,
proposed_file=payload, proposed_file=payload,
justification=_TOKEN_ALLOW_JUSTIFICATION, justification=_TOKEN_ALLOW_JUSTIFICATION,
@@ -449,13 +468,13 @@ class EgressAddon:
**self._req_ctx(flow), **self._req_ctx(flow),
}) + "\n") }) + "\n")
response = await self._await_token_response(proposal.id) response = await self._await_token_response(proposal.id, slug)
_sv.archive_proposal(self._supervise_slug, proposal.id) _sv.archive_proposal(slug, proposal.id)
if response is not None and response.status in ( if response is not None and response.status in (
_sv.STATUS_APPROVED, _sv.STATUS_MODIFIED, _sv.STATUS_APPROVED, _sv.STATUS_MODIFIED,
): ):
self.safe_tokens.add(result.matched) self._safe_tokens_for(slug).add(result.matched)
if self.config.log >= LOG_BLOCKS: if self.config.log >= LOG_BLOCKS:
sys.stderr.write(json.dumps({ sys.stderr.write(json.dumps({
"event": "egress_token_allowed", "event": "egress_token_allowed",
@@ -478,6 +497,7 @@ class EgressAddon:
async def _await_token_response( async def _await_token_response(
self, self,
proposal_id: str, proposal_id: str,
slug: str,
) -> "_sv.Response | None": ) -> "_sv.Response | None":
"""Poll the DB for the operator's response without blocking the """Poll the DB for the operator's response without blocking the
proxy event loop. Returns the Response, or None on timeout.""" proxy event loop. Returns the Response, or None on timeout."""
@@ -485,7 +505,7 @@ class EgressAddon:
deadline = loop.time() + self._token_allow_timeout deadline = loop.time() + self._token_allow_timeout
while True: while True:
try: try:
return _sv.read_response(self._supervise_slug, proposal_id) return _sv.read_response(slug, proposal_id)
except (OSError, ValueError, KeyError): except (OSError, ValueError, KeyError):
# Not written yet, or a partial/malformed write — retry until # Not written yet, or a partial/malformed write — retry until
# the deadline, then fail closed. # the deadline, then fail closed.
@@ -539,6 +559,9 @@ class EgressAddon:
""" """
if flow.websocket is None: # type: ignore[union-attr] if flow.websocket is None: # type: ignore[union-attr]
return return
# WebSocket DLP runs against the static config only (single-tenant); in
# the consolidated gateway self.config has no routes, so this is inert
# until websocket routing is made source-IP-aware (a separate slice).
route = match_route(self.config.routes, flow.request.pretty_host) route = match_route(self.config.routes, flow.request.pretty_host)
if route is None: if route is None:
return return
@@ -549,7 +572,7 @@ class EgressAddon:
# not an injection vector here — scan only for credential leakage. # not an injection vector here — scan only for credential leakage.
result = scan_outbound( result = scan_outbound(
route, content, os.environ, route, content, os.environ,
safe_tokens=self.safe_tokens, crlf_text="", safe_tokens=self._safe_tokens_for(self._supervise_slug), crlf_text="",
) )
if result is not None and result.severity == "block": if result is not None and result.severity == "block":
sys.stderr.write(f"egress DLP: {result.reason}\n") sys.stderr.write(f"egress DLP: {result.reason}\n")
+40 -5
View File
@@ -421,6 +421,18 @@ class PolicyResolverLike(typing.Protocol):
... ...
def _config_from_policy(policy: "str | None") -> "Config":
"""Parse a resolved policy blob into a Config, fail-closed: None / empty /
unparseable all become a deny-all Config (no routes → every request
blocked)."""
if not policy:
return Config(routes=()) # unattributed or empty → deny-all
try:
return load_config(policy)
except ValueError:
return Config(routes=()) # unparseable policy → deny
def resolve_client_config( def resolve_client_config(
resolver: PolicyResolverLike, client_ip: str, identity_token: str = "" resolver: PolicyResolverLike, client_ip: str, identity_token: str = ""
) -> "Config": ) -> "Config":
@@ -433,12 +445,33 @@ def resolve_client_config(
policy = resolver.resolve(client_ip, identity_token) policy = resolver.resolve(client_ip, identity_token)
except Exception: # noqa: BLE001 # pylint: disable=broad-exception-caught except Exception: # noqa: BLE001 # pylint: disable=broad-exception-caught
return Config(routes=()) # orchestrator unreachable/errored → deny return Config(routes=()) # orchestrator unreachable/errored → deny
if not policy: return _config_from_policy(policy)
return Config(routes=()) # unattributed or empty → deny-all
class ContextResolverLike(typing.Protocol):
"""The bit of `policy_resolver.PolicyResolver` `resolve_client_context`
needs — one round-trip returning both policy and bottle id."""
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = ...,
) -> "tuple[str | None, str | None]":
...
def resolve_client_context(
resolver: ContextResolverLike, client_ip: str, identity_token: str = "",
) -> "tuple[Config, str]":
"""The calling client's `(Config, bottle_id)` in one round-trip —
**fail-closed**. The Config follows `resolve_client_config`'s deny-all
rules; the bottle id is `""` whenever unattributed or the orchestrator
errored, which the caller treats as "supervise unavailable for this
bottle" (never another bottle's queue). One `/resolve` keys both the
per-request egress policy and the per-bottle supervise queue + safelist."""
try: try:
return load_config(policy) policy, bottle_id = resolver.resolve_policy_and_bottle_id(client_ip, identity_token)
except ValueError: except Exception: # noqa: BLE001 # pylint: disable=broad-exception-caught
return Config(routes=()) # unparseable policy → deny return Config(routes=()), "" # orchestrator unreachable/errored → deny
return _config_from_policy(policy), (bottle_id or "")
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -833,7 +866,9 @@ __all__ = [
"is_git_fetch_request", "is_git_fetch_request",
"load_config", "load_config",
"resolve_client_config", "resolve_client_config",
"resolve_client_context",
"PolicyResolverLike", "PolicyResolverLike",
"ContextResolverLike",
"match_route", "match_route",
"outbound_scan_headers", "outbound_scan_headers",
"parse_config", "parse_config",
+2
View File
@@ -46,6 +46,7 @@ from .git_gate_render import (
git_gate_known_hosts_line, git_gate_known_hosts_line,
git_gate_render_access_hook, git_gate_render_access_hook,
git_gate_render_entrypoint, git_gate_render_entrypoint,
git_gate_render_provision,
git_gate_render_gitconfig, git_gate_render_gitconfig,
git_gate_render_hook, git_gate_render_hook,
git_gate_upstreams_for_bottle, git_gate_upstreams_for_bottle,
@@ -155,6 +156,7 @@ __all__ = [
"git_gate_render_gitconfig", "git_gate_render_gitconfig",
"git_gate_known_hosts_line", "git_gate_known_hosts_line",
"git_gate_render_entrypoint", "git_gate_render_entrypoint",
"git_gate_render_provision",
"git_gate_render_hook", "git_gate_render_hook",
"git_gate_render_access_hook", "git_gate_render_access_hook",
"provision_git_gate_dynamic_keys", "provision_git_gate_dynamic_keys",
+57 -21
View File
@@ -9,6 +9,7 @@ own; `git_gate` re-exports these names for API stability."""
from __future__ import annotations from __future__ import annotations
import re
import shlex import shlex
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
@@ -125,22 +126,18 @@ def git_gate_known_hosts_line(host: str, port: str, key: str) -> str:
return f"{target} {key}\n" return f"{target} {key}\n"
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str: def _git_gate_init_repo_fn(repo_root: str, creds_dir: str) -> list[str]:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then """The `init_repo` shell function, parameterized by the bare-repo root
`exec git daemon`. The function reads and the per-bottle creds dir. Single source of the credential-wiring
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into logic, shared by the single-tenant daemon entrypoint (`/git`,
the bundle by the renderer) and wires them into each bare repo's `/git-gate/creds`) and the consolidated per-bottle provisioning
config; the access-hook + pre-receive hook pick those paths up (`/git/<bottle_id>`, `/git-gate/creds/<bottle_id>`)."""
at fetch / push time.""" return [
lines = [
"#!/bin/sh",
"set -eu",
"",
"init_repo() {", "init_repo() {",
" name=$1", " name=$1",
" upstream_url=$2", " upstream_url=$2",
" keyfile=/git-gate/creds/${name}-key", f" keyfile={creds_dir}/${{name}}-key",
" hostsfile=/git-gate/creds/${name}-known_hosts", f" hostsfile={creds_dir}/${{name}}-known_hosts",
"", "",
# `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the # `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the
# host, so chmod-syscalls fail with EROFS. The files already # host, so chmod-syscalls fail with EROFS. The files already
@@ -153,14 +150,14 @@ def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
" chmod 600 \"$hostsfile\" 2>/dev/null || true", " chmod 600 \"$hostsfile\" 2>/dev/null || true",
" fi", " fi",
"", "",
" repo=/git/${name}.git", f" repo={repo_root}/${{name}}.git",
" if [ ! -d \"$repo\" ]; then", " if [ ! -d \"$repo\" ]; then",
" git init --bare \"$repo\" >/dev/null", " git init --bare \"$repo\" >/dev/null",
# --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so", # --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so a later
# a later `git fetch origin` mirrors the upstream's full ref", # `git fetch origin` mirrors the upstream's full ref graph (heads,
# graph (heads, tags, notes) into the bare repo at canonical", # tags, notes) into the bare repo at canonical paths. It does NOT set
# paths. It does NOT set remote.origin.mirror=true, so an", # remote.origin.mirror=true, so an explicit `git push origin
# explicit `git push origin <ref>:<ref>` still pushes one ref.", # <ref>:<ref>` still pushes one ref.
" git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"", " git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"",
" fi", " fi",
" git -C \"$repo\" config git-gate.identityFile \"$keyfile\"", " git -C \"$repo\" config git-gate.identityFile \"$keyfile\"",
@@ -170,9 +167,19 @@ def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
" git -C \"$repo\" config http.receivepack true", " git -C \"$repo\" config http.receivepack true",
" install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"", " install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"",
"}", "}",
"",
"mkdir -p /git",
] ]
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then
`exec git daemon`. The function reads
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into
the bundle by the renderer) and wires them into each bare repo's
config; the access-hook + pre-receive hook pick those paths up
at fetch / push time."""
lines = ["#!/bin/sh", "set -eu", ""]
lines += _git_gate_init_repo_fn("/git", "/git-gate/creds")
lines += ["", "mkdir -p /git"]
for u in upstreams: for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}") lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
lines.extend([ lines.extend([
@@ -190,6 +197,35 @@ def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
return "\n".join(lines) + "\n" return "\n".join(lines) + "\n"
# A bottle id namespaces the consolidated gateway's repo + creds dirs; it is
# embedded unquoted in the provisioning script, so restrict it to a shell- and
# path-safe alphabet (registry ids are token_hex — this is defense in depth).
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
def git_gate_render_provision(
bottle_id: str, upstreams: tuple[GitGateUpstream, ...],
) -> str:
"""Posix-sh script that provisions ONE bottle's bare repos into the
consolidated gateway (PRD 0070), under `/git/<bottle_id>/` with creds
read from `/git-gate/creds/<bottle_id>/`. Init-only — no `git daemon`,
since the shared gateway already serves every bottle; run inside the
running gateway when the bottle is registered.
Isolating each bottle's repo root and creds dir by id is what keeps one
bottle's push credentials out of another's repos on the shared gateway."""
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise ValueError(f"git-gate: unsafe bottle id {bottle_id!r}")
repo_root = f"/git/{bottle_id}"
creds_dir = f"/git-gate/creds/{bottle_id}"
lines = ["#!/bin/sh", "set -eu", ""]
lines += _git_gate_init_repo_fn(repo_root, creds_dir)
lines += ["", f"mkdir -p {shlex.quote(repo_root)}"]
for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
return "\n".join(lines) + "\n"
def git_gate_render_hook() -> str: def git_gate_render_hook() -> str:
"""The shared pre-receive hook: gitleaks-scan all incoming refs, """The shared pre-receive hook: gitleaks-scan all incoming refs,
then forward each accepted ref to the real upstream (`origin`) then forward each accepted ref to the real upstream (`origin`)
+8
View File
@@ -188,6 +188,14 @@ class GitHttpHandler(BaseHTTPRequestHandler):
"SERVER_PORT": str(self.server.server_port), # type: ignore "SERVER_PORT": str(self.server.server_port), # type: ignore
"SERVER_PROTOCOL": self.request_version, "SERVER_PROTOCOL": self.request_version,
}) })
# Consolidated mode: attribute the gitleaks-allow supervise proposal
# (written by receive-pack's pre-receive hook, a child of the CGI we
# spawn below) to the calling bottle. The namespaced root is
# `<base>/<bottle_id>`, so its final component is the bottle id — the
# same per-bottle key egress uses. Single-tenant leaves the hook's
# container-stamped SUPERVISE_BOTTLE_SLUG untouched.
if getattr(self.server, "policy_resolver", None) is not None:
env["SUPERVISE_BOTTLE_SLUG"] = sandbox_root.name
for header, variable in ( for header, variable in (
("accept", "HTTP_ACCEPT"), ("accept", "HTTP_ACCEPT"),
("content-encoding", "HTTP_CONTENT_ENCODING"), ("content-encoding", "HTTP_CONTENT_ENCODING"),
+141
View File
@@ -0,0 +1,141 @@
"""Orchestrator process lifecycle (PRD 0070, docker slice).
Before the CLI can register or launch bottles against the consolidated
model, exactly one orchestrator control plane — and the single per-host
gateway it manages — must be running. This starts the orchestrator
dev-harness (`python -m bot_bottle.orchestrator`) as a background host
process and health-checks it.
It is an **idempotent singleton**: `ensure_running` returns immediately if a
healthy control plane already answers on the port, and otherwise spawns one
and waits for it to come up. The control-plane port is the singleton key —
a second orchestrator can't bind it, so a stray double-start fails fast
rather than forking a rival.
Host-process (not container) on purpose: the PRD sequences the orchestrator
as a plain-process dev-harness first (fast iteration, and it already has the
host user's docker access to broker launches), while the data-plane
*gateway* it manages runs as a container. Wrapping the orchestrator itself
in a backend-native unit is a later step.
"""
from __future__ import annotations
import subprocess
import sys
import time
import urllib.error
import urllib.request
from .. import log
from ..paths import bot_bottle_root
DEFAULT_HOST = "127.0.0.1"
DEFAULT_PORT = 8080
# Poll cadence + default ceiling while waiting for a freshly-spawned control
# plane to answer /health (the first start also builds/boots the gateway).
_HEALTH_POLL_SECONDS = 0.25
DEFAULT_STARTUP_TIMEOUT_SECONDS = 45.0
_HEALTH_REQUEST_TIMEOUT_SECONDS = 1.0
class OrchestratorStartError(RuntimeError):
"""The orchestrator process did not become healthy within the timeout."""
class OrchestratorProcess:
"""Manages the local orchestrator control-plane process for the docker
backend. Backend-neutral callers only need `ensure_running()` + `url`."""
def __init__(
self,
host: str = DEFAULT_HOST,
port: int = DEFAULT_PORT,
*,
broker: str = "docker",
gateway: bool = True,
) -> None:
self.host = host
self.port = port
self._broker = broker
self._gateway = gateway
@property
def url(self) -> str:
"""The control-plane base URL — also what the data plane's
BOT_BOTTLE_ORCHESTRATOR_URL points at."""
return f"http://{self.host}:{self.port}"
def is_healthy(self, *, timeout: float = _HEALTH_REQUEST_TIMEOUT_SECONDS) -> bool:
"""True iff a control plane answers `GET /health` with 200 — the
singleton liveness check."""
try:
with urllib.request.urlopen(f"{self.url}/health", timeout=timeout) as resp:
return resp.status == 200
except (urllib.error.URLError, TimeoutError, OSError):
return False
def ensure_running(
self, *, startup_timeout: float = DEFAULT_STARTUP_TIMEOUT_SECONDS,
) -> str:
"""Return the control-plane URL, starting the orchestrator first if it
isn't already healthy. Idempotent — a healthy control plane is left
untouched. Raises `OrchestratorStartError` if a freshly-spawned one
doesn't answer within `startup_timeout`."""
if self.is_healthy():
return self.url
log.info("starting orchestrator", context={"url": self.url, "broker": self._broker})
self._spawn()
deadline = time.monotonic() + startup_timeout
while time.monotonic() < deadline:
if self.is_healthy():
log.info("orchestrator healthy", context={"url": self.url})
return self.url
time.sleep(_HEALTH_POLL_SECONDS)
raise OrchestratorStartError(
f"orchestrator at {self.url} did not become healthy within "
f"{startup_timeout:g}s"
)
def _argv(self) -> list[str]:
"""`python -m bot_bottle.orchestrator ...` — static flags only."""
argv = [
sys.executable, "-m", "bot_bottle.orchestrator",
"--host", self.host, "--port", str(self.port),
"--broker", self._broker,
]
if self._gateway:
argv.append("--gateway")
return argv
def _log_path(self) -> str:
"""Where the detached orchestrator's stdout/stderr goes so a failed
start is diagnosable after the CLI has moved on."""
return str(bot_bottle_root() / "orchestrator.log")
def _spawn(self) -> None:
"""Launch the orchestrator detached so it outlives this CLI process,
with its output tee'd to a log file under the bot-bottle root."""
root = bot_bottle_root()
root.mkdir(parents=True, exist_ok=True)
logfile = open(self._log_path(), "a", encoding="utf-8") # noqa: SIM115 # pylint: disable=consider-using-with
try:
subprocess.Popen( # noqa: S603 # pylint: disable=consider-using-with
self._argv(),
stdout=logfile,
stderr=subprocess.STDOUT,
stdin=subprocess.DEVNULL,
start_new_session=True,
)
finally:
# The child inherits its own dup'd fd; this handle is ours to drop.
logfile.close()
__all__ = [
"OrchestratorProcess",
"OrchestratorStartError",
"DEFAULT_HOST",
"DEFAULT_PORT",
"DEFAULT_STARTUP_TIMEOUT_SECONDS",
]
+57
View File
@@ -0,0 +1,57 @@
"""Consolidated registration inputs (PRD 0070, docker slice).
Bridges the existing per-bottle `prepare` output to the consolidated
registry: turns a prepared bottle's egress plan into the backend-neutral
inputs `Orchestrator.launch_bottle` takes — the egress **policy** blob and
launch **metadata**.
The policy blob is the exact routes YAML the per-bottle egress sidecar used
to read from a file; in the consolidated model the multi-tenant gateway's
`PolicyResolver` fetches it from the registry per request (keyed by source
IP) instead. Same render, so consolidated and single-tenant egress apply
byte-identical policy — a bottle's allow-list doesn't change when it moves
onto the shared gateway.
Host-side glue (imports `bot_bottle.egress`), used by the launch path — not
by the lean orchestrator control-plane process itself.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from ..egress import EgressPlan, egress_render_routes
@dataclass(frozen=True)
class RegistrationInputs:
"""What `Orchestrator.launch_bottle` needs to register a bottle, derived
from its prepared plan. `policy` is served verbatim by the gateway's
`/resolve`; `metadata` is opaque forward-compat state — it carries the
human slug so the console / supervise can show a name, not just the
minted bottle id."""
policy: str
metadata: str
def egress_policy(plan: EgressPlan) -> str:
"""The bottle's egress policy blob: the routes YAML the gateway serves
and the addon parses with `load_config`. Identical to the per-bottle
`routes.yaml` render, so the consolidated path applies the same
allow-list."""
return egress_render_routes(plan.routes, log=plan.log)
def registration_inputs(plan: EgressPlan) -> RegistrationInputs:
"""Assemble the orchestrator registration inputs from a prepared egress
plan. `metadata` records the slug so the shared registry can map a minted
bottle id back to its human name."""
return RegistrationInputs(
policy=egress_policy(plan),
metadata=json.dumps({"slug": plan.slug}),
)
__all__ = ["RegistrationInputs", "egress_policy", "registration_inputs"]
+19
View File
@@ -100,5 +100,24 @@ class PolicyResolver:
bottle_id = payload.get("bottle_id") bottle_id = payload.get("bottle_id")
return bottle_id if isinstance(bottle_id, str) and bottle_id else None return bottle_id if isinstance(bottle_id, str) and bottle_id else None
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = "",
) -> tuple[str | None, str | None]:
"""Both the policy blob and the bottle id in a single `/resolve` — so
a caller that needs each (the egress addon: policy for routing, bottle
id to key the calling bottle's supervise queue + safelist) makes one
round-trip, not two. Returns `(None, None)` when unattributed (a clean
`403`). Raises `PolicyResolveError` if the orchestrator can't be
reached, so the caller still fails closed."""
payload = self._post_resolve(source_ip, identity_token)
if payload is None:
return None, None
policy = payload.get("policy")
bottle_id = payload.get("bottle_id")
return (
policy if isinstance(policy, str) else "",
bottle_id if isinstance(bottle_id, str) and bottle_id else None,
)
__all__ = ["PolicyResolver", "PolicyResolveError"] __all__ = ["PolicyResolver", "PolicyResolveError"]
+50 -5
View File
@@ -15,6 +15,12 @@ The bottle slug arrives via SUPERVISE_BOTTLE_SLUG env (stamped at
container creation by the backend's start step). SUPERVISE_DB_PATH container creation by the backend's start step). SUPERVISE_DB_PATH
points at the bind-mounted host database. points at the bind-mounted host database.
Consolidated (PRD 0070): when BOT_BOTTLE_ORCHESTRATOR_URL is set, one
shared server fronts every bottle and attributes each proposal to the
calling bottle by source IP (resolved from the orchestrator) instead of a
fixed slug an unattributed source fails closed. Unset the legacy
per-bottle single-tenant server, unchanged.
Speaks MCP over HTTP+JSON-RPC. Methods handled: Speaks MCP over HTTP+JSON-RPC. Methods handled:
* `initialize` handshake; returns server info + caps. * `initialize` handshake; returns server info + caps.
@@ -40,16 +46,18 @@ import time
import typing import typing
import urllib.error import urllib.error
import urllib.request import urllib.request
from dataclasses import dataclass from dataclasses import dataclass, replace
try: try:
# Same-directory imports inside the bundle container; these files are # Same-directory imports inside the bundle container; these files are
# COPYed flat under /app by Dockerfile.sidecars. # COPYed flat under /app by Dockerfile.sidecars.
from egress_addon_core import LOG_OFF, load_config from egress_addon_core import LOG_OFF, load_config
from policy_resolver import PolicyResolveError, PolicyResolver
import supervise as _sv import supervise as _sv
except ModuleNotFoundError: except ModuleNotFoundError:
# Package imports for host-side tests and tooling. # Package imports for host-side tests and tooling.
from .egress_addon_core import LOG_OFF, load_config from .egress_addon_core import LOG_OFF, load_config
from .policy_resolver import PolicyResolveError, PolicyResolver
from . import supervise as _sv from . import supervise as _sv
@@ -73,6 +81,12 @@ DEFAULT_RESPONSE_TIMEOUT_SECONDS = 30.0
MIN_RESPONSE_POLL_INTERVAL_SECONDS = 0.05 MIN_RESPONSE_POLL_INTERVAL_SECONDS = 0.05
EGRESS_LIST_TIMEOUT_SECONDS = 5.0 EGRESS_LIST_TIMEOUT_SECONDS = 5.0
# Consolidated (multi-tenant) mode: when set, one shared supervise server
# fronts every bottle and attributes each proposal to the calling bottle by
# source IP (resolved from the orchestrator), instead of a single
# SUPERVISE_BOTTLE_SLUG env. Unset → legacy per-bottle single-tenant.
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
@dataclass(frozen=True) @dataclass(frozen=True)
class JsonRpcRequest: class JsonRpcRequest:
@@ -511,9 +525,30 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
if method == "tools/list": if method == "tools/list":
return handle_tools_list(req.params) return handle_tools_list(req.params)
if method == "tools/call": if method == "tools/call":
return handle_tools_call(req.params, config) # Attribute the proposal to the calling bottle. Single-tenant → the
# env slug on `config`; consolidated → the source-IP-resolved
# bottle id, so one shared server queues each bottle's proposal
# under its own slug.
return handle_tools_call(req.params, self._attributed_config(config))
raise _RpcClientError(ERR_METHOD_NOT_FOUND, f"method not found: {method}") raise _RpcClientError(ERR_METHOD_NOT_FOUND, f"method not found: {method}")
def _attributed_config(self, config: ServerConfig) -> ServerConfig:
"""The ServerConfig with `bottle_slug` bound to *this request's* bottle.
Single-tenant (no resolver): unchanged. Consolidated: the bottle id
attributed from the source IP **fail-closed**, an unattributed or
unreachable source raises so no proposal is queued under the wrong (or
empty) slug."""
resolver = getattr(self.server, "policy_resolver", None)
if resolver is None:
return config
try:
bottle_id = resolver.resolve_bottle_id(self.client_address[0])
except PolicyResolveError as e:
raise _RpcInternalError(f"orchestrator unreachable, cannot attribute: {e}") from e
if not bottle_id:
raise _RpcInternalError("request source is not attributed to a bottle")
return replace(config, bottle_slug=bottle_id)
def _write_jsonrpc(self, body: bytes) -> None: def _write_jsonrpc(self, body: bytes) -> None:
self.send_response(200) self.send_response(200)
self.send_header("Content-Type", "application/json") self.send_header("Content-Type", "application/json")
@@ -537,6 +572,9 @@ class MCPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
allow_reuse_address = True allow_reuse_address = True
daemon_threads = True daemon_threads = True
config: ServerConfig = ServerConfig(bottle_slug="") config: ServerConfig = ServerConfig(bottle_slug="")
# None → single-tenant (proposals use config.bottle_slug); set → consolidated
# (each proposal attributed to the source-IP-resolved bottle).
policy_resolver: "PolicyResolver | None" = None
# --- Entry point ----------------------------------------------------------- # --- Entry point -----------------------------------------------------------
@@ -548,15 +586,17 @@ def serve(
port: int = _sv.SUPERVISE_PORT, port: int = _sv.SUPERVISE_PORT,
bind: str = "0.0.0.0", bind: str = "0.0.0.0",
response_timeout_seconds: float = DEFAULT_RESPONSE_TIMEOUT_SECONDS, response_timeout_seconds: float = DEFAULT_RESPONSE_TIMEOUT_SECONDS,
resolver: "PolicyResolver | None" = None,
) -> typing.NoReturn: ) -> typing.NoReturn:
server = MCPServer((bind, port), MCPHandler) server = MCPServer((bind, port), MCPHandler)
server.config = ServerConfig( server.config = ServerConfig(
bottle_slug=bottle_slug, bottle_slug=bottle_slug,
response_timeout_seconds=response_timeout_seconds, response_timeout_seconds=response_timeout_seconds,
) )
server.policy_resolver = resolver
mode = "multi-tenant" if resolver else f"slug={bottle_slug!r}"
sys.stderr.write( sys.stderr.write(
f"supervise listening on {bind}:{port}; " f"supervise listening on {bind}:{port}; {mode}; "
f"slug={bottle_slug!r}; "
f"tools: {', '.join(t['name'] for t in TOOL_DEFINITIONS)}\n" # type: ignore[arg-type] f"tools: {', '.join(t['name'] for t in TOOL_DEFINITIONS)}\n" # type: ignore[arg-type]
) )
sys.stderr.flush() sys.stderr.flush()
@@ -571,8 +611,12 @@ def serve(
def main(argv: list[str]) -> int: def main(argv: list[str]) -> int:
del argv # config is env-only, no CLI flags del argv # config is env-only, no CLI flags
orch_url = os.environ.get(ORCHESTRATOR_URL_ENV, "").strip()
resolver = PolicyResolver(orch_url) if orch_url else None
bottle_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "") bottle_slug = os.environ.get("SUPERVISE_BOTTLE_SLUG", "")
if not bottle_slug: # Consolidated mode resolves the slug per request, so the env slug is
# optional there; single-tenant still requires it.
if not bottle_slug and resolver is None:
sys.stderr.write("supervise: SUPERVISE_BOTTLE_SLUG env is unset\n") sys.stderr.write("supervise: SUPERVISE_BOTTLE_SLUG env is unset\n")
return 2 return 2
port = int(os.environ.get("SUPERVISE_PORT", str(_sv.SUPERVISE_PORT))) port = int(os.environ.get("SUPERVISE_PORT", str(_sv.SUPERVISE_PORT)))
@@ -587,6 +631,7 @@ def main(argv: list[str]) -> int:
port=port, port=port,
bind=bind, bind=bind,
response_timeout_seconds=response_timeout_seconds, response_timeout_seconds=response_timeout_seconds,
resolver=resolver,
) )
return 0 # serve() does not return return 0 # serve() does not return
@@ -46,7 +46,7 @@ def _addon() -> EgressAddon:
"""Return a bare EgressAddon with LOG_FULL config and no routes file.""" """Return a bare EgressAddon with LOG_FULL config and no routes file."""
a: EgressAddon = EgressAddon.__new__(EgressAddon) a: EgressAddon = EgressAddon.__new__(EgressAddon)
a.config = Config(routes=(), log=LOG_FULL) a.config = Config(routes=(), log=LOG_FULL)
a.safe_tokens = set() a._safe_tokens = {}
a._supervise_slug = "" a._supervise_slug = ""
a._token_allow_timeout = 300.0 a._token_allow_timeout = 300.0
return a return a
+83 -2
View File
@@ -211,7 +211,7 @@ def _addon(config: Config) -> EgressAddon:
"""Bare EgressAddon with a supplied config and no supervise wiring.""" """Bare EgressAddon with a supplied config and no supervise wiring."""
a: EgressAddon = EgressAddon.__new__(EgressAddon) a: EgressAddon = EgressAddon.__new__(EgressAddon)
a.config = config a.config = config
a.safe_tokens = set() a._safe_tokens = {}
a._supervise_slug = "" a._supervise_slug = ""
a._token_allow_timeout = 300.0 a._token_allow_timeout = 300.0
a.routes_path = "/nonexistent/routes.yaml" a.routes_path = "/nonexistent/routes.yaml"
@@ -222,6 +222,29 @@ def _run_request(addon: EgressAddon, flow: _Flow) -> None:
asyncio.run(addon.request(flow)) # type: ignore[arg-type] asyncio.run(addon.request(flow)) # type: ignore[arg-type]
def _with_client_ip(flow: _Flow, ip: str) -> _Flow:
"""Attach a mitmproxy-style client_conn so consolidated resolution can read
the source IP (`flow.client_conn.peername[0]`)."""
flow.client_conn = types.SimpleNamespace(peername=(ip, 54321)) # type: ignore[attr-defined]
return flow
class _CtxResolver:
"""Fake orchestrator resolver: maps source IP -> bottle id, and grants the
same allow-list to any attributed bottle (unattributed -> deny)."""
def __init__(self, ip_to_bottle: dict[str, str]) -> None:
self._map = ip_to_bottle
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = "",
) -> tuple[str | None, str | None]:
del identity_token
bottle_id = self._map.get(source_ip)
policy = "routes:\n - host: api.example.com\n" if bottle_id else None
return policy, bottle_id
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
# Introspection endpoint # Introspection endpoint
# --------------------------------------------------------------------------- # ---------------------------------------------------------------------------
@@ -418,7 +441,8 @@ class TestSuperviseBranch(unittest.TestCase):
with patch.object(_ea_mod, "_sv", _fake_sv("approved")): with patch.object(_ea_mod, "_sv", _fake_sv("approved")):
_run_request(addon, flow) _run_request(addon, flow)
self.assertIsNone(flow.response) # forwarded after approval self.assertIsNone(flow.response) # forwarded after approval
self.assertIn(_OPENAI_KEY, addon.safe_tokens) # Approval lands in the calling bottle's safelist (keyed by slug).
self.assertIn(_OPENAI_KEY, addon._safe_tokens_for("test-bottle"))
def test_operator_rejection_blocks(self) -> None: def test_operator_rejection_blocks(self) -> None:
addon = self._supervised_addon() addon = self._supervised_addon()
@@ -735,5 +759,62 @@ class TestLogFullRequest(unittest.TestCase):
self.assertTrue(any(e.get("event") == "egress_request" for e in logged)) self.assertTrue(any(e.get("event") == "egress_request" for e in logged))
class TestSuperviseMultiTenant(unittest.TestCase):
"""Consolidated gateway: supervise proposals + the DLP safelist are keyed
per bottle, resolved by source IP (PRD 0070)."""
def _consolidated_addon(self) -> EgressAddon:
# Static config is empty; the resolver supplies each bottle's config.
addon = _addon(Config(routes=()))
addon._resolver = cast(Any, _CtxResolver({"10.0.0.1": "bottle-a", "10.0.0.2": "bottle-b"}))
addon._token_allow_timeout = 0.05
return addon
def test_approval_is_scoped_to_the_calling_bottle(self) -> None:
addon = self._consolidated_addon()
# bottle-a (10.0.0.1) sends the token; the operator approves.
flow = _with_client_ip(
_Flow(_Request(host="api.example.com", method="POST", body=f"k={_OPENAI_KEY}")),
"10.0.0.1",
)
with patch.object(_ea_mod, "_sv", _fake_sv("approved")):
_run_request(addon, flow)
self.assertIsNone(flow.response) # forwarded after approval
# The approval lands ONLY in bottle-a's safelist — never bottle-b's.
# A global set here would be the cross-tenant leak this slice closes.
self.assertIn(_OPENAI_KEY, addon._safe_tokens_for("bottle-a"))
self.assertNotIn(_OPENAI_KEY, addon._safe_tokens_for("bottle-b"))
def test_proposal_is_attributed_to_the_source_ip_bottle(self) -> None:
addon = self._consolidated_addon()
seen: list[str] = []
fake = _fake_sv("approved")
def _capture(**kw: Any) -> Any:
seen.append(kw["bottle_slug"])
return types.SimpleNamespace(id="p")
fake.Proposal = types.SimpleNamespace(new=_capture)
flow = _with_client_ip(
_Flow(_Request(host="api.example.com", method="POST", body=f"k={_OPENAI_KEY}")),
"10.0.0.2",
)
with patch.object(_ea_mod, "_sv", fake):
_run_request(addon, flow)
self.assertEqual(["bottle-b"], seen) # proposal keyed by the resolved bottle
def test_unattributed_source_ip_cannot_supervise(self) -> None:
addon = self._consolidated_addon()
# 10.9.9.9 is not in the resolver map -> deny-all config, empty slug.
flow = _with_client_ip(
_Flow(_Request(host="api.example.com", method="POST", body=f"k={_OPENAI_KEY}")),
"10.9.9.9",
)
with patch.object(_ea_mod, "_sv", _fake_sv("approved")):
_run_request(addon, flow)
self.assertIsNotNone(flow.response) # blocked (no route, no supervise)
self.assertNotIn(_OPENAI_KEY, addon._safe_tokens_for(""))
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+49 -2
View File
@@ -1,10 +1,10 @@
"""Unit: resolve_client_config — fail-closed per-client egress config (PRD 0070).""" """Unit: fail-closed per-client egress resolution — config + context (PRD 0070)."""
from __future__ import annotations from __future__ import annotations
import unittest import unittest
from bot_bottle.egress_addon_core import resolve_client_config from bot_bottle.egress_addon_core import resolve_client_config, resolve_client_context
from bot_bottle.policy_resolver import PolicyResolveError from bot_bottle.policy_resolver import PolicyResolveError
@@ -49,5 +49,52 @@ class TestResolveClientConfig(unittest.TestCase):
self.assertEqual(("10.243.0.1", "tok"), r.calls[0]) self.assertEqual(("10.243.0.1", "tok"), r.calls[0])
class _FakeContextResolver:
def __init__(
self, policy: str | None = None, bottle_id: str | None = None, raises: bool = False,
) -> None:
self._policy = policy
self._bottle_id = bottle_id
self._raises = raises
def resolve_policy_and_bottle_id(
self, source_ip: str, identity_token: str = "",
) -> tuple[str | None, str | None]:
if self._raises:
raise PolicyResolveError("orchestrator down")
return self._policy, self._bottle_id
class TestResolveClientContext(unittest.TestCase):
def test_returns_config_and_bottle_id(self) -> None:
cfg, slug = resolve_client_context(
_FakeContextResolver(policy="routes:\n - host: example.com\n", bottle_id="b1"),
"10.243.0.1",
)
self.assertEqual(("example.com",), tuple(r.host for r in cfg.routes))
self.assertEqual("b1", slug)
def test_unattributed_denies_and_empty_slug(self) -> None:
cfg, slug = resolve_client_context(
_FakeContextResolver(policy=None, bottle_id=None), "10.243.0.9",
)
self.assertEqual((), cfg.routes)
self.assertEqual("", slug) # no bottle → supervise unavailable
def test_resolver_error_denies_and_empty_slug(self) -> None:
cfg, slug = resolve_client_context(_FakeContextResolver(raises=True), "10.243.0.1")
self.assertEqual((), cfg.routes)
self.assertEqual("", slug)
def test_unparseable_policy_denies_but_keeps_slug(self) -> None:
# A bad policy denies egress, but the bottle is still attributed (its
# supervise queue is keyed by the id, independent of route parsing).
cfg, slug = resolve_client_context(
_FakeContextResolver(policy="routes: notalist\n", bottle_id="b2"), "10.243.0.1",
)
self.assertEqual((), cfg.routes)
self.assertEqual("b2", slug)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
@@ -0,0 +1,67 @@
"""Unit: consolidated per-bottle git-gate provisioning render (PRD 0070)."""
from __future__ import annotations
import unittest
from bot_bottle.git_gate_render import (
GitGateUpstream,
git_gate_render_entrypoint,
git_gate_render_provision,
)
def _ups(*names: str) -> tuple[GitGateUpstream, ...]:
return tuple(
GitGateUpstream(
name=n,
upstream_url=f"ssh://git@github.com/x/{n}.git",
upstream_host="github.com",
upstream_port="22",
identity_file="",
known_host_key="",
)
for n in names
)
class TestProvisionRender(unittest.TestCase):
def test_namespaces_repos_and_creds_by_bottle_id(self) -> None:
script = git_gate_render_provision("bottleab12", _ups("foo"))
self.assertIn("repo=/git/bottleab12/${name}.git", script)
self.assertIn("keyfile=/git-gate/creds/bottleab12/${name}-key", script)
self.assertIn("mkdir -p /git/bottleab12", script)
def test_one_init_repo_call_per_upstream(self) -> None:
script = git_gate_render_provision("b1", _ups("foo", "bar"))
calls = [l for l in script.splitlines() if l.startswith("init_repo ")]
self.assertEqual(2, len(calls))
def test_provision_does_not_start_the_daemon(self) -> None:
# The shared gateway already serves; provisioning is init-only.
self.assertNotIn("git daemon", git_gate_render_provision("b1", _ups("foo")))
def test_installs_pre_receive_hook(self) -> None:
script = git_gate_render_provision("b1", _ups("foo"))
self.assertIn("install -m 755 /etc/git-gate/pre-receive", script)
def test_rejects_unsafe_bottle_id(self) -> None:
for bad in ("../etc", "a/b", "a b", "a;rm", ""):
with self.assertRaises(ValueError):
git_gate_render_provision(bad, _ups("foo"))
class TestEntrypointUnchanged(unittest.TestCase):
"""The shared `_git_gate_init_repo_fn` refactor must not alter the
single-tenant daemon entrypoint's output."""
def test_entrypoint_still_single_tenant_flat(self) -> None:
script = git_gate_render_entrypoint(_ups("foo"))
self.assertIn("repo=/git/${name}.git", script) # flat, not namespaced
self.assertIn("keyfile=/git-gate/creds/${name}-key", script)
self.assertIn("--base-path=/git", script)
self.assertIn("exec git daemon", script)
if __name__ == "__main__":
unittest.main()
+67
View File
@@ -13,6 +13,17 @@ from bot_bottle.git_gate import GIT_GATE_TIMEOUT_SECS
from bot_bottle.git_http_backend import GitHttpHandler, MAX_BODY_BYTES from bot_bottle.git_http_backend import GitHttpHandler, MAX_BODY_BYTES
class _FixedResolver:
"""Maps every source IP to one bottle id (consolidated-mode stub)."""
def __init__(self, bottle_id: str) -> None:
self._bottle_id = bottle_id
def resolve_bottle_id(self, source_ip: str, identity_token: str = "") -> str:
del source_ip, identity_token
return self._bottle_id
class TestGitHttpBackend(unittest.TestCase): class TestGitHttpBackend(unittest.TestCase):
def test_real_git_push_reaches_bare_repo(self): def test_real_git_push_reaches_bare_repo(self):
from http.server import ThreadingHTTPServer from http.server import ThreadingHTTPServer
@@ -94,6 +105,62 @@ class TestGitHttpBackend(unittest.TestCase):
).strip() ).strip()
self.assertEqual(head, cloned) self.assertEqual(head, cloned)
def test_consolidated_push_stamps_bottle_slug_for_the_hook(self):
# In consolidated mode the backend attributes the push by source IP and
# stamps SUPERVISE_BOTTLE_SLUG=<bottle_id> into the CGI env, so the
# gitleaks-allow pre-receive hook queues its proposal under the right
# bottle. The hook here just records what it received.
from http.server import ThreadingHTTPServer
bottle_id = "bottleab12"
with tempfile.TemporaryDirectory() as tmp:
root = Path(tmp)
bare = root / bottle_id / "repo.git" # namespaced per bottle (slice 10)
bare.parent.mkdir(parents=True)
subprocess.run(["git", "init", "--bare", str(bare)],
check=True, capture_output=True, text=True)
subprocess.run(
["git", "-C", str(bare), "config", "http.receivepack", "true"],
check=True,
)
capture = root / "slug-capture"
hook = bare / "hooks" / "pre-receive"
hook.write_text(
f"#!/bin/sh\nprintf '%s' \"${{SUPERVISE_BOTTLE_SLUG:-UNSET}}\" > "
f"{capture}\ncat >/dev/null\nexit 0\n"
)
hook.chmod(0o755)
old_root = os.environ.get("GIT_PROJECT_ROOT")
os.environ["GIT_PROJECT_ROOT"] = str(root) # base; backend nests per bottle
self.addCleanup(self._restore_env, old_root)
server = ThreadingHTTPServer(("127.0.0.1", 0), GitHttpHandler)
server.policy_resolver = _FixedResolver(bottle_id) # type: ignore[attr-defined]
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
self.addCleanup(server.shutdown)
self.addCleanup(server.server_close)
work = root / "work"
work.mkdir()
subprocess.run(["git", "init"], cwd=work, check=True,
capture_output=True, text=True)
subprocess.run(["git", "config", "user.name", "test"], cwd=work, check=True)
subprocess.run(["git", "config", "user.email", "t@example.invalid"],
cwd=work, check=True)
(work / "README.md").write_text("test\n")
subprocess.run(["git", "add", "README.md"], cwd=work, check=True)
subprocess.run(["git", "commit", "-m", "init"], cwd=work,
check=True, capture_output=True, text=True)
url = f"http://127.0.0.1:{server.server_port}/repo.git"
subprocess.run(
["git", "push", url, "HEAD:refs/heads/main"],
cwd=work, check=True, capture_output=True, text=True, timeout=5,
)
self.assertEqual(bottle_id, capture.read_text())
def test_post_forwards_git_cgi_headers(self): def test_post_forwards_git_cgi_headers(self):
from http.server import ThreadingHTTPServer from http.server import ThreadingHTTPServer
+87
View File
@@ -0,0 +1,87 @@
"""Unit: orchestrator process lifecycle — idempotent singleton (PRD 0070)."""
from __future__ import annotations
import tempfile
import unittest
import urllib.error
from pathlib import Path
from unittest.mock import MagicMock, patch
from bot_bottle.orchestrator.lifecycle import (
OrchestratorProcess,
OrchestratorStartError,
)
from tests.unit import use_bottle_root
_URLOPEN = "bot_bottle.orchestrator.lifecycle.urllib.request.urlopen"
_POPEN = "bot_bottle.orchestrator.lifecycle.subprocess.Popen"
_SLEEP = "bot_bottle.orchestrator.lifecycle.time.sleep"
_MONOTONIC = "bot_bottle.orchestrator.lifecycle.time.monotonic"
def _health(status: int) -> MagicMock:
"""A urlopen() context-manager whose `.status` is `status`."""
m = MagicMock()
m.__enter__.return_value.status = status
return m
class TestOrchestratorProcess(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.addCleanup(self._tmp.cleanup)
self.addCleanup(use_bottle_root(Path(self._tmp.name)))
self.p = OrchestratorProcess(port=8099)
def test_url(self) -> None:
self.assertEqual("http://127.0.0.1:8099", self.p.url)
def test_is_healthy_true_on_200(self) -> None:
with patch(_URLOPEN, return_value=_health(200)):
self.assertTrue(self.p.is_healthy())
def test_is_healthy_false_on_error(self) -> None:
with patch(_URLOPEN, side_effect=urllib.error.URLError("refused")):
self.assertFalse(self.p.is_healthy())
def test_ensure_running_noop_when_already_healthy(self) -> None:
with patch(_URLOPEN, return_value=_health(200)), patch(_POPEN) as popen:
self.assertEqual(self.p.url, self.p.ensure_running())
popen.assert_not_called() # a live control plane is left untouched
def test_ensure_running_spawns_then_waits_for_health(self) -> None:
# First check (before spawn) fails; after spawn the poll succeeds.
with patch(_URLOPEN, side_effect=[urllib.error.URLError("down"), _health(200)]), \
patch(_POPEN) as popen, patch(_SLEEP):
url = self.p.ensure_running()
self.assertEqual(self.p.url, url)
popen.assert_called_once()
def test_ensure_running_raises_on_startup_timeout(self) -> None:
with patch(_URLOPEN, side_effect=urllib.error.URLError("down")), \
patch(_POPEN), patch(_SLEEP), \
patch(_MONOTONIC, side_effect=[0.0, 0.5, 2.0]):
with self.assertRaises(OrchestratorStartError):
self.p.ensure_running(startup_timeout=1.0)
def test_argv_includes_gateway_and_broker(self) -> None:
argv = OrchestratorProcess(port=8099, broker="docker", gateway=True)._argv()
self.assertIn("--gateway", argv)
self.assertIn("bot_bottle.orchestrator", argv)
self.assertEqual("docker", argv[argv.index("--broker") + 1])
def test_argv_omits_gateway_when_disabled(self) -> None:
self.assertNotIn("--gateway", OrchestratorProcess(gateway=False)._argv())
def test_spawn_launches_detached_and_logs(self) -> None:
with patch(_POPEN) as popen:
self.p._spawn()
popen.assert_called_once()
kwargs = popen.call_args.kwargs
self.assertTrue(kwargs["start_new_session"]) # outlives the CLI
self.assertTrue((Path(self._tmp.name) / "orchestrator.log").exists())
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,56 @@
"""Unit: consolidated registration inputs — egress policy round-trip (PRD 0070)."""
from __future__ import annotations
import json
import unittest
from pathlib import Path
from bot_bottle.egress import EgressPlan, EgressRoute
from bot_bottle.egress_addon_core import LOG_BLOCKS, load_config
from bot_bottle.orchestrator.registration import (
RegistrationInputs,
egress_policy,
registration_inputs,
)
def _plan(routes: tuple[EgressRoute, ...], *, slug: str = "demo", log: int = 0) -> EgressPlan:
return EgressPlan(
slug=slug,
routes_path=Path("/unused/routes.yaml"),
routes=routes,
token_env_map={},
log=log,
)
class TestEgressPolicy(unittest.TestCase):
def test_policy_round_trips_through_load_config(self) -> None:
# The policy the gateway serves must parse back to the same allow-list
# the per-bottle sidecar applied — moving onto the shared gateway must
# not change a bottle's egress.
routes = (EgressRoute(host="api.example.com"), EgressRoute(host="pypi.org"))
cfg = load_config(egress_policy(_plan(routes)))
self.assertEqual(("api.example.com", "pypi.org"), tuple(r.host for r in cfg.routes))
def test_policy_preserves_log_level(self) -> None:
plan = _plan((EgressRoute(host="x.example.com"),), log=LOG_BLOCKS)
self.assertEqual(LOG_BLOCKS, load_config(egress_policy(plan)).log)
def test_empty_routes_yield_deny_all(self) -> None:
cfg = load_config(egress_policy(_plan(())))
self.assertEqual((), cfg.routes) # no routes → default-deny
class TestRegistrationInputs(unittest.TestCase):
def test_bundles_policy_and_slug_metadata(self) -> None:
plan = _plan((EgressRoute(host="api.example.com"),), slug="my-bot")
inputs = registration_inputs(plan)
self.assertIsInstance(inputs, RegistrationInputs)
self.assertEqual("my-bot", json.loads(inputs.metadata)["slug"])
self.assertEqual(egress_policy(plan), inputs.policy) # same blob egress_policy renders
if __name__ == "__main__":
unittest.main()
+14
View File
@@ -88,6 +88,20 @@ class TestPolicyResolver(unittest.TestCase):
with self.assertRaises(PolicyResolveError): with self.assertRaises(PolicyResolveError):
self.r.resolve_bottle_id("10.243.0.1", "tok") self.r.resolve_bottle_id("10.243.0.1", "tok")
def test_resolve_policy_and_bottle_id_one_call(self) -> None:
with patch(_URLOPEN, return_value=_resp({"bottle_id": "b1", "policy": "P"})) as m:
self.assertEqual(("P", "b1"), self.r.resolve_policy_and_bottle_id("10.243.0.1", "t"))
self.assertEqual(1, m.call_count) # both from a single /resolve
def test_resolve_policy_and_bottle_id_403_is_none_none(self) -> None:
with patch(_URLOPEN, side_effect=_http_error(403)):
self.assertEqual((None, None), self.r.resolve_policy_and_bottle_id("10.243.0.9"))
def test_resolve_policy_and_bottle_id_error_raises(self) -> None:
with patch(_URLOPEN, side_effect=urllib.error.URLError("refused")):
with self.assertRaises(PolicyResolveError):
self.r.resolve_policy_and_bottle_id("10.243.0.1")
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()
+53
View File
@@ -6,6 +6,7 @@ import sys
import tempfile import tempfile
import threading import threading
import time import time
import types
import unittest import unittest
from pathlib import Path from pathlib import Path
from unittest.mock import patch from unittest.mock import patch
@@ -629,5 +630,57 @@ class TestHttpEndToEnd(unittest.TestCase):
conn.close() conn.close()
class _FakeResolver:
def __init__(self, bottle_id: str | None = None, raises: bool = False) -> None:
self._bottle_id = bottle_id
self._raises = raises
self.calls: list[str] = []
def resolve_bottle_id(self, source_ip: str, identity_token: str = "") -> str | None:
del identity_token
self.calls.append(source_ip)
if self._raises:
# Raise the exact class supervise_server catches (it imports
# policy_resolver flat inside the bundle, package-side in tests).
raise supervise_server.PolicyResolveError("orchestrator down")
return self._bottle_id
def _handler(resolver: object) -> MCPHandler:
"""A bare MCPHandler wired with a server (carrying the resolver) and a
client address, enough to exercise `_attributed_config` off-socket."""
h: MCPHandler = MCPHandler.__new__(MCPHandler)
h.server = types.SimpleNamespace(policy_resolver=resolver) # type: ignore[assignment]
h.client_address = ("10.0.0.7", 4321)
return h
class TestAttributedConfig(unittest.TestCase):
"""Consolidated supervise: each proposal is attributed to the calling
bottle by source IP; single-tenant keeps the env slug (PRD 0070)."""
def test_single_tenant_keeps_env_slug(self) -> None:
cfg = _handler(None)._attributed_config(ServerConfig(bottle_slug="dev"))
self.assertEqual("dev", cfg.bottle_slug)
def test_consolidated_binds_source_ip_bottle(self) -> None:
r = _FakeResolver(bottle_id="bottle-x")
cfg = _handler(r)._attributed_config(ServerConfig(bottle_slug="ignored"))
self.assertEqual("bottle-x", cfg.bottle_slug) # resolved slug wins
self.assertEqual(["10.0.0.7"], r.calls)
def test_unattributed_source_fails_closed(self) -> None:
with self.assertRaises(_RpcInternalError):
_handler(_FakeResolver(bottle_id=None))._attributed_config(
ServerConfig(bottle_slug="x")
)
def test_resolver_error_fails_closed(self) -> None:
with self.assertRaises(_RpcInternalError):
_handler(_FakeResolver(raises=True))._attributed_config(
ServerConfig(bottle_slug="x")
)
if __name__ == "__main__": if __name__ == "__main__":
unittest.main() unittest.main()