refactor(gateway): split data-plane files into egress/supervisor/git_gate services
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 49s
lint / lint (push) Failing after 2m49s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 18s
test / publish-infra (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 17s
test / unit (pull_request) Successful in 49s
lint / lint (push) Failing after 2m49s
test / integration-firecracker (pull_request) Successful in 3m35s
test / coverage (pull_request) Successful in 18s
test / publish-infra (pull_request) Has been skipped
Group the gateway's data-plane modules into three service sub-packages mirroring the host-side trio (bot_bottle.egress / .supervisor / .git_gate): gateway/egress/ addon_core, addon, dlp_config, dlp_detectors gateway/supervisor/ server (was supervise_server) gateway/git_gate/ render, http_backend Prefix-stripped filenames now that the package namespaces them; each sub-package has a thin docstring __init__ (no eager imports, cheap leaf loads). The two cross-cutting files stay at the gateway root: policy_resolver (shared per-client lookup) and gateway_init, renamed to bootstrap now that gateway/ already namespaces it. Updated all importers (bot_bottle + tests), the in-VM/container `-m` launch strings, the Dockerfile.gateway addon shim + ENTRYPOINT, and the five gateway entries in scripts/critical-modules.txt. Full unit suite green (2243). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,7 @@
|
||||
"""Gateway-side (data-plane) git-gate service: pure host-side hook rendering
|
||||
(PRD 0008) and the smart-HTTP backend that fronts git-gate repos.
|
||||
|
||||
The host-side git-gate service (provisioning, preflight) lives in
|
||||
`bot_bottle.git_gate`; this is the data-plane counterpart. Import the concrete
|
||||
modules directly (`from bot_bottle.gateway.git_gate.render import ...`).
|
||||
"""
|
||||
@@ -0,0 +1,287 @@
|
||||
"""Tiny smart-HTTP wrapper for git-gate repos.
|
||||
|
||||
Used where `git://` push traffic over a host-published Docker port can
|
||||
hang before receive-pack reaches hooks (e.g. the firecracker backend,
|
||||
where the guest reaches the gateway over the point-to-point TAP). The
|
||||
wrapper serves the same `/git/*.git` bare repos through
|
||||
`git http-backend`, so pre-receive and upstream forwarding remain the
|
||||
git-gate enforcement point.
|
||||
|
||||
One shared gateway serves every bottle (PRD 0070): each request is served
|
||||
from the calling bottle's repo namespace (`<root>/<bottle_id>`), attributed
|
||||
from the unspoofable source IP via the orchestrator. Per-repo credentials +
|
||||
hooks scope by repo directory, so isolating the *root* per bottle isolates
|
||||
its creds too. Unattributed clients — and a missing/unreachable orchestrator
|
||||
— fail closed (404). `BOT_BOTTLE_ORCHESTRATOR_URL` is mandatory: there is no
|
||||
single-tenant flat-root fallback.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
import typing
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
from pathlib import Path
|
||||
from urllib.parse import urlsplit
|
||||
|
||||
from bot_bottle.constants import GIT_GATE_TIMEOUT_SECS, IDENTITY_HEADER
|
||||
from bot_bottle.gateway.policy_resolver import PolicyResolveError, PolicyResolver
|
||||
|
||||
|
||||
DEFAULT_PORT = 9420
|
||||
|
||||
# The per-host orchestrator control plane the backend attributes each request
|
||||
# to, serving from the *calling* bottle's repo namespace selected by source IP.
|
||||
# Mandatory — the same env the egress addon requires; there is no single flat
|
||||
# repo-root fallback.
|
||||
ORCHESTRATOR_URL_ENV = "BOT_BOTTLE_ORCHESTRATOR_URL"
|
||||
|
||||
# The base under which each bottle's `<bottle_id>` repo namespace is nested.
|
||||
DEFAULT_REPO_ROOT = "/git"
|
||||
|
||||
|
||||
class ResolverLike(typing.Protocol):
|
||||
"""Structural type for the resolver `resolve_sandbox_root` needs — just
|
||||
`resolve_bottle_id`. A Protocol so tests can pass a fake without
|
||||
importing PolicyResolver (mirrors egress_addon_core.PolicyResolverLike)."""
|
||||
|
||||
def resolve_bottle_id(
|
||||
self, source_ip: str, identity_token: str = ...,
|
||||
) -> "str | None": ...
|
||||
|
||||
|
||||
def resolve_sandbox_root(
|
||||
resolver: "ResolverLike",
|
||||
base_root: Path,
|
||||
source_ip: str,
|
||||
identity_token: str = "",
|
||||
) -> Path | None:
|
||||
"""The per-sandbox repo root to serve this request from — `base_root/
|
||||
<bottle_id>`, where the sandbox is attributed from the source IP via the
|
||||
orchestrator — or None to deny (404). Fail-closed: an unattributed client, a
|
||||
resolver error, or a namespace that would escape `base_root` all deny, so one
|
||||
sandbox can never reach another's repos."""
|
||||
try:
|
||||
bottle_id = resolver.resolve_bottle_id(source_ip, identity_token)
|
||||
except PolicyResolveError:
|
||||
return None # orchestrator unreachable/errored → deny
|
||||
if not bottle_id:
|
||||
return None # unattributed → deny
|
||||
base = base_root.resolve()
|
||||
namespace = (base / bottle_id).resolve()
|
||||
if base not in namespace.parents:
|
||||
return None # bottle_id tried to escape the root → deny
|
||||
return namespace
|
||||
|
||||
# Bound memory use while still allowing ordinary git push packfiles.
|
||||
MAX_BODY_BYTES = 100 * 1024 * 1024
|
||||
|
||||
|
||||
class GitHttpHandler(BaseHTTPRequestHandler):
|
||||
server_version = "bot-bottle-git-http/1"
|
||||
|
||||
def do_GET(self) -> None:
|
||||
self._run_backend()
|
||||
|
||||
def do_POST(self) -> None:
|
||||
self._run_backend()
|
||||
|
||||
def _sandbox_root(self) -> Path | None:
|
||||
"""This request's per-sandbox repo root (the calling bottle's source-IP-
|
||||
selected `<base>/<bottle_id>` namespace), or None to deny. `GIT_PROJECT_
|
||||
ROOT` keeps git's own env-var name."""
|
||||
base = Path(os.environ.get("GIT_PROJECT_ROOT", DEFAULT_REPO_ROOT))
|
||||
resolver = getattr(self.server, "policy_resolver", None)
|
||||
if resolver is None:
|
||||
return None # server started without a resolver (misconfig) → deny
|
||||
token = self.headers.get(IDENTITY_HEADER, "")
|
||||
return resolve_sandbox_root(resolver, base, self.client_address[0], token)
|
||||
|
||||
def _run_backend(self) -> None:
|
||||
sandbox_root = self._sandbox_root()
|
||||
if sandbox_root is None:
|
||||
# Unattributed / resolver error: deny before touching any repo.
|
||||
self.send_error(404)
|
||||
return
|
||||
parsed = urlsplit(self.path)
|
||||
if self._is_upload_pack(parsed.path, parsed.query):
|
||||
repo_dir = self._repo_dir(sandbox_root, parsed.path)
|
||||
if repo_dir is None:
|
||||
self.send_error(404)
|
||||
return
|
||||
hook_path = os.environ.get(
|
||||
"GIT_GATE_ACCESS_HOOK", "/etc/git-gate/access-hook",
|
||||
)
|
||||
peer = self.client_address[0]
|
||||
try:
|
||||
hook = subprocess.run(
|
||||
[hook_path, "upload-pack", str(repo_dir), peer, peer],
|
||||
capture_output=True,
|
||||
check=False,
|
||||
timeout=GIT_GATE_TIMEOUT_SECS,
|
||||
)
|
||||
except (OSError, subprocess.SubprocessError) as exc:
|
||||
# The access-hook couldn't be run (missing, not executable,
|
||||
# timed out, …). Fail closed with a real HTTP error rather
|
||||
# than letting the exception kill the handler thread — an
|
||||
# unhandled exception closes the socket with no response, which
|
||||
# the client sees as an opaque "empty reply from server".
|
||||
self.log_message(
|
||||
"access-hook could not run for %s: %s", parsed.path, exc,
|
||||
)
|
||||
self.send_error(503, "git-gate access-hook unavailable")
|
||||
return
|
||||
if hook.returncode != 0:
|
||||
detail = (hook.stderr or hook.stdout).decode(
|
||||
"utf-8", errors="replace",
|
||||
).rstrip()
|
||||
if detail:
|
||||
for line in detail.splitlines():
|
||||
self.log_message("access-hook denied %s: %s",
|
||||
parsed.path, line)
|
||||
else:
|
||||
self.log_message(
|
||||
"access-hook denied %s: exit=%d (no output)",
|
||||
parsed.path, hook.returncode,
|
||||
)
|
||||
self.send_response(403)
|
||||
self.send_header("Content-Type", "text/plain; charset=utf-8")
|
||||
self.end_headers()
|
||||
self.wfile.write(hook.stderr or hook.stdout)
|
||||
return
|
||||
env = os.environ.copy()
|
||||
env.update({
|
||||
"GIT_PROJECT_ROOT": str(sandbox_root),
|
||||
"GIT_HTTP_EXPORT_ALL": "1",
|
||||
"REQUEST_METHOD": self.command,
|
||||
"PATH_INFO": parsed.path,
|
||||
"QUERY_STRING": parsed.query,
|
||||
"CONTENT_TYPE": self.headers.get("content-type", ""),
|
||||
"CONTENT_LENGTH": self.headers.get("content-length", "0"),
|
||||
"REMOTE_ADDR": self.client_address[0],
|
||||
"REMOTE_PORT": str(self.client_address[1]),
|
||||
"REMOTE_USER": "",
|
||||
"SERVER_NAME": self.server.server_name, # type: ignore
|
||||
"SERVER_PORT": str(self.server.server_port), # type: ignore
|
||||
"SERVER_PROTOCOL": self.request_version,
|
||||
})
|
||||
# Attribute the gitleaks-allow supervise proposal (queued by
|
||||
# receive-pack's pre-receive hook, a child of the CGI we spawn below) to
|
||||
# the calling bottle. The hook queues over the control-plane RPC (PRD
|
||||
# 0070 / issue #469), which re-resolves the bottle from these — the same
|
||||
# (source_ip, identity_token) pair that selected the namespaced root
|
||||
# above — so it can only ever queue its own proposals.
|
||||
env["SUPERVISE_SOURCE_IP"] = self.client_address[0]
|
||||
env["SUPERVISE_IDENTITY_TOKEN"] = self.headers.get(IDENTITY_HEADER, "")
|
||||
for header, variable in (
|
||||
("accept", "HTTP_ACCEPT"),
|
||||
("content-encoding", "HTTP_CONTENT_ENCODING"),
|
||||
("git-protocol", "HTTP_GIT_PROTOCOL"),
|
||||
("user-agent", "HTTP_USER_AGENT"),
|
||||
):
|
||||
value = self.headers.get(header)
|
||||
if value:
|
||||
env[variable] = value
|
||||
raw_length = self.headers.get("content-length", "0") or "0"
|
||||
try:
|
||||
length = int(raw_length)
|
||||
except ValueError:
|
||||
self.send_error(400, "Bad Content-Length")
|
||||
return
|
||||
if length < 0:
|
||||
self.send_error(400, "Negative Content-Length")
|
||||
return
|
||||
if length > MAX_BODY_BYTES:
|
||||
self.send_error(413, "Request body too large")
|
||||
return
|
||||
body = self.rfile.read(length) if length else b""
|
||||
proc = subprocess.run(
|
||||
["git", "http-backend"],
|
||||
input=body,
|
||||
env=env,
|
||||
capture_output=True,
|
||||
check=False,
|
||||
timeout=GIT_GATE_TIMEOUT_SECS,
|
||||
)
|
||||
self._write_cgi_response(proc.stdout)
|
||||
|
||||
def _repo_dir(self, sandbox_root: Path, path: str) -> Path | None:
|
||||
root = sandbox_root.resolve()
|
||||
relative = path.lstrip("/").split(".git", 1)[0] + ".git"
|
||||
candidate = (root / relative).resolve()
|
||||
if root not in (candidate, *candidate.parents):
|
||||
return None
|
||||
if not candidate.is_dir():
|
||||
return None
|
||||
return candidate
|
||||
|
||||
@staticmethod
|
||||
def _is_upload_pack(path: str, query: str) -> bool:
|
||||
if path.endswith("/git-upload-pack"):
|
||||
return True
|
||||
if path.endswith("/info/refs"):
|
||||
return any(
|
||||
pair == "service=git-upload-pack"
|
||||
for pair in query.split("&")
|
||||
)
|
||||
return False
|
||||
|
||||
def _write_cgi_response(self, raw: bytes) -> None:
|
||||
head, sep, body = raw.partition(b"\r\n\r\n")
|
||||
line_sep = b"\r\n"
|
||||
if not sep:
|
||||
head, sep, body = raw.partition(b"\n\n")
|
||||
line_sep = b"\n"
|
||||
status = 200
|
||||
headers: list[tuple[str, str]] = []
|
||||
for line in head.split(line_sep):
|
||||
if not line:
|
||||
continue
|
||||
key, _, value = line.decode("latin1").partition(":")
|
||||
value = value.strip()
|
||||
if key.lower() == "status":
|
||||
try:
|
||||
status = int(value.split()[0])
|
||||
except (ValueError, IndexError):
|
||||
self.log_message(
|
||||
"malformed CGI Status header %r; using 500", value,
|
||||
)
|
||||
status = 500
|
||||
else:
|
||||
headers.append((key, value))
|
||||
self.send_response(status)
|
||||
for key, value in headers:
|
||||
self.send_header(key, value)
|
||||
self.end_headers()
|
||||
self.wfile.write(body)
|
||||
|
||||
def log_message(self, format: str, *args: object) -> None: # type: ignore # noqa: A002
|
||||
sys.stdout.write(format % args + "\n")
|
||||
sys.stdout.flush()
|
||||
|
||||
|
||||
def main() -> int:
|
||||
port = int(os.environ.get("GIT_HTTP_PORT", str(DEFAULT_PORT)))
|
||||
orch_url = os.environ.get(ORCHESTRATOR_URL_ENV, "").strip()
|
||||
if not orch_url:
|
||||
# Resolver-only: without an orchestrator the backend can't attribute a
|
||||
# request to a bottle namespace, so it must not serve (fail-closed).
|
||||
sys.stderr.write(
|
||||
f"git-http: {ORCHESTRATOR_URL_ENV} is required "
|
||||
"(no single-tenant flat-root fallback)\n"
|
||||
)
|
||||
return 1
|
||||
server = ThreadingHTTPServer(("0.0.0.0", port), GitHttpHandler)
|
||||
# Resolve each request's sandbox namespace by source IP against the
|
||||
# orchestrator control plane.
|
||||
server.policy_resolver = PolicyResolver(orch_url) # type: ignore[attr-defined]
|
||||
sys.stdout.write(f"git-http listening on 0.0.0.0:{port} (multi-tenant)\n")
|
||||
sys.stdout.flush()
|
||||
server.serve_forever()
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,562 @@
|
||||
"""Pure host-side rendering for the per-agent git-gate (PRD 0008).
|
||||
|
||||
Builds the agent's `.gitconfig` insteadOf rewrites, the known_hosts
|
||||
line, and the entrypoint / pre-receive / access-hook scripts the gateway
|
||||
runs. No docker or forge calls — exposed for tests and reuse across
|
||||
backends. The host-side service (`git_gate.GitGate`) and the deploy-key
|
||||
lifecycle (`git_gate.provision`) each read on their own; `git_gate`
|
||||
re-exports these names for API stability."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
import shlex
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
|
||||
from ...constants import GIT_GATE_TIMEOUT_SECS, IDENTITY_HEADER
|
||||
from ...manifest import ManifestBottle, ManifestGitEntry
|
||||
|
||||
# Short network alias for git-gate inside the gateway. The
|
||||
# agent's `.gitconfig` insteadOf rewrites resolve through this name.
|
||||
GIT_GATE_HOSTNAME = "git-gate"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class GitGateUpstream:
|
||||
"""One bare repo on the gate. `name` drives the bare-repo path
|
||||
(`/git/<name>.git`), the agent's URL after insteadOf rewrite
|
||||
(`git://<gate>/<name>.git`), and the per-upstream credential
|
||||
paths inside the gate (`/git-gate/creds/<name>-key` and
|
||||
`/git-gate/creds/<name>-known_hosts`).
|
||||
|
||||
`identity_file` is the host-side absolute path the gate's start
|
||||
step will docker-cp into the container. `known_host_key` is the
|
||||
KnownHostKey string from the manifest; the gate's start step
|
||||
materialises it into a known_hosts file if non-empty.
|
||||
|
||||
the gate credential paths inside the running gateway."""
|
||||
|
||||
name: str
|
||||
upstream_url: str
|
||||
upstream_host: str
|
||||
upstream_port: str
|
||||
identity_file: str
|
||||
known_host_key: str
|
||||
known_hosts_file: Path = Path()
|
||||
|
||||
def git_gate_upstreams_for_bottle(bottle: ManifestBottle) -> tuple[GitGateUpstream, ...]:
|
||||
"""Lift each `bottle.git` entry into a GitGateUpstream. Unique-Name
|
||||
validation already ran in `manifest.ManifestBottle.from_dict`."""
|
||||
return tuple(
|
||||
GitGateUpstream(
|
||||
name=e.Name,
|
||||
upstream_url=e.Upstream,
|
||||
upstream_host=e.UpstreamHost,
|
||||
upstream_port=e.UpstreamPort,
|
||||
identity_file=e.IdentityFile,
|
||||
known_host_key=e.KnownHostKey,
|
||||
)
|
||||
for e in bottle.git
|
||||
)
|
||||
|
||||
|
||||
def _gitconfig_validate_value(field: str, value: str) -> None:
|
||||
"""Raise ValueError if value contains characters that break gitconfig line syntax."""
|
||||
if "\n" in value or "\r" in value:
|
||||
raise ValueError(
|
||||
f"git-gate: {field} contains a newline, which would inject "
|
||||
f"arbitrary gitconfig keys; rejecting manifest entry"
|
||||
)
|
||||
|
||||
|
||||
def git_gate_render_gitconfig(
|
||||
entries: tuple[ManifestGitEntry, ...], gate_host: str, *, scheme: str = "git",
|
||||
identity_token: str = "",
|
||||
) -> str:
|
||||
"""Render the agent's ~/.gitconfig content for git-gate
|
||||
`insteadOf` rewrites. Pure host-side, no docker / VM;
|
||||
exposed for tests + reuse across backends.
|
||||
|
||||
`gate_host` is the part of the URL between `<scheme>://` and the
|
||||
repo path — backends differ here:
|
||||
- docker: `git-gate` (the short network alias)
|
||||
- firecracker: `<bundle_ip>:<port>` (no DNS on the
|
||||
point-to-point TAP link)
|
||||
|
||||
Empty `entries` returns an empty string so callers can no-op
|
||||
cleanly without conditional formatting at the call site."""
|
||||
if not entries:
|
||||
return ""
|
||||
out = [
|
||||
"# bot-bottle git-gate (PRD 0008): every git operation against\n",
|
||||
"# a declared upstream routes through the gate, which mirrors\n",
|
||||
"# the upstream bidirectionally (gitleaks-scanned push;\n",
|
||||
"# fetch-from-upstream-before-every-upload-pack via access-hook).\n",
|
||||
]
|
||||
# Over the smart-HTTP transport (VM backends), attach the per-bottle
|
||||
# identity token as a request header on requests to the gate, scoped to
|
||||
# its URL so it never goes to any other remote. git-http requires it (the
|
||||
# gateway's mandatory (source_ip, token) attribution). git:// (single-tenant
|
||||
# docker) carries no header — attribution there is the network alias.
|
||||
if identity_token and scheme == "http":
|
||||
_gitconfig_validate_value("identity_token", identity_token)
|
||||
out.append(f'[http "http://{gate_host}/"]\n')
|
||||
out.append(f"\textraHeader = {IDENTITY_HEADER}: {identity_token}\n")
|
||||
for entry in entries:
|
||||
_gitconfig_validate_value(f"repos[{entry.Name!r}].url", entry.Upstream)
|
||||
out.append(f'[url "{scheme}://{gate_host}/{entry.Name}.git"]\n')
|
||||
out.append(f"\tinsteadOf = {entry.Upstream}\n")
|
||||
if entry.RemoteKey and entry.RemoteKey != entry.UpstreamHost:
|
||||
port = (
|
||||
f":{entry.UpstreamPort}"
|
||||
if entry.UpstreamPort and entry.UpstreamPort != "22"
|
||||
else ""
|
||||
)
|
||||
alias = (
|
||||
f"ssh://{entry.UpstreamUser}@{entry.RemoteKey}{port}/"
|
||||
f"{entry.UpstreamPath}"
|
||||
)
|
||||
_gitconfig_validate_value(f"repos[{entry.Name!r}].url (resolved alias)", alias)
|
||||
out.append(f"\tinsteadOf = {alias}\n")
|
||||
return "".join(out)
|
||||
|
||||
|
||||
def git_gate_known_hosts_line(host: str, port: str, key: str) -> str:
|
||||
"""Format `host[:port] key` for OpenSSH's known_hosts. Non-default
|
||||
ports use the bracketed `[host]:port` form (the form OpenSSH writes
|
||||
on disk for hosts reached via a non-22 port)."""
|
||||
if port and port != "22":
|
||||
target = f"[{host}]:{port}"
|
||||
else:
|
||||
target = host
|
||||
return f"{target} {key}\n"
|
||||
|
||||
|
||||
def _git_gate_init_repo_fn(repo_root: str, creds_dir: str) -> list[str]:
|
||||
"""The `init_repo` shell function, parameterized by the bare-repo root
|
||||
and the per-bottle creds dir. Single source of the credential-wiring
|
||||
logic, shared by the single-tenant daemon entrypoint (`/git`,
|
||||
`/git-gate/creds`) and the consolidated per-bottle provisioning
|
||||
(`/git/<bottle_id>`, `/git-gate/creds/<bottle_id>`)."""
|
||||
return [
|
||||
"init_repo() {",
|
||||
" name=$1",
|
||||
" upstream_url=$2",
|
||||
f" keyfile={creds_dir}/${{name}}-key",
|
||||
f" hostsfile={creds_dir}/${{name}}-known_hosts",
|
||||
"",
|
||||
# `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the
|
||||
# host, so chmod-syscalls fail with EROFS. The files already
|
||||
# have the right perms on the host (SSH requires 0600 to load
|
||||
# the key in the first place), so the chmod is best-effort
|
||||
# cleanup for the legacy docker-cp path where the file
|
||||
# landed at the host's umask perms.
|
||||
" chmod 600 \"$keyfile\" 2>/dev/null || true",
|
||||
" if [ -f \"$hostsfile\" ]; then",
|
||||
" chmod 600 \"$hostsfile\" 2>/dev/null || true",
|
||||
" fi",
|
||||
"",
|
||||
f" repo={repo_root}/${{name}}.git",
|
||||
" if [ ! -d \"$repo\" ]; then",
|
||||
" git init --bare \"$repo\" >/dev/null",
|
||||
# --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so a later
|
||||
# `git fetch origin` mirrors the upstream's full ref graph (heads,
|
||||
# tags, notes) into the bare repo at canonical paths. It does NOT set
|
||||
# remote.origin.mirror=true, so an explicit `git push origin
|
||||
# <ref>:<ref>` still pushes one ref.
|
||||
" git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"",
|
||||
" fi",
|
||||
" git -C \"$repo\" config git-gate.identityFile \"$keyfile\"",
|
||||
" git -C \"$repo\" config git-gate.knownHosts \"$hostsfile\"",
|
||||
" git -C \"$repo\" config receive.denyCurrentBranch ignore",
|
||||
" git -C \"$repo\" config receive.advertisePushOptions true",
|
||||
" git -C \"$repo\" config http.receivepack true",
|
||||
" install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"",
|
||||
"}",
|
||||
]
|
||||
|
||||
|
||||
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
|
||||
"""Posix-sh entrypoint. One `init_repo` call per upstream, then
|
||||
`exec git daemon`. The function reads
|
||||
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into
|
||||
the bundle by the renderer) and wires them into each bare repo's
|
||||
config; the access-hook + pre-receive hook pick those paths up
|
||||
at fetch / push time."""
|
||||
lines = ["#!/bin/sh", "set -eu", ""]
|
||||
lines += _git_gate_init_repo_fn("/git", "/git-gate/creds")
|
||||
lines += ["", "mkdir -p /git"]
|
||||
for u in upstreams:
|
||||
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
|
||||
lines.extend([
|
||||
"",
|
||||
"exec git daemon \\",
|
||||
" --reuseaddr \\",
|
||||
f" --timeout={GIT_GATE_TIMEOUT_SECS} \\",
|
||||
f" --init-timeout={GIT_GATE_TIMEOUT_SECS} \\",
|
||||
" --base-path=/git \\",
|
||||
" --export-all \\",
|
||||
" --enable=receive-pack \\",
|
||||
" --access-hook=/etc/git-gate/access-hook \\",
|
||||
" --verbose",
|
||||
])
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# A bottle id namespaces the consolidated gateway's repo + creds dirs; it is
|
||||
# embedded unquoted in the provisioning script, so restrict it to a shell- and
|
||||
# path-safe alphabet (registry ids are token_hex — this is defense in depth).
|
||||
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
|
||||
|
||||
|
||||
def git_gate_render_provision(
|
||||
bottle_id: str, upstreams: tuple[GitGateUpstream, ...],
|
||||
) -> str:
|
||||
"""Posix-sh script that provisions ONE bottle's bare repos into the
|
||||
consolidated gateway (PRD 0070), under `/git/<bottle_id>/` with creds
|
||||
read from `/git-gate/creds/<bottle_id>/`. Init-only — no `git daemon`,
|
||||
since the shared gateway already serves every bottle; run inside the
|
||||
running gateway when the bottle is registered.
|
||||
|
||||
Isolating each bottle's repo root and creds dir by id is what keeps one
|
||||
bottle's push credentials out of another's repos on the shared gateway."""
|
||||
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
|
||||
raise ValueError(f"git-gate: unsafe bottle id {bottle_id!r}")
|
||||
repo_root = f"/git/{bottle_id}"
|
||||
creds_dir = f"/git-gate/creds/{bottle_id}"
|
||||
lines = ["#!/bin/sh", "set -eu", ""]
|
||||
lines += _git_gate_init_repo_fn(repo_root, creds_dir)
|
||||
lines += ["", f"mkdir -p {shlex.quote(repo_root)}"]
|
||||
for u in upstreams:
|
||||
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
def git_gate_render_hook() -> str:
|
||||
"""The shared pre-receive hook: gitleaks-scan all incoming refs,
|
||||
then forward each accepted ref to the real upstream (`origin`)
|
||||
using the per-repo credential. Failure in either phase aborts
|
||||
the push so the agent sees a real rejection. POSIX sh.
|
||||
|
||||
Two phases (scan all, then push all) keeps a hit on ref N from
|
||||
half-pushing refs 1..N-1; both phases re-read stdin from a temp
|
||||
file because pre-receive's stdin is a one-shot stream."""
|
||||
return r"""#!/bin/sh
|
||||
# git-gate pre-receive (PRD 0008). Stdin: <old> <new> <ref> per line.
|
||||
set -u
|
||||
|
||||
refs_file=$(mktemp)
|
||||
trap 'rm -f "$refs_file"' EXIT
|
||||
cat > "$refs_file"
|
||||
|
||||
zero=0000000000000000000000000000000000000000
|
||||
|
||||
supervise_gitleaks_allow() {
|
||||
log_opts=$1
|
||||
ref=$2
|
||||
report_file=$(mktemp)
|
||||
if ! gitleaks git \
|
||||
--log-opts="$log_opts" \
|
||||
--no-banner \
|
||||
--redact \
|
||||
--ignore-gitleaks-allow \
|
||||
--report-format=json \
|
||||
--report-path="$report_file" \
|
||||
--exit-code 0 \
|
||||
1>&2; then
|
||||
rm -f "$report_file"
|
||||
echo "git-gate: gitleaks inline-suppression scan failed for $ref" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
proposal_id=$(
|
||||
PYTHONPATH="/app${PYTHONPATH:+:$PYTHONPATH}" GITLEAKS_ALLOW_REF="$ref" python3 - "$report_file" <<'PY'
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Queue over the control-plane RPC — the git-gate data plane no longer opens
|
||||
# bot-bottle.db (PRD 0070 / issue #469). Attribution is by (source_ip,
|
||||
# identity_token), resolved server-side, so the proposal lands under the
|
||||
# calling bottle exactly as a direct write once did.
|
||||
try:
|
||||
from bot_bottle.gateway.policy_resolver import PolicyResolver, PolicyResolveError
|
||||
from bot_bottle.supervisor.types import TOOL_GITLEAKS_ALLOW
|
||||
except ImportError:
|
||||
from policy_resolver import PolicyResolver, PolicyResolveError
|
||||
from supervisor.types import TOOL_GITLEAKS_ALLOW
|
||||
|
||||
report_path = Path(sys.argv[1])
|
||||
source_ip = os.environ.get("SUPERVISE_SOURCE_IP", "")
|
||||
token = os.environ.get("SUPERVISE_IDENTITY_TOKEN", "")
|
||||
orch_url = os.environ.get("BOT_BOTTLE_ORCHESTRATOR_URL", "")
|
||||
if not source_ip or not orch_url:
|
||||
sys.exit(2)
|
||||
|
||||
try:
|
||||
raw = json.loads(report_path.read_text() or "[]")
|
||||
except json.JSONDecodeError:
|
||||
sys.exit(3)
|
||||
if not isinstance(raw, list):
|
||||
sys.exit(3)
|
||||
if not raw:
|
||||
sys.exit(0)
|
||||
|
||||
ref = os.environ.get("GITLEAKS_ALLOW_REF", "")
|
||||
lines = [
|
||||
"gitleaks inline suppression requires supervisor approval",
|
||||
f"ref: {ref}",
|
||||
"",
|
||||
]
|
||||
for i, finding in enumerate(raw, 1):
|
||||
if not isinstance(finding, dict):
|
||||
continue
|
||||
file_path = finding.get("File", "")
|
||||
line_no = finding.get("StartLine", finding.get("Line", ""))
|
||||
rule_id = finding.get("RuleID", "")
|
||||
commit = finding.get("Commit", "")
|
||||
line = finding.get("Line", "")
|
||||
lines.extend([
|
||||
f"finding {i}:",
|
||||
f" file: {file_path}",
|
||||
f" line: {line_no}",
|
||||
f" rule: {rule_id}",
|
||||
f" commit: {commit}",
|
||||
f" code: {line}",
|
||||
"",
|
||||
])
|
||||
|
||||
payload = "\n".join(lines).rstrip() + "\n"
|
||||
try:
|
||||
proposal_id = PolicyResolver(orch_url).propose_supervise(
|
||||
source_ip, token,
|
||||
tool=TOOL_GITLEAKS_ALLOW,
|
||||
proposed_file=payload,
|
||||
justification=(
|
||||
"git-gate found gitleaks findings hidden by # gitleaks:allow; "
|
||||
"approve only for dummy test fixtures or confirmed false positives"
|
||||
),
|
||||
)
|
||||
except PolicyResolveError:
|
||||
sys.exit(4)
|
||||
if not proposal_id:
|
||||
sys.exit(2)
|
||||
print(proposal_id)
|
||||
PY
|
||||
)
|
||||
rc=$?
|
||||
rm -f "$report_file"
|
||||
if [ "$rc" -eq 0 ] && [ -z "$proposal_id" ]; then
|
||||
return 0
|
||||
fi
|
||||
if [ "$rc" -ne 0 ]; then
|
||||
echo "git-gate: cannot route # gitleaks:allow finding to supervisor; refusing push" >&2
|
||||
return 1
|
||||
fi
|
||||
|
||||
timeout=${SUPERVISE_GITLEAKS_ALLOW_TIMEOUT_SECONDS:-300}
|
||||
case "$timeout" in
|
||||
''|*[!0-9]*)
|
||||
echo "git-gate: invalid SUPERVISE_GITLEAKS_ALLOW_TIMEOUT_SECONDS=$timeout" >&2
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
echo "git-gate: queued # gitleaks:allow supervisor approval $proposal_id" >&2
|
||||
echo "git-gate: approve with './cli.py supervise' to continue this push" >&2
|
||||
waited=0
|
||||
while [ "$waited" -lt "$timeout" ]; do
|
||||
status=$(PYTHONPATH="/app${PYTHONPATH:+:$PYTHONPATH}" python3 - "$proposal_id" <<'PY'
|
||||
import os
|
||||
import sys
|
||||
|
||||
# Non-blocking poll over the control plane. A decided proposal is archived
|
||||
# server-side on read, so no separate archive step is needed here.
|
||||
try:
|
||||
from bot_bottle.gateway.policy_resolver import PolicyResolver, PolicyResolveError
|
||||
except ImportError:
|
||||
from policy_resolver import PolicyResolver, PolicyResolveError
|
||||
|
||||
source_ip = os.environ.get("SUPERVISE_SOURCE_IP", "")
|
||||
token = os.environ.get("SUPERVISE_IDENTITY_TOKEN", "")
|
||||
orch_url = os.environ.get("BOT_BOTTLE_ORCHESTRATOR_URL", "")
|
||||
try:
|
||||
result = PolicyResolver(orch_url).poll_supervise(source_ip, token, sys.argv[1])
|
||||
except PolicyResolveError:
|
||||
sys.exit(2)
|
||||
if result is None:
|
||||
sys.exit(2)
|
||||
status = result.get("status", "")
|
||||
if status in ("pending", "unknown"):
|
||||
sys.exit(2)
|
||||
print(status)
|
||||
PY
|
||||
)
|
||||
rc=$?
|
||||
if [ "$rc" -eq 2 ]; then
|
||||
status=""
|
||||
elif [ "$rc" -ne 0 ]; then
|
||||
status="invalid"
|
||||
fi
|
||||
if [ -n "$status" ]; then
|
||||
case "$status" in
|
||||
approved|modified)
|
||||
echo "git-gate: supervisor approved # gitleaks:allow for $ref" >&2
|
||||
return 0
|
||||
;;
|
||||
rejected)
|
||||
echo "git-gate: supervisor rejected # gitleaks:allow for $ref" >&2
|
||||
return 1
|
||||
;;
|
||||
*)
|
||||
echo "git-gate: invalid supervisor response for # gitleaks:allow" >&2
|
||||
return 1
|
||||
;;
|
||||
esac
|
||||
fi
|
||||
sleep 1
|
||||
waited=$((waited + 1))
|
||||
done
|
||||
echo "git-gate: supervisor approval timed out for # gitleaks:allow; refusing push" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
# Phase 1: gitleaks scan each ref's incoming commits.
|
||||
while IFS=' ' read -r old new ref; do
|
||||
[ -z "$ref" ] && continue
|
||||
[ "$new" = "$zero" ] && continue
|
||||
# Scan only the commits this push introduces — those reachable from
|
||||
# $new but not from any ref the gate already has. Everything already
|
||||
# on the gate arrived via upstream mirror-fetch or a previously
|
||||
# gitleaks-scanned push, so it's already-upstream or already-scanned;
|
||||
# re-scanning it only resurfaces historical fixture findings.
|
||||
#
|
||||
# Applies to both new refs and updates. The old existing-branch range
|
||||
# `$old..$new` walks commits reachable from the new tip but not the
|
||||
# *old branch tip*: on a rebase/force-push onto a freshly-advanced
|
||||
# main that pulls in all of main's new history (incl. the deliberate
|
||||
# sandbox-escape gitleaks fixtures), blocking the push. `--not --all`
|
||||
# excludes anything already on the gate regardless of ancestry, so it
|
||||
# is also correct for non-fast-forward pushes (a rebase can skip
|
||||
# commits off the direct path). Security-equivalent per PRD 0028's
|
||||
# analysis: the bare repo's refs come only from trusted upstream
|
||||
# mirror-fetch or gitleaks-gated pushes.
|
||||
# See PRD 0028 (open question) / issues #106, #346.
|
||||
log_opts="$new --not --all"
|
||||
echo "git-gate: gitleaks scanning $ref ($log_opts)" >&2
|
||||
if ! gitleaks git --log-opts="$log_opts" --no-banner --redact 1>&2; then
|
||||
echo "git-gate: gitleaks rejected push to $ref" >&2
|
||||
exit 1
|
||||
fi
|
||||
if ! supervise_gitleaks_allow "$log_opts" "$ref"; then
|
||||
exit 1
|
||||
fi
|
||||
done < "$refs_file"
|
||||
|
||||
# Phase 2: forward each ref to the upstream (`origin`, configured
|
||||
# in the entrypoint via `git remote add --mirror=fetch`).
|
||||
keyfile=$(git config --get git-gate.identityFile)
|
||||
hostsfile=$(git config --get git-gate.knownHosts)
|
||||
if [ ! -f "$hostsfile" ]; then
|
||||
echo "git-gate: no KnownHostKey configured for this upstream; refusing to push" >&2
|
||||
echo "git-gate: add KnownHostKey to the bottle.git entry and restart the bottle" >&2
|
||||
exit 1
|
||||
fi
|
||||
ssh_cmd="ssh -i $keyfile -o UserKnownHostsFile=$hostsfile -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10"
|
||||
|
||||
push_option_count=${GIT_PUSH_OPTION_COUNT:-0}
|
||||
case "$push_option_count" in
|
||||
''|*[!0-9]*)
|
||||
echo "git-gate: invalid GIT_PUSH_OPTION_COUNT=$push_option_count" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
set --
|
||||
i=0
|
||||
while [ "$i" -lt "$push_option_count" ]; do
|
||||
opt=$(printenv "GIT_PUSH_OPTION_$i" || :)
|
||||
set -- "$@" --push-option="$opt"
|
||||
i=$((i + 1))
|
||||
done
|
||||
|
||||
while IFS=' ' read -r old new ref; do
|
||||
[ -z "$ref" ] && continue
|
||||
if [ "$new" = "$zero" ]; then
|
||||
refspec=":$ref"
|
||||
elif [ "$old" != "$zero" ] && ! git merge-base --is-ancestor "$old" "$new" 2>/dev/null; then
|
||||
refspec="+$new:$ref"
|
||||
else
|
||||
refspec="$new:$ref"
|
||||
fi
|
||||
echo "git-gate: forwarding $ref to origin" >&2
|
||||
if ! GIT_SSH_COMMAND="$ssh_cmd" git push "$@" origin "$refspec" 1>&2; then
|
||||
echo "git-gate: upstream push failed for $ref" >&2
|
||||
exit 1
|
||||
fi
|
||||
done < "$refs_file"
|
||||
|
||||
exit 0
|
||||
"""
|
||||
|
||||
|
||||
def git_gate_render_access_hook() -> str:
|
||||
"""`git daemon --access-hook` script. Runs before each protocol
|
||||
service; for `upload-pack` (fetch / clone / ls-remote / pull) it
|
||||
refreshes the bare repo from upstream first, so the response
|
||||
reflects upstream's current state. For other services (notably
|
||||
`receive-pack`) it returns 0 immediately and lets the existing
|
||||
pre-receive hook gate the operation. POSIX sh.
|
||||
|
||||
The hook receives:
|
||||
$1 service name (`upload-pack`, `receive-pack`, ...)
|
||||
$2 absolute path to the resolved repo
|
||||
$3 client hostname (unused)
|
||||
$4 client tcp address (unused)
|
||||
|
||||
Fail-closed on upstream errors: the agent's fetch fails too,
|
||||
so it never silently sees stale data — matches the PRD's
|
||||
'equivalent to operations against the upstream' contract."""
|
||||
return r"""#!/bin/sh
|
||||
# git-gate access-hook (PRD 0008). $1=service $2=repo $3=host $4=peer
|
||||
set -u
|
||||
service=$1
|
||||
repo_dir=$2
|
||||
|
||||
# Push path keeps its own gating in pre-receive (gitleaks +
|
||||
# forward). Only refresh-from-upstream on fetch operations.
|
||||
if [ "$service" != "upload-pack" ]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
keyfile=$(git -C "$repo_dir" config --get git-gate.identityFile 2>/dev/null || true)
|
||||
hostsfile=$(git -C "$repo_dir" config --get git-gate.knownHosts 2>/dev/null || true)
|
||||
if [ -z "$keyfile" ] || [ ! -f "$hostsfile" ]; then
|
||||
echo "git-gate: missing credentials for $repo_dir; refusing fetch" >&2
|
||||
exit 1
|
||||
fi
|
||||
ssh_cmd="ssh -i $keyfile -o UserKnownHostsFile=$hostsfile -o StrictHostKeyChecking=yes -o IdentitiesOnly=yes -o BatchMode=yes -o ConnectTimeout=10"
|
||||
|
||||
echo "git-gate: refreshing $repo_dir from upstream" >&2
|
||||
if ! GIT_SSH_COMMAND="$ssh_cmd" git -C "$repo_dir" fetch origin --prune >&2; then
|
||||
echo "git-gate: upstream fetch failed for $repo_dir; refusing to serve stale data" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Sync the bare repo's HEAD to upstream's HEAD on the first fetch
|
||||
# (when it still points at the `git init --bare` default of
|
||||
# refs/heads/master and upstream uses something else, the cloned
|
||||
# checkout would fail with "remote HEAD refers to nonexistent ref").
|
||||
# Costs one extra ls-remote on first fetch only; subsequent fetches
|
||||
# skip the branch. If upstream's default branch changes after the
|
||||
# gate has cached it, restart the bottle to resync.
|
||||
if ! git -C "$repo_dir" rev-parse --verify HEAD >/dev/null 2>&1; then
|
||||
upstream_head=$(GIT_SSH_COMMAND="$ssh_cmd" git -C "$repo_dir" \
|
||||
ls-remote --symref origin HEAD 2>/dev/null \
|
||||
| awk '/^ref:/ {print $2; exit}')
|
||||
if [ -n "$upstream_head" ]; then
|
||||
git -C "$repo_dir" symbolic-ref HEAD "$upstream_head" || true
|
||||
fi
|
||||
fi
|
||||
exit 0
|
||||
"""
|
||||
Reference in New Issue
Block a user