Compare commits

...

3 Commits

Author SHA1 Message Date
didericis 77aaabae63 feat(orchestrator): slice 13c(ii) — shared-gateway source-IP allocator
lint / lint (push) Successful in 2m2s
test / unit (pull_request) Successful in 1m4s
test / integration (pull_request) Successful in 20s
test / coverage (pull_request) Successful in 1m12s
Deterministic per-bottle address assignment on the consolidated docker
gateway's shared network — the address the gateway uses as its attribution
key. Pure ipaddress logic; the docker-specific inputs (subnet CIDR, the
gateway container's own address) are gathered by the caller and passed as
`taken`, keeping it testable without docker.

- next_free_ip(cidr, taken): lowest host address not reserved (network +
  broadcast excluded by hosts(); docker's router .1 excluded here) and not
  already assigned (gateway container + live bottles from the registry).
  NoFreeAddressError when the subnet is exhausted.

pyright 0 errors; pylint 9.83/10; unit suite green (the 13 test_sidecar_init
/bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 21:20:13 -04:00
didericis d0b35b5506 feat(git-gate+orchestrator): slice 13c(i) — per-bottle git-gate provisioning render
lint / lint (push) Successful in 2m9s
test / unit (pull_request) Successful in 1m8s
test / integration (pull_request) Successful in 23s
test / coverage (pull_request) Successful in 1m14s
The consolidated gateway serves every bottle's repos under /git/<bottle_id>/
(slice 10), so each bottle's bare repos + per-repo credential config must be
provisioned into that namespace. This adds the renderer for that; placing
the creds + running it inside the gateway is the launch-wiring step.

- Extract the credential-wiring `init_repo` shell into a shared
  `_git_gate_init_repo_fn(repo_root, creds_dir)` — one source for the
  security-sensitive logic. The single-tenant daemon entrypoint is
  byte-unchanged (still /git + /git-gate/creds; existing tests green).
- git_gate_render_provision(bottle_id, upstreams): posix-sh that inits ONE
  bottle's bare repos under /git/<bottle_id>/ reading creds from
  /git-gate/creds/<bottle_id>/. Init-only (no `git daemon` — the shared
  gateway already serves). Isolating each bottle's repo root + creds dir by
  id is what keeps one bottle's push credentials out of another's repos.
- bottle_id is embedded unquoted in the script, so it's restricted to a
  shell/path-safe alphabet (registry ids are token_hex; defense in depth).

pyright 0 errors; pylint 9.83/10; unit suite green (1724 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 21:15:21 -04:00
didericis ea0c070cfe feat(orchestrator): slice 13b — consolidated registration inputs (egress policy)
lint / lint (push) Successful in 2m10s
test / unit (pull_request) Successful in 1m12s
test / integration (pull_request) Successful in 29s
test / coverage (pull_request) Successful in 1m20s
Bridges the per-bottle `prepare` output to the consolidated registry:
`registration_inputs(plan)` turns a prepared egress plan into the
backend-neutral inputs `Orchestrator.launch_bottle` takes.

- egress_policy(plan): the policy blob = the exact routes YAML the
  per-bottle sidecar read from a file; the multi-tenant gateway's
  PolicyResolver now fetches it from the registry per request instead.
  Same egress_render_routes, so a bottle's allow-list is byte-identical
  moving onto the shared gateway.
- RegistrationInputs bundles policy + metadata; metadata carries the human
  slug so the shared registry can map a minted bottle id back to a name
  (console / supervise display).

Host-side glue (imports bot_bottle.egress) used by the launch path — not
the lean orchestrator control-plane process. Not yet wired into launch
(that's 13c/13d), consistent with the slice-6/10 pattern of landing the
primitive before the cut-over.

Key test: the policy round-trips through load_config back to the same
routes + log level — the resolver-served policy reconstructs the per-bottle
egress config exactly.

pyright 0 errors; pylint 9.83/10; unit suite green (1718 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
2026-07-13 20:40:38 -04:00
7 changed files with 328 additions and 21 deletions
+47
View File
@@ -0,0 +1,47 @@
"""Shared-gateway source-IP allocation for the consolidated docker backend
(PRD 0070).
In the consolidated model one gateway container serves every bottle over a
single shared docker network, and each agent bottle attaches with a pinned,
deterministic address that the gateway uses as its **attribution key**. This
allocates those addresses from the network's subnet, skipping the reserved
ones — the network address and broadcast (excluded by `hosts()`), docker's
router `.1`, and everything already in use (`taken`: the gateway container
plus every live bottle, which the caller reads from the registry).
Pure `ipaddress` logic — the docker-specific bits (the subnet CIDR, the
gateway container's own address) are gathered by the caller and passed in, so
this stays testable without docker.
"""
from __future__ import annotations
import ipaddress
from collections.abc import Iterable
class NoFreeAddressError(RuntimeError):
"""The shared gateway network's subnet is exhausted — every host address
is reserved or already assigned to a bottle."""
def next_free_ip(cidr: str, taken: Iterable[str]) -> str:
"""The lowest host address in `cidr` not in `taken` and not docker's
router (`.1`). `taken` must include the gateway container's own address
and every live bottle's. Raises `NoFreeAddressError` if the subnet is
full."""
net = ipaddress.ip_network(cidr, strict=False)
reserved = {str(a) for a in taken}
# Docker assigns the network's first host (.1) to the bridge router; a
# bottle must never be handed that address.
reserved.add(str(net.network_address + 1))
for host in net.hosts(): # hosts() already excludes network + broadcast
candidate = str(host)
if candidate not in reserved:
return candidate
raise NoFreeAddressError(
f"no free address in {cidr} ({len(reserved)} reserved/assigned)"
)
__all__ = ["next_free_ip", "NoFreeAddressError"]
+2
View File
@@ -46,6 +46,7 @@ from .git_gate_render import (
git_gate_known_hosts_line, git_gate_known_hosts_line,
git_gate_render_access_hook, git_gate_render_access_hook,
git_gate_render_entrypoint, git_gate_render_entrypoint,
git_gate_render_provision,
git_gate_render_gitconfig, git_gate_render_gitconfig,
git_gate_render_hook, git_gate_render_hook,
git_gate_upstreams_for_bottle, git_gate_upstreams_for_bottle,
@@ -155,6 +156,7 @@ __all__ = [
"git_gate_render_gitconfig", "git_gate_render_gitconfig",
"git_gate_known_hosts_line", "git_gate_known_hosts_line",
"git_gate_render_entrypoint", "git_gate_render_entrypoint",
"git_gate_render_provision",
"git_gate_render_hook", "git_gate_render_hook",
"git_gate_render_access_hook", "git_gate_render_access_hook",
"provision_git_gate_dynamic_keys", "provision_git_gate_dynamic_keys",
+57 -21
View File
@@ -9,6 +9,7 @@ own; `git_gate` re-exports these names for API stability."""
from __future__ import annotations from __future__ import annotations
import re
import shlex import shlex
from dataclasses import dataclass from dataclasses import dataclass
from pathlib import Path from pathlib import Path
@@ -125,22 +126,18 @@ def git_gate_known_hosts_line(host: str, port: str, key: str) -> str:
return f"{target} {key}\n" return f"{target} {key}\n"
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str: def _git_gate_init_repo_fn(repo_root: str, creds_dir: str) -> list[str]:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then """The `init_repo` shell function, parameterized by the bare-repo root
`exec git daemon`. The function reads and the per-bottle creds dir. Single source of the credential-wiring
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into logic, shared by the single-tenant daemon entrypoint (`/git`,
the bundle by the renderer) and wires them into each bare repo's `/git-gate/creds`) and the consolidated per-bottle provisioning
config; the access-hook + pre-receive hook pick those paths up (`/git/<bottle_id>`, `/git-gate/creds/<bottle_id>`)."""
at fetch / push time.""" return [
lines = [
"#!/bin/sh",
"set -eu",
"",
"init_repo() {", "init_repo() {",
" name=$1", " name=$1",
" upstream_url=$2", " upstream_url=$2",
" keyfile=/git-gate/creds/${name}-key", f" keyfile={creds_dir}/${{name}}-key",
" hostsfile=/git-gate/creds/${name}-known_hosts", f" hostsfile={creds_dir}/${{name}}-known_hosts",
"", "",
# `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the # `|| true`: PRD 0018 chunk 3+ bind-mounts these RO from the
# host, so chmod-syscalls fail with EROFS. The files already # host, so chmod-syscalls fail with EROFS. The files already
@@ -153,14 +150,14 @@ def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
" chmod 600 \"$hostsfile\" 2>/dev/null || true", " chmod 600 \"$hostsfile\" 2>/dev/null || true",
" fi", " fi",
"", "",
" repo=/git/${name}.git", f" repo={repo_root}/${{name}}.git",
" if [ ! -d \"$repo\" ]; then", " if [ ! -d \"$repo\" ]; then",
" git init --bare \"$repo\" >/dev/null", " git init --bare \"$repo\" >/dev/null",
# --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so", # --mirror=fetch sets remote.origin.fetch = +refs/*:refs/* so a later
# a later `git fetch origin` mirrors the upstream's full ref", # `git fetch origin` mirrors the upstream's full ref graph (heads,
# graph (heads, tags, notes) into the bare repo at canonical", # tags, notes) into the bare repo at canonical paths. It does NOT set
# paths. It does NOT set remote.origin.mirror=true, so an", # remote.origin.mirror=true, so an explicit `git push origin
# explicit `git push origin <ref>:<ref>` still pushes one ref.", # <ref>:<ref>` still pushes one ref.
" git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"", " git -C \"$repo\" remote add --mirror=fetch origin \"$upstream_url\"",
" fi", " fi",
" git -C \"$repo\" config git-gate.identityFile \"$keyfile\"", " git -C \"$repo\" config git-gate.identityFile \"$keyfile\"",
@@ -170,9 +167,19 @@ def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
" git -C \"$repo\" config http.receivepack true", " git -C \"$repo\" config http.receivepack true",
" install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"", " install -m 755 /etc/git-gate/pre-receive \"$repo/hooks/pre-receive\"",
"}", "}",
"",
"mkdir -p /git",
] ]
def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
"""Posix-sh entrypoint. One `init_repo` call per upstream, then
`exec git daemon`. The function reads
`/git-gate/creds/<name>-{key,known_hosts}` (bind-mounted into
the bundle by the renderer) and wires them into each bare repo's
config; the access-hook + pre-receive hook pick those paths up
at fetch / push time."""
lines = ["#!/bin/sh", "set -eu", ""]
lines += _git_gate_init_repo_fn("/git", "/git-gate/creds")
lines += ["", "mkdir -p /git"]
for u in upstreams: for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}") lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
lines.extend([ lines.extend([
@@ -190,6 +197,35 @@ def git_gate_render_entrypoint(upstreams: tuple[GitGateUpstream, ...]) -> str:
return "\n".join(lines) + "\n" return "\n".join(lines) + "\n"
# A bottle id namespaces the consolidated gateway's repo + creds dirs; it is
# embedded unquoted in the provisioning script, so restrict it to a shell- and
# path-safe alphabet (registry ids are token_hex — this is defense in depth).
_SAFE_BOTTLE_ID = re.compile(r"[A-Za-z0-9_-]+")
def git_gate_render_provision(
bottle_id: str, upstreams: tuple[GitGateUpstream, ...],
) -> str:
"""Posix-sh script that provisions ONE bottle's bare repos into the
consolidated gateway (PRD 0070), under `/git/<bottle_id>/` with creds
read from `/git-gate/creds/<bottle_id>/`. Init-only — no `git daemon`,
since the shared gateway already serves every bottle; run inside the
running gateway when the bottle is registered.
Isolating each bottle's repo root and creds dir by id is what keeps one
bottle's push credentials out of another's repos on the shared gateway."""
if not _SAFE_BOTTLE_ID.fullmatch(bottle_id):
raise ValueError(f"git-gate: unsafe bottle id {bottle_id!r}")
repo_root = f"/git/{bottle_id}"
creds_dir = f"/git-gate/creds/{bottle_id}"
lines = ["#!/bin/sh", "set -eu", ""]
lines += _git_gate_init_repo_fn(repo_root, creds_dir)
lines += ["", f"mkdir -p {shlex.quote(repo_root)}"]
for u in upstreams:
lines.append(f"init_repo {shlex.quote(u.name)} {shlex.quote(u.upstream_url)}")
return "\n".join(lines) + "\n"
def git_gate_render_hook() -> str: def git_gate_render_hook() -> str:
"""The shared pre-receive hook: gitleaks-scan all incoming refs, """The shared pre-receive hook: gitleaks-scan all incoming refs,
then forward each accepted ref to the real upstream (`origin`) then forward each accepted ref to the real upstream (`origin`)
+57
View File
@@ -0,0 +1,57 @@
"""Consolidated registration inputs (PRD 0070, docker slice).
Bridges the existing per-bottle `prepare` output to the consolidated
registry: turns a prepared bottle's egress plan into the backend-neutral
inputs `Orchestrator.launch_bottle` takes — the egress **policy** blob and
launch **metadata**.
The policy blob is the exact routes YAML the per-bottle egress sidecar used
to read from a file; in the consolidated model the multi-tenant gateway's
`PolicyResolver` fetches it from the registry per request (keyed by source
IP) instead. Same render, so consolidated and single-tenant egress apply
byte-identical policy — a bottle's allow-list doesn't change when it moves
onto the shared gateway.
Host-side glue (imports `bot_bottle.egress`), used by the launch path — not
by the lean orchestrator control-plane process itself.
"""
from __future__ import annotations
import json
from dataclasses import dataclass
from ..egress import EgressPlan, egress_render_routes
@dataclass(frozen=True)
class RegistrationInputs:
"""What `Orchestrator.launch_bottle` needs to register a bottle, derived
from its prepared plan. `policy` is served verbatim by the gateway's
`/resolve`; `metadata` is opaque forward-compat state — it carries the
human slug so the console / supervise can show a name, not just the
minted bottle id."""
policy: str
metadata: str
def egress_policy(plan: EgressPlan) -> str:
"""The bottle's egress policy blob: the routes YAML the gateway serves
and the addon parses with `load_config`. Identical to the per-bottle
`routes.yaml` render, so the consolidated path applies the same
allow-list."""
return egress_render_routes(plan.routes, log=plan.log)
def registration_inputs(plan: EgressPlan) -> RegistrationInputs:
"""Assemble the orchestrator registration inputs from a prepared egress
plan. `metadata` records the slug so the shared registry can map a minted
bottle id back to its human name."""
return RegistrationInputs(
policy=egress_policy(plan),
metadata=json.dumps({"slug": plan.slug}),
)
__all__ = ["RegistrationInputs", "egress_policy", "registration_inputs"]
+42
View File
@@ -0,0 +1,42 @@
"""Unit: shared-gateway source-IP allocation (PRD 0070)."""
from __future__ import annotations
import unittest
from bot_bottle.backend.docker.gateway_net import NoFreeAddressError, next_free_ip
class TestNextFreeIp(unittest.TestCase):
def test_skips_router_and_returns_first_host(self) -> None:
# .1 is docker's router; the first assignable address is .2.
self.assertEqual("172.18.0.2", next_free_ip("172.18.0.0/16", []))
def test_skips_taken_addresses(self) -> None:
# Gateway container holds .2, a live bottle holds .3 -> next is .4.
self.assertEqual(
"172.18.0.4", next_free_ip("172.18.0.0/16", ["172.18.0.2", "172.18.0.3"]),
)
def test_taken_order_does_not_matter(self) -> None:
self.assertEqual(
"172.18.0.2", next_free_ip("172.18.0.0/16", ["172.18.0.3", "172.18.0.5"]),
)
def test_allocation_is_deterministic(self) -> None:
taken = ["172.18.0.2"]
self.assertEqual(next_free_ip("172.18.0.0/16", taken),
next_free_ip("172.18.0.0/16", taken))
def test_raises_when_subnet_exhausted(self) -> None:
# /30: hosts are .1 (router, reserved) and .2; taking .2 leaves none.
with self.assertRaises(NoFreeAddressError):
next_free_ip("10.9.9.0/30", ["10.9.9.2"])
def test_accepts_host_bits_set_cidr(self) -> None:
# A container's inspected address arrives as e.g. 172.18.0.2/16.
self.assertEqual("172.18.0.2", next_free_ip("172.18.0.5/16", []))
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,67 @@
"""Unit: consolidated per-bottle git-gate provisioning render (PRD 0070)."""
from __future__ import annotations
import unittest
from bot_bottle.git_gate_render import (
GitGateUpstream,
git_gate_render_entrypoint,
git_gate_render_provision,
)
def _ups(*names: str) -> tuple[GitGateUpstream, ...]:
return tuple(
GitGateUpstream(
name=n,
upstream_url=f"ssh://git@github.com/x/{n}.git",
upstream_host="github.com",
upstream_port="22",
identity_file="",
known_host_key="",
)
for n in names
)
class TestProvisionRender(unittest.TestCase):
def test_namespaces_repos_and_creds_by_bottle_id(self) -> None:
script = git_gate_render_provision("bottleab12", _ups("foo"))
self.assertIn("repo=/git/bottleab12/${name}.git", script)
self.assertIn("keyfile=/git-gate/creds/bottleab12/${name}-key", script)
self.assertIn("mkdir -p /git/bottleab12", script)
def test_one_init_repo_call_per_upstream(self) -> None:
script = git_gate_render_provision("b1", _ups("foo", "bar"))
calls = [l for l in script.splitlines() if l.startswith("init_repo ")]
self.assertEqual(2, len(calls))
def test_provision_does_not_start_the_daemon(self) -> None:
# The shared gateway already serves; provisioning is init-only.
self.assertNotIn("git daemon", git_gate_render_provision("b1", _ups("foo")))
def test_installs_pre_receive_hook(self) -> None:
script = git_gate_render_provision("b1", _ups("foo"))
self.assertIn("install -m 755 /etc/git-gate/pre-receive", script)
def test_rejects_unsafe_bottle_id(self) -> None:
for bad in ("../etc", "a/b", "a b", "a;rm", ""):
with self.assertRaises(ValueError):
git_gate_render_provision(bad, _ups("foo"))
class TestEntrypointUnchanged(unittest.TestCase):
"""The shared `_git_gate_init_repo_fn` refactor must not alter the
single-tenant daemon entrypoint's output."""
def test_entrypoint_still_single_tenant_flat(self) -> None:
script = git_gate_render_entrypoint(_ups("foo"))
self.assertIn("repo=/git/${name}.git", script) # flat, not namespaced
self.assertIn("keyfile=/git-gate/creds/${name}-key", script)
self.assertIn("--base-path=/git", script)
self.assertIn("exec git daemon", script)
if __name__ == "__main__":
unittest.main()
@@ -0,0 +1,56 @@
"""Unit: consolidated registration inputs — egress policy round-trip (PRD 0070)."""
from __future__ import annotations
import json
import unittest
from pathlib import Path
from bot_bottle.egress import EgressPlan, EgressRoute
from bot_bottle.egress_addon_core import LOG_BLOCKS, load_config
from bot_bottle.orchestrator.registration import (
RegistrationInputs,
egress_policy,
registration_inputs,
)
def _plan(routes: tuple[EgressRoute, ...], *, slug: str = "demo", log: int = 0) -> EgressPlan:
return EgressPlan(
slug=slug,
routes_path=Path("/unused/routes.yaml"),
routes=routes,
token_env_map={},
log=log,
)
class TestEgressPolicy(unittest.TestCase):
def test_policy_round_trips_through_load_config(self) -> None:
# The policy the gateway serves must parse back to the same allow-list
# the per-bottle sidecar applied — moving onto the shared gateway must
# not change a bottle's egress.
routes = (EgressRoute(host="api.example.com"), EgressRoute(host="pypi.org"))
cfg = load_config(egress_policy(_plan(routes)))
self.assertEqual(("api.example.com", "pypi.org"), tuple(r.host for r in cfg.routes))
def test_policy_preserves_log_level(self) -> None:
plan = _plan((EgressRoute(host="x.example.com"),), log=LOG_BLOCKS)
self.assertEqual(LOG_BLOCKS, load_config(egress_policy(plan)).log)
def test_empty_routes_yield_deny_all(self) -> None:
cfg = load_config(egress_policy(_plan(())))
self.assertEqual((), cfg.routes) # no routes → default-deny
class TestRegistrationInputs(unittest.TestCase):
def test_bundles_policy_and_slug_metadata(self) -> None:
plan = _plan((EgressRoute(host="api.example.com"),), slug="my-bot")
inputs = registration_inputs(plan)
self.assertIsInstance(inputs, RegistrationInputs)
self.assertEqual("my-bot", json.loads(inputs.metadata)["slug"])
self.assertEqual(egress_policy(plan), inputs.policy) # same blob egress_policy renders
if __name__ == "__main__":
unittest.main()