Compare commits

...

23 Commits

Author SHA1 Message Date
didericis-codex 016e59e029 docs(prd): tighten audit ordering and chain guarantees
prd-number-check / require-numbered-prds (pull_request) Failing after 12s
tracker-policy-pr / check-pr (pull_request) Successful in 15s
2026-07-26 17:44:41 +00:00
didericis-claude f3664dea9f docs(prd): drop policy_version (manifest is the policy); add engine field
prd-number-check / require-numbered-prds (pull_request) Failing after 6s
tracker-policy-pr / check-pr (pull_request) Successful in 5s
Per review: bot-bottle has no separate policy artifact — a bottled agent's
egress routes etc. are declared in its manifest (manifest/egress.py), so
manifest_digest already pins the policy in force. Remove the redundant
policy_version. Given a fixed manifest, the only other axis that changes an
outcome is the enforcing code, so add 'engine' (bot-bottle version + git
SHA) as a trusted field. Runtime operator overrides (supervise egress-allow)
are themselves audit events, so effective policy = manifest_digest + logged
deltas, reconstructable from the chain. Note the build must stamp the git
SHA (only version=0.1.0 exists today).

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 08:28:10 +00:00
didericis-claude 88b82a169e docs(prd): complete audit-event contract to #487 acceptance checklist
prd-number-check / require-numbered-prds (pull_request) Failing after 10s
tracker-policy-pr / check-pr (pull_request) Successful in 11s
Expand the PRD from a schema sketch to the full contract the issue mandates
(issue is spec-only: 'defines the contract; implementation may be split
into follow-up PRs'):

- Envelope: add observed vs event timestamps, bottle/activation ids,
  manifest_digest + policy_version, actor/action/resource/outcome,
  correlation_id/causation_id, sensitivity class, typed payload, segment id.
- Add a per-field trust-provenance table (trusted vs claimed for every
  common field); per-type trusted/claimed in the registry.
- Canonicalization: normative, reproducible hash-chain test vectors;
  idempotency (id key, UPSERT), ordering guarantees, and behavior across
  rotation/restart/import/truncation (truncated-tail vs gap).
- Storage: indexable fields + local audit query/verify/rebuild/import CLI.
- Registry: cover all mandated groups incl hostctl.*, egress
  request/decision/cutoff/anomaly, commit.signed (#480), auth/authz, and
  audit.* self-events; schema-evolution + backward-compatible reader rules.
- Export: #324 delivery contract (payload, (epoch,seq) cursor, dedup,
  backpressure, retention ordering); #480 mapping preserving its
  byte-to-activation-key guarantee.
- No raw prompt/response/body capture by default.
- Add an acceptance-criteria coverage table mapping each #487 checkbox to a
  section.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 08:08:22 +00:00
didericis-claude 5a9428cc86 docs(prd): flatten to one untrusted region + address CloudEvents/OTel export
prd-number-check / require-numbered-prds (pull_request) Failing after 7s
tracker-policy-pr / check-pr (pull_request) Successful in 19s
- Remove the trusted sub-block: everything outside untrusted (chain
  metadata, producer/host, bottled_agent, ts_*) is trusted by construction.
  A field is trusted unless deliberately placed under untrusted (#495).
- Add Export/interoperability section: the flattened envelope projects
  cleanly onto CloudEvents JSON (top-level scalars -> context/extension
  attributes, untrusted -> data) and the OpenTelemetry Logs data model
  (ts_wall -> Timestamp, trusted -> botbottle.* attributes, untrusted ->
  botbottle.untrusted.*). Attribution preserved structurally; integrity
  fields carried as data with verification always on the native journal.
  Satisfies #487's export/interop requirement. Export adapters = chunk 6.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 07:45:58 +00:00
didericis-claude 60039f2eb3 docs(prd): address review #495 on audit-event schema
prd-number-check / require-numbered-prds (pull_request) Failing after 7s
tracker-policy-pr / check-pr (pull_request) Successful in 8s
- Move ts_wall/ts_mono and producer inside the trusted block (host-supplied).
- Rename subject to bottled_agent everywhere (field + lifecycle.bottled_agent_* leaves).
- Add explicit epoch (writer-boot) counter + chain-head carry for ordering across host-controller restarts.
- Commit to a single writer per host (host controller owns it).
- Reuse egress dlp_detectors (scan_token_patterns/redact_tokens) for redaction; exclude scan_entropy as brittle on structured audit values.
- Retention: carry rotated-out chain head as new segment genesis prev.
- Fold resolved points into design; trim open questions.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 07:35:16 +00:00
didericis-claude bc42836327 docs(prd): canonical tamper-evident audit-event schema (#487)
prd-number-check / require-numbered-prds (pull_request) Failing after 8s
tracker-policy-pr / check-pr (pull_request) Successful in 6s
Draft PRD for a unified, versioned audit-event envelope with a
trusted/untrusted field split, canonical JSON + per-writer hash chain,
an append-only JSONL journal as source of truth, a rebuildable SQLite
index for local query, and an initial event registry. Scheduled to land
immediately after the host controller (#468), which becomes its first
producer.

Refs #487

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 07:12:40 +00:00
didericis-codex 0fc5457e41 docs(adr): align tracker policy title
test / integration-docker (push) Successful in 17s
test / unit (push) Successful in 48s
lint / lint (push) Successful in 59s
test / coverage (push) Successful in 16s
Update Quality Badges / update-badges (push) Successful in 4m4s
2026-07-26 02:54:56 -04:00
didericis-codex c0493f0b01 ci(tracker): require one metadata owner per pull request 2026-07-26 02:54:56 -04:00
didericis-codex 5828f5e900 ci(tracker): allow labelled standalone pull requests 2026-07-26 02:54:56 -04:00
didericis-codex be025ff8fb docs(prd): explain superseded dashboard designs 2026-07-26 02:54:56 -04:00
didericis-codex 9537c96586 docs: streamline design workflow guidance 2026-07-26 02:54:56 -04:00
didericis-codex 6b43fe73c1 docs: add design workflow guide 2026-07-26 02:54:56 -04:00
didericis-codex d3370a88bb ci(prd): require manual numbering before merge
prd-number-check / require-numbered-prds (pull_request) Successful in 10s
tracker-policy-pr / check-pr (pull_request) Successful in 14s
test / unit (pull_request) Successful in 54s
test / integration-docker (pull_request) Successful in 54s
test / coverage (pull_request) Successful in 18s
test / integration-docker (push) Successful in 16s
test / unit (push) Successful in 50s
lint / lint (push) Successful in 1m0s
test / coverage (push) Successful in 15s
Update Quality Badges / update-badges (push) Successful in 4m1s
2026-07-26 02:33:14 -04:00
didericis-codex 39167528db ci(test): publish infra after pre-release checks
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / integration-docker (pull_request) Successful in 20s
test / unit (pull_request) Successful in 4m11s
test / coverage (pull_request) Successful in 17s
test / integration-docker (push) Successful in 15s
test / unit (push) Successful in 49s
test / coverage (push) Successful in 23s
2026-07-26 06:24:28 +00:00
didericis-codex b25ace4c00 ci(test): split automated and pre-release suites
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / integration-docker (pull_request) Successful in 22s
test / unit (pull_request) Successful in 54s
test / coverage (pull_request) Successful in 17s
2026-07-26 06:22:53 +00:00
didericis-codex 9c06702b32 docs(prd): number merged placeholder documents
prd-number / assign-numbers (push) Failing after 25s
lint / lint (push) Successful in 4m7s
Update Quality Badges / update-badges (push) Failing after 13m37s
test / coverage (push) Has been skipped
test / integration-docker (push) Failing after 13m51s
test / integration-firecracker (push) Failing after 13m48s
test / integration-macos (push) Failing after 13m54s
test / unit (push) Failing after 13m53s
test / publish-infra (push) Has been skipped
2026-07-26 06:06:04 +00:00
didericis-codex cd0983d943 docs(prd): update shipped draft statuses
prd-number / assign-numbers (push) Failing after 24s
2026-07-26 06:04:15 +00:00
didericis-codex 99176b1edf docs(prd): mark superseded architecture contracts 2026-07-26 06:02:47 +00:00
Quality Badge Bot 652f14dcb1 chore: update quality badges
- Coverage: 83%
- Core coverage: 94%

[skip ci]
2026-07-26 02:36:35 +00:00
didericis-claude 38c13708c7 Merge pull request 'PRD prd-new: macOS (Apple Container) CI runner' (#479) from prd-macos-container-ci-runner into main
test / integration-macos (push) Has been skipped
prd-number / assign-numbers (push) Failing after 27s
test / integration-docker (push) Successful in 16s
lint / lint (push) Successful in 1m7s
Update Quality Badges / update-badges (push) Successful in 58s
test / unit (push) Successful in 2m11s
test / integration-firecracker (push) Successful in 5m2s
test / coverage (push) Successful in 25s
test / publish-infra (push) Successful in 2m1s
2026-07-25 22:34:47 -04:00
didericis-claude 82669b22d5 fix: doctor probes backend readiness; install.sh resolves user-scripts dir
test / integration-docker (push) Successful in 20s
prd-number / assign-numbers (push) Failing after 24s
test / unit (push) Successful in 57s
lint / lint (push) Successful in 1m1s
Update Quality Badges / update-badges (push) Failing after 54s
test / integration-firecracker (push) Successful in 5m7s
test / coverage (push) Successful in 27s
test / publish-infra (push) Successful in 2m34s
Addresses the third review round on PR #481.

- `bot-bottle doctor` now checks `is_backend_ready()` (a full backend
  status() probe: daemon reachable, network pool present, KVM usable)
  instead of the cheap PATH-only `is_backend_available()`. A host with a
  stopped Docker daemon or half-configured Firecracker no longer reports
  `ok: backend` / exit 0 when `start` can't actually work; each not-ready
  backend prints its own diagnostics, and doctor passes only if at least
  one backend is ready.
- `install.sh` resolves the pip `--user` scripts directory from the
  interpreter (`sysconfig.get_path("scripts", get_preferred_scheme("user"))`)
  instead of hardcoding `~/.local/bin`, which is wrong on a python.org
  macOS interpreter (`~/Library/Python/<X.Y>/bin`). The PATH guidance now
  prints the actual directory.

Tests: doctor tests mock `is_backend_ready` (the readiness contract) and
cover the not-ready → fail path; a new install-script test drives the
macOS `osx_framework_user` scheme and asserts it resolves a
non-~/.local/bin directory.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:34:25 -04:00
didericis-claude 1a4b390e8a fix: make installed wheel self-contained + harden install.sh prereqs
Addresses the review on PR #481.

Self-contained wheel (review point 1): the gateway/infra/orchestrator
images build from a context that must hold bot_bottle/, pyproject.toml,
and the root-level Dockerfiles. Modules previously located these by
walking __file__ to the repo root, so an installed wheel (package in
site-packages, no repo root) passed `doctor` but failed `start`.

- Add bot_bottle/resources.py: build_root() returns the repo root in a
  checkout (unchanged) or a staged copy from the wheel's bundled
  _resources/ otherwise; dockerfile()/nix_netpool_module()/
  netpool_script() derive from it.
- setup.py bundles the root Dockerfiles, nix module, netpool script, and
  pyproject.toml into bot_bottle/_resources/ at build; MANIFEST.in ships
  them in the sdist.
- Route every _REPO_ROOT/_REPO_DIR call site (docker/macos launch, macos
  infra, firecracker infra_vm/infra_artifact/setup, orchestrator
  lifecycle/gateway) through resources. Checkout behavior is unchanged.

install.sh prerequisites (review point 2): check for git when installing
a git+ spec, and — before the pip fallback — that pip is usable and the
interpreter isn't externally managed (PEP 668), pointing at pipx.

Tests: test_resources covers checkout + staged-wheel layouts;
test_wheel_install builds the wheel, installs it into an isolated venv,
and asserts `doctor` runs and build_root() yields a valid context.
Running `start` end-to-end still needs a Docker/KVM host (CI).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:34:25 -04:00
didericis-claude 955cb3bcbd feat: add quick install script and packaging (#197)
Give bot-bottle a real distribution path so new users can install
without cloning the repo:

- pyproject.toml: full project metadata, a `bot-bottle` console-script
  entry point (bot_bottle.cli:main), and package-data for the runtime
  assets (Dockerfiles, egress entrypoint, netpool defaults, macos init).
  Still zero runtime pip dependencies.
- install.sh: POSIX, sudo-free, idempotent bootstrapper — checks Python
  >= 3.11, creates ~/.bot-bottle/{agents,bottles,contrib}, installs via
  pipx (pip --user fallback), then runs `bot-bottle doctor`.
- `bot-bottle doctor`: new store-free subcommand reporting Python
  version, backend availability (reuses is_backend_available rather than
  hardcoding Docker), and config-dir presence. Exits non-zero when a hard
  prerequisite is unmet.
- PRD prd-new-install-script and unit tests for doctor, the packaging
  contract, and the install script.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 22:34:25 -04:00
70 changed files with 2737 additions and 490 deletions
+6
View File
@@ -20,3 +20,9 @@ omit =
bot_bottle/cli/tui.py
bot_bottle/cli/init.py
tests/*
# Build-time only: setuptools invokes it out-of-process to build the
# wheel/sdist (it's never imported by the running app), so in-process
# coverage can't reach it. Its one job — bundling the root resources into
# bot_bottle/_resources/ — is exercised end-to-end by test_wheel_install,
# which builds and installs a real wheel and checks the result.
setup.py
+26
View File
@@ -0,0 +1,26 @@
name: prd-number-check
on:
pull_request:
types: [opened, reopened, synchronize]
jobs:
require-numbered-prds:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Reject unnumbered PRDs
run: |
unnumbered=$(find docs/prds -maxdepth 1 -type f \
-name 'prd-new-*.md' -print | sort)
if [ -n "$unnumbered" ]; then
echo "::error::Assign every new PRD its final sequential number before merge."
echo "Unnumbered PRDs:"
echo "$unnumbered"
exit 1
fi
echo "All PRDs have final numbers."
-122
View File
@@ -1,122 +0,0 @@
# Assign sequential numbers to prd-new-*.md files on merge to main.
#
# When a PR merges to main and includes prd-new-*.md files this workflow:
# 1. Finds the next available NNNN number by scanning existing PRDs.
# 2. Renames each prd-new-*.md to NNNN-<slug>.md.
# 3. Updates the title header (# PRD prd-new: → # PRD NNNN:).
# 4. Flips Status: Draft → Active when the push touched files outside
# docs/prds/ anywhere in its commit range (i.e. the implementation
# shipped together with the PRD).
# 5. Commits the renaming back to main.
#
# No-op if the working tree contains no prd-new-*.md files.
#
# NOTE: The workflow scans the working tree (not just HEAD~1..HEAD) because
# PRs land as multi-commit pushes and the prd-new file is often added in an
# earlier commit on the branch, not in the final squash/merge commit.
name: prd-number
on:
push:
branches:
- main
paths:
- 'docs/prds/prd-new-*.md'
jobs:
assign-numbers:
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
token: ${{ secrets.GITHUB_TOKEN }}
# No actions/setup-python: the inline script is stdlib-only on the
# image's system Python 3.12 (older act_runner mishandles its PATH).
- name: Configure git
run: |
git config user.name "github-actions[bot]"
git config user.email "github-actions[bot]@users.noreply.github.com"
- name: Assign PRD numbers
run: |
python3 - <<'EOF'
import os
import re
import subprocess
import sys
from pathlib import Path
prds_dir = Path("docs/prds")
# Scan the working tree — prd-new files may have landed in any
# commit of a multi-commit push, not just HEAD.
new_prds = sorted(prds_dir.glob("prd-new-*.md"))
if not new_prds:
print("No prd-new-*.md files found — nothing to do.")
sys.exit(0)
# Determine whether non-PRD files were also changed anywhere in
# the push range (BEFORE_SHA → HEAD). Falls back to HEAD~1 when
# the env var isn't set (e.g. local act runs).
before_sha = os.environ.get("GITHUB_EVENT_BEFORE", "HEAD~1")
all_changed = subprocess.run(
["git", "diff", "--name-only", before_sha, "HEAD"],
capture_output=True, text=True, check=True,
).stdout.splitlines()
non_prd_changed = any(
not f.startswith("docs/prds/") for f in all_changed
)
# Find next available number.
existing = sorted(
int(m.group(1))
for p in prds_dir.glob("*.md")
if (m := re.match(r"^(\d{4})-", p.name))
)
next_num = (max(existing) + 1) if existing else 1
for prd_path in sorted(new_prds):
slug = re.sub(r"^prd-new-", "", prd_path.stem)
new_name = f"{next_num:04d}-{slug}.md"
new_path = prds_dir / new_name
print(f" {prd_path.name} → {new_name}")
content = prd_path.read_text()
# Update title header.
content = re.sub(
r"^(#\s+PRD\s+)prd-new(:)",
rf"\g<1>{next_num:04d}\2",
content,
count=1,
flags=re.MULTILINE,
)
# Conditionally flip Status.
if non_prd_changed:
content = re.sub(
r"(\*\*Status:\*\*\s*)Draft",
r"\g<1>Active",
content,
count=1,
)
new_path.write_text(content)
subprocess.run(["git", "rm", str(prd_path)], check=True)
subprocess.run(["git", "add", str(new_path)], check=True)
next_num += 1
subprocess.run(
["git", "commit", "-m", "ci(prd): assign sequential numbers to new PRDs"],
check=True,
)
subprocess.run(["git", "push"], check=True)
EOF
+337
View File
@@ -0,0 +1,337 @@
# Run the complete backend test suite before a release. This workflow is
# intentionally manual because Firecracker and macOS use privileged,
# self-hosted runners.
#
# The suite uses stdlib `unittest` discovery — no external Python
# dependencies are required to execute it. Tests are split by directory:
#
# tests/unit/ — pure unit tests; always run
# tests/integration/ — need a reachable backend; skip cleanly when
# the backend isn't available on the runner
# tests/canaries/ — upstream regression canaries; run on a separate
# schedule (see canaries.yml), not here
#
# Unit, Docker, and Firecracker run once under coverage and upload a small
# .coverage.* artifact for the combined coverage job. macOS reports coverage
# in place because it is an advisory host-mode runner.
name: pre-release-test
on:
workflow_dispatch:
jobs:
unit:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# No actions/setup-python: the runner image already ships Python 3.12,
# and older act_runner engines mishandle setup-python's PATH (coverage
# lands in one interpreter, `python3` resolves to another). Install
# straight into the ephemeral job container's system Python —
# --break-system-packages is safe because the container is disposable.
- name: Install dev requirements
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
- name: Run unit tests with coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.unit
run: python3 -m coverage run -m unittest discover -t . -s tests/unit -v
- name: Report unit coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.unit
run: python3 -m coverage report -m
# upload-artifact@v3's glob skips dotfiles, so a bare `.coverage.unit`
# silently uploads nothing ("No files were found"). Stage it under a
# non-dot name; the coverage job renames it back before `coverage
# combine`. `cp` also fails loudly if coverage never wrote the file.
- name: Stage unit coverage for upload
run: cp .coverage.unit coverage-unit.dat
- name: Upload unit coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-unit
path: coverage-unit.dat
integration-docker:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
# No actions/setup-python (see the note in the `unit` job); the
# container's system Python 3.12 runs the stdlib test suite directly.
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
# Fail loudly if the backend this job promises isn't actually usable,
# rather than letting every test silently `unittest.skip` and the job
# go green on zero coverage. `backend status` prints a clear per-check
# summary (docker on PATH, daemon reachable) and exits non-zero when a
# prerequisite is missing — the same readiness check the skip guards
# gate on via `has_backend`.
- name: Preflight — Docker backend is ready
run: |
python3 --version
python3 cli.py backend status --backend=docker
- name: Run integration tests (docker) with coverage
env:
BOT_BOTTLE_BACKEND: docker
COVERAGE_FILE: ${{ github.workspace }}/.coverage.docker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
# Non-dot name so upload-artifact's dotfile-skipping glob picks it up.
- name: Stage docker coverage for upload
run: cp .coverage.docker coverage-docker.dat
- name: Upload docker coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-docker
path: coverage-docker.dat
# Integration tests against the Firecracker backend. Runs on a self-hosted
# KVM runner (label `kvm`) where /dev/kvm and the TAP/nft pool are available.
#
# Manual only: the privileged KVM runner does not execute proposed changes
# unattended.
#
# Runner prerequisites (provision once; see README "Firecracker on Linux"):
# `firecracker` on PATH, `/dev/kvm` accessible, cached kernel +
# static dropbear at /var/cache/bot-bottle-fc/dropbear, and the pool as a
# persistent systemd unit.
#
# The infra candidate is built here directly (no artifact download) to
# eliminate the ~70 s ubuntu-latest upload + ~83 s combined download that
# the old build-infra → integration-firecracker + coverage chain incurred.
integration-firecracker:
runs-on: [self-hosted, kvm]
if: github.event_name == 'workflow_dispatch'
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Preflight — Firecracker host is ready
run: |
command -v firecracker >/dev/null || {
echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
# `backend status` exits non-zero unless the TAP pool is up + no
# range overlap; it prints the exact `backend setup` fix.
python3 cli.py backend status --backend=firecracker
- name: Build infra candidate from this checkout
env:
BOT_BOTTLE_FC_DROPBEAR: /var/cache/bot-bottle-fc/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --output infra-candidate --reuse-published
- name: Replace the persistent infra VM with the candidate
run: python3 -c 'from bot_bottle.backend.firecracker import infra_vm; infra_vm.stop()'
# No dev-requirements install: `coverage` is already provided by the
# self-hosted runner's Nix python env, and that env has no `pip`
# module to install into anyway.
- name: Run integration tests (firecracker) with coverage
env:
BOT_BOTTLE_BACKEND: firecracker
BOT_BOTTLE_INFRA_ARTIFACT_DIR: ${{ github.workspace }}/infra-candidate
COVERAGE_FILE: ${{ github.workspace }}/.coverage.firecracker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
- name: Stage firecracker coverage for upload
run: cp .coverage.firecracker coverage-firecracker.dat
- name: Upload firecracker coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-firecracker
path: coverage-firecracker.dat
- name: Upload tested rootfs
uses: actions/upload-artifact@v3
with:
name: infra-candidate
path: infra-candidate/
- name: Upload dropbear for publish verification
uses: actions/upload-artifact@v3
with:
name: firecracker-inputs
path: /var/cache/bot-bottle-fc/dropbear
# Integration tests against the macOS Apple Container backend. Runs on a
# self-hosted macOS runner (label `macos`) registered in HOST mode — Apple
# Container needs the host `container` CLI + virtualization framework and
# cannot run inside a Linux container, so this cannot reuse the KVM runner.
#
# Advisory only: workflow_dispatch (manual) exclusively — never push or
# pull_request. A single non-redundant laptop that sleeps/roams must not run
# unattended on every push to main, let alone block a PR merge, so this job is
# deliberately NOT in the `coverage` job's `needs` and its coverage never
# feeds the diff-coverage gate. Dispatch-only also means no fork PR (or any
# push) ever executes on the host-mode runner.
#
# The infra container is a singleton (`bot-bottle-mac-infra`); the
# `concurrency` group serializes runs so two never collide on it (#425), and
# the always-run teardown removes it so a crashed run can't wedge the next.
#
# Runner prerequisites (provision once; see README "macOS Apple Container"):
# the `container` CLI on PATH with `container system status` running, and a
# Python >=3.11 with `coverage` importable on the launchd service PATH.
integration-macos:
runs-on: [self-hosted, macos]
if: github.event_name == 'workflow_dispatch'
concurrency:
group: integration-macos-infra
cancel-in-progress: false
steps:
- name: Checkout
uses: actions/checkout@v4
# Fail loudly if the backend this job promises isn't actually usable,
# rather than letting every test silently `unittest.skip` and the job go
# green on zero coverage. `backend status` exits non-zero (and prints the
# per-check summary) when the `container` CLI or its system service is
# missing — the same readiness check the skip guards gate on.
- name: Preflight — Apple Container backend is ready
run: |
command -v container >/dev/null || {
echo "container CLI not on PATH — provision the runner (README: macOS Apple Container)"; exit 1; }
container system status || {
echo "container system service not running — run 'container system start'"; exit 1; }
python3 cli.py backend status --backend=macos-container
# `coverage` comes from the runner's provisioned Python (no pip install
# into the host interpreter). Advisory job: report coverage in-line for
# visibility but don't upload — it never feeds the combined gate.
- name: Run integration tests (macos-container) with coverage
env:
BOT_BOTTLE_BACKEND: macos-container
COVERAGE_FILE: ${{ github.workspace }}/.coverage.macos
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
- name: Report macos coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.macos
run: python3 -m coverage report -m
# On failure, capture the infra containers' state and logs BEFORE the
# teardown below removes them — otherwise a control-plane crash is
# undiagnosable from CI, since `stop()` deletes the orchestrator (and its
# logs) on every run. Best-effort: never let the diagnostics themselves
# fail the job, and keep going if a container is already gone.
- name: Dump infra diagnostics (on failure)
if: failure()
run: |
set +e
echo "=== containers ==="
container ls -a | grep bot-bottle-mac || echo "(no bot-bottle-mac containers)"
echo "=== networks ==="
container network ls | grep bot-bottle-mac || echo "(no bot-bottle-mac networks)"
for c in bot-bottle-mac-orchestrator bot-bottle-mac-infra; do
echo "=== inspect $c ==="
container inspect "$c" || echo "($c not found)"
echo "=== logs $c ==="
container logs "$c" || echo "($c logs unavailable)"
done
exit 0
# Remove the singleton infra container so a crashed or cancelled run
# cannot leave `bot-bottle-mac-infra` wedged for the next job.
- name: Teardown infra singleton
if: always()
run: python3 -c 'from bot_bottle.backend.macos_container.infra import MacosInfraService; MacosInfraService().stop()'
# Combined coverage gate: aggregates .coverage.* artifacts uploaded by each
# test job, then runs the diff-coverage gate (new/changed lines >= 90%).
#
# Runs on ubuntu-latest — no KVM needed, no test reruns. Coverage files use
# relative_files = True (.coveragerc) so they combine cleanly across runners.
# Each test job sets COVERAGE_FILE to an absolute path so coverage.py writes
# to a known location that upload-artifact can find regardless of runner env.
#
coverage:
needs: [unit, integration-docker, integration-firecracker]
timeout-minutes: 15
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
- name: Download unit coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-unit
path: ${{ github.workspace }}
- name: Download docker coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-docker
path: ${{ github.workspace }}
- name: Download firecracker coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-firecracker
path: ${{ github.workspace }}
# Rename the non-dot upload names back to the .coverage.* files that
# `coverage combine` discovers (see the staging steps in each test job).
- name: Reassemble coverage data files
run: |
mv coverage-unit.dat .coverage.unit
mv coverage-docker.dat .coverage.docker
mv coverage-firecracker.dat .coverage.firecracker
- name: Combined coverage (unit + integration, incl. firecracker)
run: PYTHON=python3 bash scripts/coverage.sh aggregate critical
- name: Diff-coverage gate (changed lines >= 90%)
run: |
git fetch --no-tags origin main:refs/remotes/origin/main
python3 scripts/diff_coverage.py --base origin/main --min 90
publish-infra:
needs:
- unit
- integration-docker
- integration-firecracker
- integration-macos
- coverage
runs-on: ubuntu-latest
steps:
- name: Checkout the tested revision
uses: actions/checkout@v4
- name: Download the tested rootfs
uses: actions/download-artifact@v3
with:
name: infra-candidate
path: infra-candidate
# publish_infra re-derives the version from the checkout to confirm the
# bundle matches before uploading, and the version hashes the dropbear
# bytes. Download the same dropbear integration-firecracker used.
- name: Download the staged dropbear
uses: actions/download-artifact@v3
with:
name: firecracker-inputs
path: firecracker-inputs
- name: Publish the tested candidate
env:
BOT_BOTTLE_INFRA_ARTIFACT_TOKEN: ${{ secrets.BOT_BOTTLE_INFRA_ARTIFACT_TOKEN }}
BOT_BOTTLE_FC_DROPBEAR: ${{ github.workspace }}/firecracker-inputs/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --publish-dir infra-candidate
+7 -259
View File
@@ -1,21 +1,6 @@
# Run the project's test suite when package or runtime inputs change on a PR
# or on push to main.
#
# The suite uses stdlib `unittest` discovery — no external Python
# dependencies are required to execute it. Tests are split by directory:
#
# tests/unit/ — pure unit tests; always run
# tests/integration/ — need a reachable backend; skip cleanly when
# the backend isn't available on the runner
# tests/canaries/ — upstream regression canaries; run on a separate
# schedule (see canaries.yml), not here
#
# Each test job runs once under coverage and uploads a small .coverage.*
# artifact. The `coverage` job combines them — no test reruns, no KVM
# dependency on that job. For main-branch pushes only, the tested rootfs
# and matching dropbear are uploaded so `publish-infra` can publish the
# byte-identical artifact that was tested. PRs avoid the ~194 MB rootfs
# transfer entirely.
# Run the automated test gate when package or runtime inputs change on a PR
# or on push to main. Privileged self-hosted backends live in the manually
# dispatched pre-release-test workflow.
name: test
@@ -37,6 +22,7 @@ on:
- 'requirements-dev.txt'
- '.coveragerc'
- '.dockerignore'
- '.gitea/workflows/test.yml'
pull_request:
paths:
- 'bot_bottle/**'
@@ -52,7 +38,7 @@ on:
- 'requirements-dev.txt'
- '.coveragerc'
- '.dockerignore'
workflow_dispatch:
- '.gitea/workflows/test.yml'
jobs:
unit:
@@ -61,11 +47,6 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
# No actions/setup-python: the runner image already ships Python 3.12,
# and older act_runner engines mishandle setup-python's PATH (coverage
# lands in one interpreter, `python3` resolves to another). Install
# straight into the ephemeral job container's system Python —
# --break-system-packages is safe because the container is disposable.
- name: Install dev requirements
run: python3 -m pip install --break-system-packages -r requirements-dev.txt
@@ -79,10 +60,6 @@ jobs:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.unit
run: python3 -m coverage report -m
# upload-artifact@v3's glob skips dotfiles, so a bare `.coverage.unit`
# silently uploads nothing ("No files were found"). Stage it under a
# non-dot name; the coverage job renames it back before `coverage
# combine`. `cp` also fails loudly if coverage never wrote the file.
- name: Stage unit coverage for upload
run: cp .coverage.unit coverage-unit.dat
@@ -98,17 +75,9 @@ jobs:
- name: Checkout
uses: actions/checkout@v4
# No actions/setup-python (see the note in the `unit` job); the
# container's system Python 3.12 runs the stdlib test suite directly.
- name: Install coverage
run: python3 -m pip install --break-system-packages coverage
# Fail loudly if the backend this job promises isn't actually usable,
# rather than letting every test silently `unittest.skip` and the job
# go green on zero coverage. `backend status` prints a clear per-check
# summary (docker on PATH, daemon reachable) and exits non-zero when a
# prerequisite is missing — the same readiness check the skip guards
# gate on via `has_backend`.
- name: Preflight — Docker backend is ready
run: |
python3 --version
@@ -120,7 +89,6 @@ jobs:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.docker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
# Non-dot name so upload-artifact's dotfile-skipping glob picks it up.
- name: Stage docker coverage for upload
run: cp .coverage.docker coverage-docker.dat
@@ -130,190 +98,10 @@ jobs:
name: coverage-docker
path: coverage-docker.dat
# Integration tests against the Firecracker backend. Runs on a self-hosted
# KVM runner (label `kvm`) where /dev/kvm and the TAP/nft pool are available.
#
# Restricted to same-repo PRs, push to main, and workflow_dispatch — fork
# PRs don't execute untrusted code on the privileged runner.
#
# Runner prerequisites (provision once; see README "Firecracker on Linux"):
# `firecracker` on PATH, `/dev/kvm` accessible, cached kernel +
# static dropbear at /var/cache/bot-bottle-fc/dropbear, and the pool as a
# persistent systemd unit.
#
# The infra candidate is built here directly (no artifact download) to
# eliminate the ~70 s ubuntu-latest upload + ~83 s combined download that
# the old build-infra → integration-firecracker + coverage chain incurred.
# For main-branch pushes the tested rootfs and matching dropbear are
# uploaded so publish-infra can publish the byte-identical artifact; PRs
# skip those uploads entirely.
integration-firecracker:
runs-on: [self-hosted, kvm]
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository)
steps:
- name: Checkout
uses: actions/checkout@v4
- name: Preflight — Firecracker host is ready
run: |
command -v firecracker >/dev/null || {
echo "firecracker not on PATH — provision the runner (README: Firecracker on Linux)"; exit 1; }
test -e /dev/kvm || { echo "/dev/kvm missing — KVM not available on this runner"; exit 1; }
# `backend status` exits non-zero unless the TAP pool is up + no
# range overlap; it prints the exact `backend setup` fix.
python3 cli.py backend status --backend=firecracker
- name: Build infra candidate from this checkout
env:
BOT_BOTTLE_FC_DROPBEAR: /var/cache/bot-bottle-fc/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --output infra-candidate --reuse-published
- name: Replace the persistent infra VM with the candidate
run: python3 -c 'from bot_bottle.backend.firecracker import infra_vm; infra_vm.stop()'
# No dev-requirements install: `coverage` is already provided by the
# self-hosted runner's Nix python env, and that env has no `pip`
# module to install into anyway.
- name: Run integration tests (firecracker) with coverage
env:
BOT_BOTTLE_BACKEND: firecracker
BOT_BOTTLE_INFRA_ARTIFACT_DIR: ${{ github.workspace }}/infra-candidate
COVERAGE_FILE: ${{ github.workspace }}/.coverage.firecracker
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
# Non-dot name so upload-artifact's dotfile-skipping glob picks it up.
- name: Stage firecracker coverage for upload
run: cp .coverage.firecracker coverage-firecracker.dat
- name: Upload firecracker coverage artifact
uses: actions/upload-artifact@v3
with:
name: coverage-firecracker
path: coverage-firecracker.dat
# Only upload the large rootfs artifact on main-branch pushes;
# PRs avoid the ~194 MB transfer. publish-infra only runs on main
# and downloads these to publish the byte-identical tested rootfs.
- name: Upload tested rootfs (main branch only)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v3
with:
name: infra-candidate
path: infra-candidate/
- name: Upload dropbear for publish verification (main branch only)
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v3
with:
name: firecracker-inputs
path: /var/cache/bot-bottle-fc/dropbear
# Integration tests against the macOS Apple Container backend. Runs on a
# self-hosted macOS runner (label `macos`) registered in HOST mode — Apple
# Container needs the host `container` CLI + virtualization framework and
# cannot run inside a Linux container, so this cannot reuse the KVM runner.
#
# Advisory only: workflow_dispatch (manual) exclusively — never push or
# pull_request. A single non-redundant laptop that sleeps/roams must not run
# unattended on every push to main, let alone block a PR merge, so this job is
# deliberately NOT in the `coverage` job's `needs` and its coverage never
# feeds the diff-coverage gate. Dispatch-only also means no fork PR (or any
# push) ever executes on the host-mode runner.
#
# The infra container is a singleton (`bot-bottle-mac-infra`); the
# `concurrency` group serializes runs so two never collide on it (#425), and
# the always-run teardown removes it so a crashed run can't wedge the next.
#
# Runner prerequisites (provision once; see README "macOS Apple Container"):
# the `container` CLI on PATH with `container system status` running, and a
# Python >=3.11 with `coverage` importable on the launchd service PATH.
integration-macos:
runs-on: [self-hosted, macos]
if: github.event_name == 'workflow_dispatch'
concurrency:
group: integration-macos-infra
cancel-in-progress: false
steps:
- name: Checkout
uses: actions/checkout@v4
# Fail loudly if the backend this job promises isn't actually usable,
# rather than letting every test silently `unittest.skip` and the job go
# green on zero coverage. `backend status` exits non-zero (and prints the
# per-check summary) when the `container` CLI or its system service is
# missing — the same readiness check the skip guards gate on.
- name: Preflight — Apple Container backend is ready
run: |
command -v container >/dev/null || {
echo "container CLI not on PATH — provision the runner (README: macOS Apple Container)"; exit 1; }
container system status || {
echo "container system service not running — run 'container system start'"; exit 1; }
python3 cli.py backend status --backend=macos-container
# `coverage` comes from the runner's provisioned Python (no pip install
# into the host interpreter). Advisory job: report coverage in-line for
# visibility but don't upload — it never feeds the combined gate.
- name: Run integration tests (macos-container) with coverage
env:
BOT_BOTTLE_BACKEND: macos-container
COVERAGE_FILE: ${{ github.workspace }}/.coverage.macos
run: python3 -m coverage run -m unittest discover -t . -s tests/integration -v
- name: Report macos coverage
env:
COVERAGE_FILE: ${{ github.workspace }}/.coverage.macos
run: python3 -m coverage report -m
# On failure, capture the infra containers' state and logs BEFORE the
# teardown below removes them — otherwise a control-plane crash is
# undiagnosable from CI, since `stop()` deletes the orchestrator (and its
# logs) on every run. Best-effort: never let the diagnostics themselves
# fail the job, and keep going if a container is already gone.
- name: Dump infra diagnostics (on failure)
if: failure()
run: |
set +e
echo "=== containers ==="
container ls -a | grep bot-bottle-mac || echo "(no bot-bottle-mac containers)"
echo "=== networks ==="
container network ls | grep bot-bottle-mac || echo "(no bot-bottle-mac networks)"
for c in bot-bottle-mac-orchestrator bot-bottle-mac-infra; do
echo "=== inspect $c ==="
container inspect "$c" || echo "($c not found)"
echo "=== logs $c ==="
container logs "$c" || echo "($c logs unavailable)"
done
exit 0
# Remove the singleton infra container so a crashed or cancelled run
# cannot leave `bot-bottle-mac-infra` wedged for the next job.
- name: Teardown infra singleton
if: always()
run: python3 -c 'from bot_bottle.backend.macos_container.infra import MacosInfraService; MacosInfraService().stop()'
# Combined coverage gate: aggregates .coverage.* artifacts uploaded by each
# test job, then runs the diff-coverage gate (new/changed lines >= 90%).
#
# Runs on ubuntu-latest — no KVM needed, no test reruns. Coverage files use
# relative_files = True (.coveragerc) so they combine cleanly across runners.
# Each test job sets COVERAGE_FILE to an absolute path so coverage.py writes
# to a known location that upload-artifact can find regardless of runner env.
#
# Restricted to the same events as integration-firecracker: it depends on
# that job's coverage artifact and skips for fork PRs alongside it.
coverage:
needs: [unit, integration-docker, integration-firecracker]
needs: [unit, integration-docker]
timeout-minutes: 15
runs-on: ubuntu-latest
if: >-
github.event_name == 'push' ||
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'pull_request' &&
github.event.pull_request.head.repo.full_name == github.repository)
steps:
- name: Checkout
uses: actions/checkout@v4
@@ -335,55 +123,15 @@ jobs:
name: coverage-docker
path: ${{ github.workspace }}
- name: Download firecracker coverage artifact
uses: actions/download-artifact@v3
with:
name: coverage-firecracker
path: ${{ github.workspace }}
# Rename the non-dot upload names back to the .coverage.* files that
# `coverage combine` discovers (see the staging steps in each test job).
- name: Reassemble coverage data files
run: |
mv coverage-unit.dat .coverage.unit
mv coverage-docker.dat .coverage.docker
mv coverage-firecracker.dat .coverage.firecracker
- name: Combined coverage (unit + integration, incl. firecracker)
- name: Combined coverage (unit + docker integration)
run: PYTHON=python3 bash scripts/coverage.sh aggregate critical
- name: Diff-coverage gate (changed lines >= 90%)
run: |
git fetch --no-tags origin main:refs/remotes/origin/main
python3 scripts/diff_coverage.py --base origin/main --min 90
publish-infra:
needs: [unit, integration-docker, integration-firecracker, coverage]
runs-on: ubuntu-latest
if: github.event_name == 'push' && github.ref == 'refs/heads/main'
steps:
- name: Checkout the tested revision
uses: actions/checkout@v4
- name: Download the tested rootfs
uses: actions/download-artifact@v3
with:
name: infra-candidate
path: infra-candidate
# publish_infra re-derives the version from the checkout to confirm the
# bundle matches before uploading, and the version hashes the dropbear
# bytes. Download the SAME dropbear integration-firecracker used, or
# the recheck computes a "<missing>"-dropbear version and rejects the
# candidate.
- name: Download the staged dropbear (matches build's version)
uses: actions/download-artifact@v3
with:
name: firecracker-inputs
path: firecracker-inputs
- name: Publish the tested candidate
env:
BOT_BOTTLE_INFRA_ARTIFACT_TOKEN: ${{ secrets.BOT_BOTTLE_INFRA_ARTIFACT_TOKEN }}
BOT_BOTTLE_FC_DROPBEAR: ${{ github.workspace }}/firecracker-inputs/dropbear
run: python3 -m bot_bottle.backend.firecracker.publish_infra --publish-dir infra-candidate
+3
View File
@@ -17,6 +17,9 @@ __pycache__/
*.py[cod]
*$py.class
*.egg-info/
# setuptools/build_meta output (wheels, sdists, build tree)
/build/
/dist/
.venv/
venv/
.pytest_cache/
+4 -4
View File
@@ -44,10 +44,10 @@ backend remains available with `BOT_BOTTLE_BACKEND=docker` or
- Three kinds of doc, each with its own conventions in-folder; see
`docs/README.md` for when to write which:
- **PRDs** (`docs/prds/`) — one feature per file. While a PR is open
the file is named `prd-new-<kebab>.md`; CI assigns a sequential
number on merge to `main` and renames it. A `Status:` line tracks
lifecycle: Draft → Active (shipped to `main`) →
- **PRDs** (`docs/prds/`) — one feature per file. A draft may initially
use `prd-new-<kebab>.md`, but its author must assign the next
sequential number before merge; CI rejects unnumbered PRDs. A
`Status:` line tracks lifecycle: Draft → Active (shipped to `main`) →
Superseded/Retargeted. Format in `docs/prds/README.md`.
- **Research notes** (`docs/research/`) — opinionated investigations;
unnumbered kebab-case, freeform and verdict-first. See
+8
View File
@@ -0,0 +1,8 @@
# Root-level build resources copied into bot_bottle/_resources/ at build time
# (see setup.py). Included in the sdist so `pip install` from an sdist can
# still bundle them into the wheel.
include Dockerfile.gateway
include Dockerfile.orchestrator
include Dockerfile.orchestrator.fc
include nix/firecracker-netpool.nix
include scripts/firecracker-netpool.sh
+1 -1
View File
@@ -5,7 +5,7 @@
# bot-bottle
[![test](https://gitea.dideric.is/didericis/bot-bottle/actions/workflows/test.yml/badge.svg?branch=main)](https://gitea.dideric.is/didericis/bot-bottle/actions?workflow=test.yml)
[![coverage](https://img.shields.io/badge/coverage-84%25-brightgreen)](https://coverage.readthedocs.io/)
[![coverage](https://img.shields.io/badge/coverage-83%25-brightgreen)](https://coverage.readthedocs.io/)
[![core coverage](https://img.shields.io/badge/core%20coverage-94%25-brightgreen)](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/docs/decisions/0004-coverage-policy.md)
**Problem:** Developer wants to run a coding agent without supervision, but they don't want a prompt injected or misbehaving agent wrecking their environment or exfiltrating sensitive data.
+8 -4
View File
@@ -10,9 +10,10 @@ from ...paths import (
ORCHESTRATOR_AUTH_JWT_ENV,
host_gateway_ca_dir,
)
from ... import resources
from ...gateway import (
Gateway, GatewayTransport, GATEWAY_IMAGE, GATEWAY_NAME, GATEWAY_NETWORK,
GATEWAY_DOCKERFILE, REPO_ROOT, GATEWAY_LABEL, MITMPROXY_HOME,
GATEWAY_DOCKERFILE, GATEWAY_LABEL, MITMPROXY_HOME,
DEFAULT_CA_TIMEOUT_SECONDS, CA_POLL_SECONDS, GATEWAY_CA_CERT, GatewayError
)
@@ -50,7 +51,9 @@ class DockerGateway(Gateway):
# `address` / `stop` work on an already-running gateway without it.
self._orchestrator_url = ""
self._gateway_token = ""
self._build_context = build_context or REPO_ROOT
# Resolved lazily in ensure_built() so merely constructing a gateway to
# read its CA never stages a build root from an installed wheel.
self._build_context = build_context
self._dockerfile = dockerfile
# Ports published on the host (0.0.0.0). Used by the Firecracker
# backend's dev-harness gateway so VMs can reach it via their TAP link;
@@ -72,9 +75,10 @@ class DockerGateway(Gateway):
forces a full rebuild (parity with `start --no-cache`)."""
if self._dockerfile is None:
return
context = self._build_context or resources.build_root()
argv = ["docker", "build", "-t", self.image_ref,
"-f", str(self._build_context / self._dockerfile),
str(self._build_context)]
"-f", str(context / self._dockerfile),
str(context)]
if os.environ.get("BOT_BOTTLE_NO_CACHE"):
argv.insert(2, "--no-cache")
proc = run_docker(argv)
+5 -4
View File
@@ -34,6 +34,7 @@ from .orchestrator import (
ORCHESTRATOR_NETWORK,
)
from ...paths import bot_bottle_root
from ... import resources
from ...gateway import (
GATEWAY_IMAGE,
GATEWAY_NAME,
@@ -50,8 +51,6 @@ from ...orchestrator.lifecycle import (
# the pair's public identity.
INFRA_NAME = GATEWAY_NAME # the container agents attribute against is the gateway
_REPO_ROOT = Path(__file__).resolve().parents[3]
class DockerInfraService(InfraService):
"""Composes the per-host control plane + gateway as two containers.
@@ -68,7 +67,7 @@ class DockerInfraService(InfraService):
control_network: str = ORCHESTRATOR_NETWORK,
orchestrator_image: str = ORCHESTRATOR_IMAGE,
gateway_image: str = GATEWAY_IMAGE,
repo_root: Path = _REPO_ROOT,
repo_root: Path | None = None,
host_root: Path | None = None,
orchestrator_name: str = ORCHESTRATOR_NAME,
orchestrator_label: str = ORCHESTRATOR_LABEL,
@@ -79,7 +78,9 @@ class DockerInfraService(InfraService):
self.control_network = control_network
self.orchestrator_image = orchestrator_image
self.gateway_image = gateway_image
self._repo_root = repo_root
# Build context / bind-mount source: the repo root in a checkout, a
# staged copy from the installed wheel otherwise (bot_bottle.resources).
self._repo_root = repo_root if repo_root is not None else resources.build_root()
self._host_root = host_root or bot_bottle_root()
self._orchestrator_name = orchestrator_name
self._orchestrator_label = orchestrator_label
+2 -6
View File
@@ -33,7 +33,6 @@ from __future__ import annotations
import dataclasses
import os
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...agent_provider import runtime_for
@@ -65,10 +64,7 @@ from ...orchestrator.store.config_store import resolve_teardown_timeout
from .consolidated_launch import launch_consolidated, deprovision_consolidated
from .infra import INFRA_NAME
from .gateway import DockerGateway
# Where the repo root lives, for `docker build` context. Computed once.
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
from ... import resources
def build_or_load_images(plan: DockerBottlePlan) -> BottleImages:
@@ -88,7 +84,7 @@ def build_or_load_images(plan: DockerBottlePlan) -> BottleImages:
)
info(f"using cached agent image {plan.image!r}")
return BottleImages(agent=plan.image)
docker_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
docker_mod.build_image(plan.image, str(resources.build_root()), dockerfile=plan.dockerfile_path)
docker_mod.verify_agent_image(
plan.image, runtime_for(plan.agent_provider_template).smoke_test,
)
+5 -4
View File
@@ -15,6 +15,7 @@ import time
from pathlib import Path
from ... import log
from ... import resources
from .util import run_docker
from ...paths import (
ORCHESTRATOR_TOKEN_ENV,
@@ -55,8 +56,6 @@ _ROOT_IN_CONTAINER = "/bot-bottle-root"
_HEALTH_POLL_SECONDS = 0.25
_REPO_ROOT = Path(__file__).resolve().parents[3]
class DockerOrchestrator(Orchestrator):
"""The control plane as a single fixed-name container. `ensure_built` builds
@@ -71,7 +70,7 @@ class DockerOrchestrator(Orchestrator):
label: str = ORCHESTRATOR_LABEL,
port: int = DEFAULT_PORT,
control_network: str = ORCHESTRATOR_NETWORK,
repo_root: Path = _REPO_ROOT,
repo_root: Path | None = None,
host_root: Path | None = None,
dockerfile: str | None = ORCHESTRATOR_DOCKERFILE,
) -> None:
@@ -80,7 +79,9 @@ class DockerOrchestrator(Orchestrator):
self.label = label
self.port = port
self.control_network = control_network
self._repo_root = repo_root
# Build context / bind-mount source: the repo root in a checkout, a
# staged copy from the installed wheel otherwise (bot_bottle.resources).
self._repo_root = repo_root if repo_root is not None else resources.build_root()
self._host_root = host_root or bot_bottle_root()
self._dockerfile = dockerfile
@@ -37,6 +37,7 @@ import urllib.error
import urllib.request
from pathlib import Path
from ... import resources
from ...log import die, info
from . import util
@@ -44,8 +45,6 @@ from . import util
# scheme can't collide with a cached/published artifact of the old one.
_ARTIFACT_FORMAT = "1"
_REPO_ROOT = Path(__file__).resolve().parents[3]
# The two per-plane infra VM roles. Each publishes/pulls its own rootfs artifact
# from its own generic package; the Dockerfiles baked into each differ (only the
# orchestrator rootfs carries buildah), so the versions are hashed separately.
@@ -74,7 +73,7 @@ def local_build_requested() -> bool:
def infra_artifact_version(
init_script: str, role: str, *, repo_root: Path = _REPO_ROOT,
init_script: str, role: str, *, repo_root: Path | None = None,
) -> str:
"""Content hash (16 hex) of everything baked into `role`'s infra rootfs: the
whole shipped `bot_bottle` package, that role's Dockerfiles, and its guest
@@ -89,6 +88,8 @@ def infra_artifact_version(
version or a launch host could boot a stale rootfs whose code differs from
its checkout. `__pycache__`/`.pyc` are the only exclusions build artifacts,
never copied."""
if repo_root is None:
repo_root = resources.build_root()
h = hashlib.sha256()
h.update(f"format={_ARTIFACT_FORMAT}\nrole={role}\n".encode())
pkg = repo_root / "bot_bottle"
+5 -4
View File
@@ -42,6 +42,7 @@ from dataclasses import dataclass
from pathlib import Path
from typing import Generator
from ... import resources
from ...log import die, info
from ..docker import util as docker_mod
from . import firecracker_vm, infra_artifact, netpool, util
@@ -65,7 +66,6 @@ _GUEST_GATEWAY_JWT_PATH = "/var/lib/bot-bottle/gateway-jwt"
_GATEWAY_IMAGE = "bot-bottle-gateway:latest"
_ORCHESTRATOR_IMAGE = "bot-bottle-orchestrator:latest"
_ORCHESTRATOR_FC_IMAGE = "bot-bottle-orchestrator-fc:latest"
_REPO_ROOT = Path(__file__).resolve().parents[3]
# Per-role rootfs source image + the extra free space `mke2fs` leaves for the
# guest to grow into. The orchestrator keeps buildah's large build slack; the
@@ -130,12 +130,13 @@ def build_infra_images_with_docker() -> None:
orchestrator + buildah). The gateway VM boots the gateway image directly.
The launch host uses this only in `BOT_BOTTLE_INFRA_BUILD=local` mode;
`publish_infra` uses it off-host to produce the published artifacts."""
root = str(resources.build_root())
docker_mod.build_image(
_ORCHESTRATOR_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator")
_ORCHESTRATOR_IMAGE, root, dockerfile="Dockerfile.orchestrator")
docker_mod.build_image(
_GATEWAY_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.gateway")
_GATEWAY_IMAGE, root, dockerfile="Dockerfile.gateway")
docker_mod.build_image(
_ORCHESTRATOR_FC_IMAGE, str(_REPO_ROOT), dockerfile="Dockerfile.orchestrator.fc")
_ORCHESTRATOR_FC_IMAGE, root, dockerfile="Dockerfile.orchestrator.fc")
def build_rootfs_dir(role: str) -> Path:
+5 -4
View File
@@ -20,6 +20,7 @@ import subprocess
import sys
from pathlib import Path
from ... import resources
from . import netpool
from . import util
@@ -42,13 +43,13 @@ def _has_systemd() -> bool:
def _module_path() -> str:
"""Absolute path to the importable NixOS module in this checkout."""
return str(Path(__file__).resolve().parents[3] / "nix" / "firecracker-netpool.nix")
"""Absolute path to the importable NixOS module (checkout or wheel)."""
return str(resources.nix_netpool_module())
def _script_path() -> str:
"""Absolute path to the bundled bring-up script in this checkout."""
return str(Path(__file__).resolve().parents[3] / "scripts" / "firecracker-netpool.sh")
"""Absolute path to the bundled bring-up script (checkout or wheel)."""
return str(resources.netpool_script())
def _print_prereqs() -> None:
@@ -28,6 +28,7 @@ from ...paths import (
ORCHESTRATOR_AUTH_JWT_ENV,
host_gateway_ca_dir,
)
from ... import resources
from .. import util as backend_util
from . import util as container_mod
@@ -52,8 +53,6 @@ GATEWAY_DAEMONS = "egress,git-http,supervise"
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
_REPO_ROOT = Path(__file__).resolve().parents[3]
def ensure_networks(
network: str = GATEWAY_NETWORK,
@@ -84,14 +83,16 @@ class MacosGateway(Gateway):
network: str = GATEWAY_NETWORK,
egress_network: str = GATEWAY_EGRESS_NETWORK,
control_network: str = CONTROL_NETWORK,
repo_root: Path = _REPO_ROOT,
repo_root: Path | None = None,
) -> None:
self.image_ref = image_ref
self.name = name
self.network = network
self.egress_network = egress_network
self.control_network = control_network
self._repo_root = repo_root
# Build context: the repo root in a checkout, a staged copy from the
# installed wheel otherwise (bot_bottle.resources).
self._repo_root = repo_root if repo_root is not None else resources.build_root()
# Set by `connect_to_orchestrator`: the URL the daemons resolve policy
# against + the pre-minted `gateway` token they present. The gateway
# never mints, so it never holds the signing key (#469).
+5 -4
View File
@@ -24,6 +24,7 @@ from __future__ import annotations
from pathlib import Path
from ... import resources
from ...orchestrator.lifecycle import (
DEFAULT_PORT,
DEFAULT_STARTUP_TIMEOUT_SECONDS,
@@ -53,8 +54,6 @@ from .orchestrator import (
# still import it (probe / reprovision attribute against the gateway).
INFRA_NAME = GATEWAY_NAME
_REPO_ROOT = Path(__file__).resolve().parents[3]
class MacosInfraService(InfraService):
"""Composes the per-host orchestrator + gateway containers. Callers use
@@ -70,7 +69,7 @@ class MacosInfraService(InfraService):
control_network: str = CONTROL_NETWORK,
gateway_image: str = GATEWAY_IMAGE,
orchestrator_image: str = ORCHESTRATOR_IMAGE,
repo_root: Path = _REPO_ROOT,
repo_root: Path | None = None,
orchestrator_name: str = ORCHESTRATOR_NAME,
gateway_name: str = INFRA_NAME,
db_volume: str = ORCHESTRATOR_DB_VOLUME,
@@ -81,7 +80,9 @@ class MacosInfraService(InfraService):
self.control_network = control_network
self.gateway_image = gateway_image
self.orchestrator_image = orchestrator_image
self._repo_root = repo_root
# Build context / bind-mount source: the repo root in a checkout, a
# staged copy from the installed wheel otherwise (bot_bottle.resources).
self._repo_root = repo_root if repo_root is not None else resources.build_root()
self._orchestrator_name = orchestrator_name
self._gateway_name = gateway_name
self._db_volume = db_volume
+2 -3
View File
@@ -36,7 +36,6 @@ import dataclasses
import os
import subprocess
from contextlib import ExitStack, contextmanager
from pathlib import Path
from typing import Callable, Generator
from ...bottle_state import (
@@ -49,6 +48,7 @@ from ...git_gate import GitGate
from ...gateway.git_gate.http_backend import DEFAULT_PORT as _GIT_HTTP_PORT
from ...image_cache import check_stale
from ...log import die, info, warn
from ... import resources
from .. import BottleImages
from ...supervisor.types import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
@@ -71,7 +71,6 @@ from .consolidated_launch import (
deprovision_consolidated,
)
_REPO_DIR = str(Path(__file__).resolve().parent.parent.parent.parent)
_AGENT_SLEEP_SECONDS = "2147483647"
@@ -94,7 +93,7 @@ def _agent_image(plan: MacosContainerBottlePlan) -> str:
)
info(f"using cached agent image {plan.image!r}")
return plan.image
container_mod.build_image(plan.image, _REPO_DIR, dockerfile=plan.dockerfile_path)
container_mod.build_image(plan.image, str(resources.build_root()), dockerfile=plan.dockerfile_path)
return plan.image
@@ -18,6 +18,7 @@ import urllib.request
from pathlib import Path
from ... import log
from ... import resources
from ...paths import ORCHESTRATOR_TOKEN_ENV
from ...orchestrator.lifecycle import (
DEFAULT_HEALTH_TIMEOUT_SECONDS,
@@ -45,7 +46,6 @@ _DB_ROOT_IN_CONTAINER = "/var/lib/bot-bottle"
_SRC_IN_CONTAINER = "/bot-bottle-src"
_HEALTH_POLL_SECONDS = 0.25
_REPO_ROOT = Path(__file__).resolve().parents[3]
class MacosOrchestrator(Orchestrator):
@@ -61,7 +61,7 @@ class MacosOrchestrator(Orchestrator):
label: str = ORCHESTRATOR_LABEL,
port: int = DEFAULT_PORT,
control_network: str = CONTROL_NETWORK,
repo_root: Path = _REPO_ROOT,
repo_root: Path | None = None,
db_volume: str = ORCHESTRATOR_DB_VOLUME,
) -> None:
self.image_ref = image_ref
@@ -69,7 +69,9 @@ class MacosOrchestrator(Orchestrator):
self.label = label
self.port = port
self.control_network = control_network
self._repo_root = repo_root
# Build context / bind-mount source: the repo root in a checkout, a
# staged copy from the installed wheel otherwise (bot_bottle.resources).
self._repo_root = repo_root if repo_root is not None else resources.build_root()
self._db_volume = db_volume
def url(self) -> str:
+2 -1
View File
@@ -21,6 +21,7 @@ _HANDLERS: dict[str, str] = {
"backend": "backend:cmd_backend",
"cleanup": "cleanup:cmd_cleanup",
"commit": "commit:cmd_commit",
"doctor": "doctor:cmd_doctor",
"edit": "edit:cmd_edit",
"help": "help:cmd_help",
"init": "init:cmd_init",
@@ -53,6 +54,6 @@ COMMANDS = {name: _lazy(spec) for name, spec in _HANDLERS.items()}
# gating it on the schema breaks preflight on a fresh CI runner where stdin
# isn't a TTY and the migration prompt can't be answered. `help` and `login`
# likewise never touch the store.
NO_MIGRATION_COMMANDS = frozenset({"backend", "help", "login"})
NO_MIGRATION_COMMANDS = frozenset({"backend", "doctor", "help", "login"})
__all__ = ["COMMANDS", "NO_MIGRATION_COMMANDS"]
+88
View File
@@ -0,0 +1,88 @@
"""`doctor` CLI command — validate host prerequisites for running
bot-bottle and report what's ready.
Fails (non-zero exit) only on the two hard requirements: a new-enough
Python and at least one backend that is *ready* (passes its full status
checks, so `start` can actually work). The config directory is a soft
check `install.sh` creates it, but a missing one only warrants a note,
not a failure, since `start` provisions what it needs on first run.
"""
from __future__ import annotations
import argparse
import sys
from pathlib import Path
from ...backend import is_backend_ready, known_backend_names
from ..constants import PROG
MIN_PYTHON = (3, 11)
CONFIG_DIR = ".bot-bottle"
def _ok(label: str, detail: str) -> None:
print(f"ok: {label}: {detail}")
def _warn(label: str, detail: str) -> None:
print(f"warn: {label}: {detail}")
def _fail(label: str, detail: str) -> None:
print(f"fail: {label}: {detail}")
def _check_python() -> bool:
v = sys.version_info
detail = f"{v.major}.{v.minor}.{v.micro}"
if (v.major, v.minor) >= MIN_PYTHON:
_ok("python", detail)
return True
_fail("python", f"{detail}; need {MIN_PYTHON[0]}.{MIN_PYTHON[1]} or newer")
return False
def _check_backends() -> bool:
"""At least one backend must be *ready* to run a bottle — i.e. pass its
full status() checks (daemon reachable, network pool present, KVM usable),
not merely have a binary on PATH. A binary-only check would report `ok`
on a host with a stopped Docker daemon or a half-configured Firecracker,
where `start` still can't work. Each not-ready backend prints its own
diagnostics (quiet=False) so the operator sees exactly what's missing."""
ready = []
for name in known_backend_names():
if is_backend_ready(name, quiet=False):
_ok("backend", f"{name}: ready")
ready.append(name)
else:
_warn("backend", f"{name}: not ready (see diagnostics above)")
if ready:
return True
_fail(
"backend",
"no backend is ready to run a bottle; start Docker, or finish "
"Apple Container (macOS) / Firecracker (Linux) setup",
)
return False
def _check_config_dir() -> None:
config = Path.home() / CONFIG_DIR
if config.is_dir():
_ok("config", str(config))
else:
_warn("config", f"{config} does not exist yet (created on first use)")
def cmd_doctor(argv: list[str]) -> int:
parser = argparse.ArgumentParser(
prog=f"{PROG} doctor",
description="Check host prerequisites for running bot-bottle.",
)
parser.parse_args(argv)
# Hard requirements gate the exit code; the config note is advisory.
required = [_check_python(), _check_backends()]
_check_config_dir()
return 0 if all(required) else 1
+1
View File
@@ -25,6 +25,7 @@ def cmd_help(argv: list[str] | None = None) -> int:
w(" backend set up / check / undo a backend's host prerequisites (setup|status|teardown)\n")
w(" cleanup stop and remove all active bot-bottle containers\n")
w(" commit snapshot a running bottle's container state to a Docker image\n")
w(" doctor check host prerequisites (Python, backend, config dir)\n")
w(" edit open an agent in vim for editing\n")
w(" help show this command list\n")
w(" init interactively create a new agent and add it to bot-bottle.json\n")
-1
View File
@@ -62,7 +62,6 @@ GATEWAY_CA_GLOB = "mitmproxy-ca*"
# that lands. Env override matches the backend's BOT_BOTTLE_GATEWAY_IMAGE.
GATEWAY_IMAGE = os.environ.get("BOT_BOTTLE_GATEWAY_IMAGE", "bot-bottle-gateway:latest")
GATEWAY_DOCKERFILE = "Dockerfile.gateway"
REPO_ROOT = Path(__file__).resolve().parents[2]
def rotate_gateway_ca(ca_dir: Path | None = None) -> list[Path]:
+1 -1
View File
@@ -17,7 +17,7 @@ Each queued proposal tool call:
4. On a decision within the window, returns the operator's
`{status, notes}`. On timeout, returns `status: pending` **with the
proposal id** and leaves the proposal queued the flow is
non-blocking past the grace window (PRD prd-new / issue #412).
non-blocking past the grace window (PRD 0072 / issue #412).
`check-proposal` is the non-blocking companion: given a `proposal_id`
returned by a `pending` response, it reports the current decision
@@ -113,7 +113,7 @@ _MIGRATIONS = TableMigrations(
# egress allowlist / routes / git config selected by source IP. The
# multi-tenant gateway resolves it per request via `attribute`.
"ALTER TABLE orchestrator_bottles ADD COLUMN policy TEXT NOT NULL DEFAULT ''",
# v4 — per-bottle encrypted egress secrets (PRD prd-new-secret-provider).
# v4 — per-bottle encrypted egress secrets (PRD 0080).
# One row per env-var: key (env-var name) is plaintext for auditing;
# value is the encrypted token string. The encryption key (ENV_VAR_SECRET)
# lives only in the agent's environment — a row alone cannot recover the
@@ -1,4 +1,4 @@
"""Symmetric encryption for per-bottle egress secrets (PRD prd-new-secret-provider).
"""Symmetric encryption for per-bottle egress secrets (PRD 0080).
Each agent receives a random ENV_VAR_SECRET at startup passed as an env var,
never logged or persisted. The host uses this key to encrypt each egress auth
+159
View File
@@ -0,0 +1,159 @@
"""Locate build-time resources whether bot-bottle runs from a source
checkout or an installed wheel.
The gateway / infra / orchestrator images are built from a Docker (or Apple
`container`) build context that must contain the `bot_bottle` package,
`pyproject.toml`, and the root-level Dockerfiles as siblings. In a source
checkout that context is simply the repo root, one level above the package.
An installed wheel has no repo root: the same root-level files are shipped
inside the package under ``bot_bottle/_resources/`` (see ``setup.py``), and a
repo-root-shaped build context is staged on demand into the app-data dir.
``build_root()`` is the single source of truth it returns a directory laid
out like a repo root (has ``bot_bottle/``, ``pyproject.toml``, the
Dockerfiles, ``nix/``, ``scripts/``). Every caller that needs a build
context, a Dockerfile path, the nix netpool module, or the netpool script
derives from it, so checkout and wheel installs share one downstream path.
"""
from __future__ import annotations
import fcntl
import hashlib
import os
import shutil
import tempfile
from pathlib import Path
from .paths import bot_bottle_root
_PKG = Path(__file__).resolve().parent # …/bot_bottle
_CHECKOUT_ROOT = _PKG.parent # repo root in a checkout
_BUNDLED = _PKG / "_resources" # wheel-shipped copies
# Root-level files bundled into the wheel under ``_resources/`` (paths are
# relative to the checkout root, and preserved verbatim under ``_resources/``
# and in the staged build root). ``setup.py`` copies exactly this set; keep
# the two lists in sync (``test_resources`` guards that every entry exists).
BUNDLED_RESOURCES: tuple[str, ...] = (
"pyproject.toml",
"Dockerfile.gateway",
"Dockerfile.orchestrator",
"Dockerfile.orchestrator.fc",
"nix/firecracker-netpool.nix",
"scripts/firecracker-netpool.sh",
)
# Present at a checkout root, never in a bare installed package — the cheap
# tell for which layout we're in.
_CHECKOUT_MARKER = "Dockerfile.gateway"
class ResourceError(RuntimeError):
"""Build resources are missing from the install (corrupt/partial wheel)."""
def is_source_checkout() -> bool:
"""True when running from a source tree (the root Dockerfiles sit beside
the package); False from an installed wheel."""
return (_CHECKOUT_ROOT / _CHECKOUT_MARKER).is_file()
def build_root() -> Path:
"""A directory shaped like a repo root: ``bot_bottle/``, ``pyproject.toml``,
the root Dockerfiles, ``nix/``, and ``scripts/``.
A checkout returns the repo root itself (no copying). An installed wheel
returns a staged copy under the app-data dir, materialized once and reused.
The stage is keyed by a digest of the installed package + bundled resources
(not the distribution version), so a force-reinstall of a newer commit that
keeps ``version = 0.1.0`` still rebuilds instead of reusing a stale tree."""
if is_source_checkout():
return _CHECKOUT_ROOT
return _stage_build_root()
def dockerfile(name: str) -> Path:
"""Absolute path to a root-level Dockerfile, e.g. ``Dockerfile.gateway``."""
return build_root() / name
def nix_netpool_module() -> Path:
"""Absolute path to the firecracker netpool NixOS module."""
return build_root() / "nix" / "firecracker-netpool.nix"
def netpool_script() -> Path:
"""Absolute path to the firecracker netpool bring-up script."""
return build_root() / "scripts" / "firecracker-netpool.sh"
def _content_digest() -> str:
"""A 16-hex digest of the installed package + bundled resources.
Keys the staged build root by *content*, so a force-reinstall over the same
version string (the installer defaults to a git branch + ``pipx install
--force``, and ``version`` stays ``0.1.0``) yields a different key and
re-stages, rather than reusing an old commit's tree. ``_PKG`` already
contains ``_resources``, so walking it covers both."""
h = hashlib.sha256()
for path in sorted(_PKG.rglob("*")):
if not path.is_file() or "__pycache__" in path.parts or path.suffix == ".pyc":
continue
h.update(str(path.relative_to(_PKG)).encode())
h.update(b"\0")
h.update(path.read_bytes())
return h.hexdigest()[:16]
def _stage_build_root() -> Path:
"""Materialize a repo-root-shaped build context from the installed wheel's
bundled resources, keyed by content digest. Idempotent and concurrency-safe:
a file lock serializes staging, a partial/stale tree is replaced, and the
finished tree is published with an atomic rename."""
if not _BUNDLED.is_dir():
raise ResourceError(
"bot-bottle build resources are missing from this install "
f"(expected {_BUNDLED}). Reinstall the package."
)
base = bot_bottle_root() / "build-root"
base.mkdir(parents=True, exist_ok=True)
dest = base / _content_digest()
if (dest / ".complete").is_file():
return dest
# Serialize staging across processes: a concurrent `start` after an install
# must not race on the shared tree. The lock is held only around stage +
# atomic publish; the fast path above never blocks.
with open(base / ".stage.lock", "w", encoding="utf-8") as lock:
fcntl.flock(lock, fcntl.LOCK_EX)
if (dest / ".complete").is_file(): # another process staged while we waited
return dest
# Stage into a private temp dir on the same filesystem, then publish by
# rename — never populate a shared path other processes might read.
staging = Path(tempfile.mkdtemp(prefix=".staging-", dir=base))
try:
# The package itself, minus caches and the bundled-resource copies,
# so the staged ``bot_bottle/`` matches a checkout's (keeps the
# firecracker infra-artifact hash stable across checkout and wheel).
shutil.copytree(
_PKG,
staging / "bot_bottle",
ignore=shutil.ignore_patterns("__pycache__", "*.pyc", "_resources"),
)
# The bundled root files, restored to their checkout-relative layout.
for rel in BUNDLED_RESOURCES:
dst = staging / rel
dst.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(_BUNDLED / rel, dst)
(staging / ".complete").write_text("")
# Replace any partial leftover for this digest (safe: we hold the
# lock), then publish atomically.
if dest.exists():
shutil.rmtree(dest)
os.replace(staging, dest)
staging = None # published; nothing to clean up
finally:
if staging is not None:
shutil.rmtree(staging, ignore_errors=True)
return dest
+1 -1
View File
@@ -17,7 +17,7 @@ instead would defeat that — the orchestrator holds that key, so it could forge
`ControlPlaneProvisioning` is the one seam every backend launcher uses to get the
orchestrator its key and the gateway its token, instead of re-deriving that
wiring per backend (the bug class behind PR #471 — see
`docs/prds/prd-new-control-plane-auth-provisioning.md`).
`docs/prds/0079-control-plane-auth-provisioning.md`).
Stdlib-only: the HMAC lives in `orchestrator_auth`, the key file in `paths`.
"""
+1
View File
@@ -7,6 +7,7 @@ picking the right document for what you're capturing.
| Artifact | For |
|---|---|
| **Design workflow** (`docs/design-workflow.md`) | How discussion becomes canonical design, how dependencies are recorded, and when implementation may begin. |
| **Glossary** (`docs/glossary.md`) | Canonical term definitions — what words mean in this project. |
| **PRD** (`docs/prds/`) | A feature: what to build, scope, success criteria. |
| **Research note** (`docs/research/`) | A landscape/tradeoff investigation. |
@@ -1,9 +1,13 @@
# ADR 0005: Keep tracker metadata on issues
# ADR 0005: Keep tracker metadata on one tracker object
- **Status:** Accepted
- **Date:** 2026-07-18
- **Deciders:** didericis
> **Amended 2026-07-26.** A pull request may carry labels directly instead of
> linking a tracking issue. When a PR does link an issue, the issue remains the
> canonical owner of planning metadata and the reference is validated.
## Context
Gitea exposes labels on both issues and pull requests. Applying the same labels
@@ -20,19 +24,29 @@ would make the issue history less truthful.
## Decision
Issues are the canonical tracker records and own labels. Every issue has at
least one label. An issue opened or left without labels receives
`Status/Needs Triage` automatically until it is classified.
Issues are the canonical tracker records and own labels when a separate work
item exists. Every issue has at least one label. An issue opened or left
without labels receives `Status/Needs Triage` automatically until it is
classified.
Pull requests carry no labels. Every new PR deliberately references at least
one existing issue in its title or description with one of these forms:
Every new pull request is tracked in exactly one of two mutually exclusive
ways:
1. It deliberately references at least one existing issue in its title or
description. Tracker metadata stays on that issue and the PR remains
unlabelled.
2. It carries at least one label directly when a separate issue would add no
useful planning context.
Issue references use one of these forms:
- `Closes #123`, `Fixes #123`, or `Resolves #123` when merging completes it.
- `Part of #123`, `Related to #123`, `Refs #123`, or `References #123` when it
contributes without completing it.
Gitea Actions enforces both PR rules as a status check and repairs the empty
issue-label state. Branch protection makes the PR policy check required.
Gitea Actions enforces the exclusive either/or PR rule, validates any issue
references, and repairs the empty issue-label state. Branch protection makes
the PR policy check required.
The policy applies from 2026-07-18 onward. Existing issues may be labelled as
they are encountered, but closed PRs are grandfathered: no retrospective
@@ -40,9 +54,13 @@ issues or PR labels are created solely to make history conform.
## Consequences
- Classification, priority, and workflow metadata have one source of truth.
- A PR's issue link is the navigation path to its planning metadata.
- Classification, priority, and workflow metadata have one source of truth for
each change: the linked issue when one exists, otherwise the PR.
- For issue-backed changes, the PR's issue link is the navigation path to its
planning metadata.
- Multi-PR issues do not require copied or synchronized labels.
- Small standalone changes do not require a tracking issue created solely to
satisfy automation.
- `Status/Needs Triage` is an intentional fallback, not a final
classification.
- Direct issue creation remains convenient; automation repairs a missing label
@@ -53,5 +71,6 @@ issues or PR labels are created solely to make history conform.
## Links
- Issue #405.
- `.gitea/workflows/tracker-policy.yml`.
- `.gitea/workflows/tracker-policy-pr.yml`.
- `.gitea/workflows/tracker-policy-issues.yml`.
- `scripts/tracker_policy.py`.
+218
View File
@@ -0,0 +1,218 @@
# Design workflow
How bot-bottle turns discussion into canonical design and then into
implementation without leaving the repository's architecture scattered across
issue and review threads.
The goal is not more documentation. The goal is one discoverable current answer
for every load-bearing design question.
## Sources of truth
Design artifacts have different jobs:
| Artifact | Authority |
|---|---|
| Decision records | Stable system-wide boundaries, policies, and invariants |
| PRDs | The current design for a feature |
| Research notes | Evidence and tradeoff analysis; informative, not normative |
| Issues | Work tracking, open questions, and discussion |
| Pull-request comments | Review history; never the final home of a design decision |
When a discussion changes the design, update the relevant PRD or decision
record before treating the discussion as resolved. A comment may explain why a
decision changed, but future implementers must not need to reconstruct the
decision from a thread.
Avoid duplicating the same rule in several canonical documents. Prefer one
canonical statement and links from dependent documents.
## Choosing the canonical artifact
Use a PRD when the decision describes a feature: its behavior, scope, success
criteria, trust model, implementation slices, and tests.
Use a decision record when the choice is broader than one feature or will
constrain several future features. Examples include state ownership, credential
boundaries, compatibility policy, and what the project does or does not claim
as a security guarantee.
Use a research note when the conclusion depends on comparing external systems,
protocols, or approaches. Promote any resulting project decision into a PRD or
decision record.
## From discussion to implementation
### 1. Open the design discussion
An issue may start with incomplete requirements. Record:
- the problem and desired outcome;
- known security or compatibility constraints;
- the current owner of affected state and credentials;
- related PRDs, decisions, issues, and pull requests;
- open questions that would materially change the implementation.
Do not disguise an unresolved trust-boundary or state-ownership decision as an
implementation detail.
### 2. Draft or update the canonical design
Before substantial implementation, write the feature PRD and update any
system-wide decision it changes.
An active design should make these relationships visible near its top:
```markdown
Status: Draft | Active | Superseded | Retargeted
Depends on: #...
Supersedes: ...
```
Record dependencies only on the dependent document. Do not maintain reverse
`Blocks` lists that can drift as dependent work changes.
For security-sensitive work, state:
- the exact guarantee and explicit non-guarantees;
- trusted and untrusted components;
- who creates each identity or attribution field;
- who owns durable state;
- failure and recovery behavior;
- how the design is tested at its boundaries.
### 3. Resolve review into the repository
When review settles a design-changing question:
1. Update the canonical document in the same pull request.
2. Mark conflicting documents Superseded or Retargeted, or update them.
3. Add or adjust dependency links.
4. Leave a concise resolution comment linking to the canonical change.
A useful resolution comment is:
```text
Resolution: <what was decided>
Canonicalized in: <document/section/commit>
Supersedes: <older statement, if any>
Follow-up: <remaining implementation or question>
```
The resolution is incomplete until the repository reflects it.
### 4. Check design readiness
Implementation may begin when:
- the PRD's material trust, ownership, and compatibility questions are settled;
- dependencies and blockers are explicit;
- the design agrees with current architecture and decision records;
- superseded documents are marked or updated;
- success criteria and boundary tests are concrete;
- remaining open questions can be answered during implementation without
changing the feature's guarantee or component ownership.
Small exploratory spikes may happen earlier. A spike proves feasibility; it does
not establish a production contract or silently settle the design.
### 5. Implement in ordered slices
Prefer small, independently reviewable slices after the parent design is
accepted. Record the dependency chain explicitly.
Parallel work is safe when slices do not compete for the same unsettled
interface or ownership boundary. If a foundational change will alter the
transport, schema, state owner, or trust domain used by another slice, land the
foundation first.
An implementation pull request should identify:
- the PRD or decision it implements;
- the implementation chunk;
- its base and blockers;
- any design deviation discovered during implementation.
If implementation reveals a load-bearing design change, pause that slice and
update the canonical design. Do not let the code and review thread become an
undocumented replacement for the PRD.
## Dependency and staleness management
### Dependency direction
Write dependencies in terms of contracts, not chronology:
```text
credential provisioning contract
-> host-controller authentication
-> privileged host operations
```
If only part of a feature is blocked, say so. For example, a manifest parser may
proceed while that feature's durable audit-storage chunk waits for the canonical
audit schema.
### Superseding documents
Do not silently edit history to make an old design appear to have always said
the new thing. Preserve the rationale, but make current status unmistakable:
```markdown
Status: Superseded
Superseded by: <document>
Reason: <one paragraph>
```
If part of a PRD remains valid, mark it Retargeted and identify which scope moved
elsewhere.
Add a short supersession note near the top explaining what changed, why the old
design is no longer current, and where the current design lives. For a research
note whose original analysis remains useful, preserve that analysis and append
a dated addendum with the newer finding instead of rewriting the note as though
it had always reached the new conclusion.
### Architecture sweeps
After a foundational change, do a targeted architecture sweep before building
more features on it:
1. Identify the concepts the change affects, such as `bot-bottle.db`, host
controller, orchestrator, audit ownership, or signing key.
2. Search active PRDs, decisions, and open issues for those concepts.
3. Update or supersede contradictory statements.
4. Refresh dependency links and the current architecture summary.
5. Confirm stacked implementation branches still have the correct base.
This is a milestone activity, not a recurring documentation ceremony.
## Pull-request checklist
Use the relevant items in design and implementation pull requests:
- [ ] The canonical PRD or decision is linked.
- [ ] Design-changing review decisions are reflected in-repo.
- [ ] Dependencies and blockers are explicit.
- [ ] State, credential, and trust ownership agree with current architecture.
- [ ] Superseded or retargeted documents are marked.
- [ ] Security guarantees and non-guarantees are precise.
- [ ] Open questions do not change the promised guarantee or ownership model.
- [ ] Implementation deviations updated the canonical design.
## Lightweight maintenance
Automation should enforce document shape, not pretend to understand
architecture. Useful checks include:
- active PRDs contain status and dependency metadata;
- superseded PRDs link to their replacement;
- referenced documents and issues exist;
- implementation pull requests identify their PRD and chunk;
- document filenames and lifecycle states follow repository conventions.
Human review remains responsible for detecting conflicting guarantees or
ownership claims.
The durable rule is simple: **discussion discovers the decision; the repository
records it; implementation follows it.**
@@ -1,9 +1,14 @@
# PRD 0001: Per-agent egress proxy via pipelock
- **Status:** Active
- **Status:** Superseded by [PRD 0017](0017-egress-proxy-via-mitmproxy.md)
and [PRD 0052](0052-egress-dlp-addon.md)
- **Author:** didericis
- **Created:** 2026-05-08
> **Superseded.** Pipelock was removed in issue #193. PRD 0017 moved
> egress enforcement and credential injection to mitmproxy; PRD 0052 moved DLP
> enforcement into the egress addon. The design below is retained as history.
## Summary
Run pipelock as a sidecar container on each bot-bottle agent's only
+6 -1
View File
@@ -1,9 +1,14 @@
# PRD 0006: pipelock native TLS interception
- **Status:** Active
- **Status:** Superseded by [PRD 0017](0017-egress-proxy-via-mitmproxy.md)
and [PRD 0052](0052-egress-dlp-addon.md)
- **Author:** didericis
- **Created:** 2026-05-12
> **Superseded.** Pipelock was removed in issue #193. TLS interception now
> belongs to the mitmproxy egress design in PRD 0017, with DLP implemented by
> the egress addon in PRD 0052. The design below is retained as history.
## Summary
Turn on pipelock's built-in `tls_interception` so its DLP / URL /
+7 -1
View File
@@ -1,11 +1,17 @@
# PRD 0015: pipelock block remediation
- **Status:** Active
- **Status:** Superseded by [PRD 0017](0017-egress-proxy-via-mitmproxy.md)
and [PRD 0052](0052-egress-dlp-addon.md)
- **Author:** didericis
- **Created:** 2026-05-25
- **Parent:** PRD 0012
- **Depends on:** PRD 0013
> **Superseded.** Pipelock and its restart-based allowlist remediation path
> were removed in issue #193. Current egress enforcement is the mitmproxy
> design from PRD 0017 with DLP in PRD 0052. The design below is retained as
> history.
## Summary
Wires the **pipelock block** path (PRD 0012 *Stuck categories*) end-to-end. The supervisor, on approval of a `pipelock-block` proposal, writes the new pipelock allowlist to the host and restarts pipelock; the agent's in-flight outbound calls may drop and rely on retry. The TUI gains a proactive `pipelock edit <bottle>` verb for operator-initiated edits unrelated to a tool call. The pipelock audit log (format defined in PRD 0013) is filled in with real entries on every edit.
@@ -4,6 +4,11 @@
- **Author:** didericis
- **Created:** 2026-05-26
> **Superseded.** PRD 0049 removed the active-agents pane and agent-scoped
> operator edit verbs when the dashboard was narrowed back to a proposal-only
> supervise TUI. A future agent-management surface was deferred rather than
> carried forward from this design. The design below is retained as history.
## Summary
The dashboard today is proposal-centric: it lists every pending
@@ -4,6 +4,11 @@
- **Author:** didericis
- **Created:** 2026-05-26
> **Superseded.** PRD 0049 removed start, re-attach, and stop actions from the
> dashboard when it became the proposal-only supervise TUI. Bottle lifecycle
> remains in the dedicated CLI commands; no dashboard replacement from this
> design remains active. The design below is retained as history.
## Summary
Today the dashboard is read-only: it surfaces pending proposals
@@ -4,6 +4,11 @@
- **Author:** didericis
- **Created:** 2026-05-26
> **Superseded.** PRD 0049 removed agent handoff and tmux pane management when
> the dashboard was reduced to the proposal-only supervise TUI. The split-pane
> interaction described below has no active replacement and is retained only
> as design history.
## Summary
When the dashboard runs inside tmux, lay it out as the **left
+5 -1
View File
@@ -1,9 +1,13 @@
# PRD 0024: Consolidate per-bottle sidecars into a single bundle
- **Status:** Active
- **Status:** Superseded by [PRD 0070](0070-per-host-orchestrator.md)
- **Author:** didericis
- **Created:** 2026-05-26
> **Superseded.** PRD 0070 replaced the per-bottle sidecar bundle with a
> persistent per-host gateway and separate orchestrator control plane. The
> design below is retained as history.
## Summary
Replace the four per-bottle sidecar containers in the Docker
@@ -1,10 +1,16 @@
# PRD 0037: Pipelock YAML Render Contract
- **Status:** Active
- **Status:** Superseded by [PRD 0017](0017-egress-proxy-via-mitmproxy.md)
and [PRD 0052](0052-egress-dlp-addon.md)
- **Author:** didericis-codex
- **Created:** 2026-06-02
- **Issue:** #130
> **Superseded.** Pipelock and its YAML renderer were removed in issue #193.
> Current egress configuration is consumed by the mitmproxy design from PRD
> 0017 and its DLP addon from PRD 0052. The contract below is retained as
> history.
## Summary
Lock down the contract between `pipelock_build_config` and
+8 -1
View File
@@ -1,10 +1,17 @@
# PRD 0067: SQLite local storage
- **Status:** Active
- **Status:** Retargeted by [PRD 0070](0070-per-host-orchestrator.md) and
issues #469/#471
- **Author:** codex
- **Created:** 2026-07-01
- **Issue:** #319
> **Retargeted.** The SQLite storage and migration foundation remains in use,
> but the writable data-plane database mount described below is no longer the
> active ownership model. Issues #469/#471 removed `bot-bottle.db` from the
> data plane; under PRD 0070 only the orchestrator control plane opens the
> operational database, and gateway components reach state through RPC.
## Summary
Add a small stdlib SQLite storage layer for bot-bottle host runtime state,
@@ -1,6 +1,6 @@
# PRD 0069: Firecracker-native, Docker-free backend
- **Status:** Draft (partially superseded)
- **Status:** Retargeted by [PRD 0070](0070-per-host-orchestrator.md)
- **Author:** Claude
- **Created:** 2026-07-12
- **Issue:** #348
+1 -1
View File
@@ -1,6 +1,6 @@
# PRD 0070: Per-host orchestrator service
- **Status:** Draft
- **Status:** Active
- **Author:** Claude
- **Created:** 2026-07-12
- **Issue:** #351
@@ -1,6 +1,6 @@
# PRD prd-new: Claude forward_host_credentials
# PRD 0071: Claude forward_host_credentials
- **Status:** Draft
- **Status:** Active
- **Author:** claude
- **Created:** 2026-07-01
- **Issue:** #325
@@ -1,6 +1,6 @@
# PRD prd-new: Non-blocking supervise (async approval + proposal polling)
# PRD 0072: Non-blocking supervise (async approval + proposal polling)
- **Status:** Draft
- **Status:** Active
- **Author:** didericis
- **Created:** 2026-07-18
- **Issue:** #412
@@ -1,4 +1,4 @@
# PRD prd-new: Consolidate infra backend for Docker
# PRD 0073: Consolidate infra backend for Docker
- **Status:** Active
- **Author:** Claude
@@ -1,4 +1,4 @@
# PRD prd-new: CI artifact-based coverage and local Firecracker candidate flow
# PRD 0074: CI artifact-based coverage and local Firecracker candidate flow
- **Status:** Active
- **Author:** Claude
@@ -1,6 +1,6 @@
# PRD prd-new: Containers inside a bottle
# PRD 0075: Containers inside a bottle
- **Status:** Draft
- **Status:** Active
- **Author:** Claude
- **Created:** 2026-07-21
- **Issue:** #392
@@ -1,6 +1,6 @@
# PRD prd-new: Modernize built-in agent images
# PRD 0076: Modernize built-in agent images
- **Status:** Draft
- **Status:** Active
- **Author:** Codex
- **Created:** 2026-07-21
- **Issue:** #451
@@ -1,6 +1,6 @@
# PRD prd-new: macOS (Apple Container) CI runner
# PRD 0077: macOS (Apple Container) CI runner
- **Status:** Draft
- **Status:** Active
- **Author:** Claude
- **Created:** 2026-07-25
- **Issue:** #426
+172
View File
@@ -0,0 +1,172 @@
# PRD 0078: Quick install script
- **Status:** Active
- **Author:** claude
- **Created:** 2026-07-25
- **Issue:** #197
## Summary
Add a proper Python package distribution (`pyproject.toml` with a
`bot-bottle` entry point) plus a thin `install.sh` bootstrapper, so users
can install bot-bottle with a single command instead of cloning the repo
and invoking `cli.py` directly. A new `bot-bottle doctor` subcommand
verifies host prerequisites after install.
## Problem
There is currently no install path for new users. The only way to run
bot-bottle is to clone the repo and invoke `./cli.py`. This blocks any
public demo: readers want `curl | sh` or `pipx install`, not a manual
clone-and-configure flow. There is also no single command that tells a
user whether their host is actually ready to run a bottle.
## Goals / Success Criteria
- `curl -fsSL <raw-url>/install.sh | sh` leaves a working `bot-bottle`
command on PATH.
- Python-native users can install with `pipx install bot-bottle` or
`uv tool install bot-bottle` (once published) — or from a local
checkout today.
- `install.sh` validates prerequisites (Python ≥ 3.11), creates the
`~/.bot-bottle/` config tree, installs the package, and runs
`bot-bottle doctor`. It never installs Docker or a VM backend silently
and never uses `sudo`.
- `install.sh` is idempotent — safe to re-run.
- `bot-bottle doctor` reports Python version, backend *readiness*, and
config-dir presence, exiting non-zero when a hard prerequisite is unmet.
- The package keeps **zero runtime pip dependencies** (stdlib-only,
matching the existing constraint in `AGENTS.md`).
## Non-goals
- Bundling a Python runtime or producing a standalone binary.
- Automatic Docker / VM-backend installation.
- Plugin-architecture changes (issue #197 floats a containerized-plugin
direction; that's a separate feature).
- Publishing to a package index in this PR — the package *structure* is
the deliverable; publishing is a follow-up step.
## Design
### Package structure (`pyproject.toml`)
Fill out the previously-stub `pyproject.toml` with project metadata, a
console-script entry point, and package-data for the non-Python assets the
runtime reads from inside the package:
```toml
[project]
name = "bot-bottle"
version = "0.1.0"
requires-python = ">=3.11"
dependencies = []
[project.scripts]
bot-bottle = "bot_bottle.cli:main"
```
`bot_bottle.cli:main` already exists (the `cli.py` shim calls it), so no
refactor of the entry point is needed. `package-data` ships the non-Python
assets that live *inside* the package (`egress_entrypoint.sh`, the contrib
Dockerfiles, the firecracker netpool defaults, the macos-container init
script).
### Self-contained wheel (build resources)
The gateway / infra / orchestrator images are built from a Docker (or Apple
`container`) build context that must contain the `bot_bottle` package,
`pyproject.toml`, and the **root-level** Dockerfiles as siblings. Several
modules used to locate that context by walking `__file__`'s parents to the
repo root (`_REPO_ROOT = Path(__file__)…parents[N]`) and reading
`Dockerfile.gateway`, `nix/firecracker-netpool.nix`, and
`scripts/firecracker-netpool.sh` from it. In an installed wheel the package
lives in `site-packages` with no repo root above it, so those reads fail —
`doctor` passes but `start` / backend setup breaks.
Fix: a single resolver, `bot_bottle/resources.py`.
- `build_root()` returns a directory shaped like a repo root (has
`bot_bottle/`, `pyproject.toml`, the Dockerfiles, `nix/`, `scripts/`).
In a **checkout** it's the repo root itself — unchanged behavior. From an
**installed wheel** it stages a copy under the app-data dir, keyed by a
**content digest** of the installed package + bundled resources (not the
distribution version): the installer defaults to a git branch and
`pipx install --force` while `version` stays `0.1.0`, so a version key
would reuse a previous commit's tree — the digest key re-stages instead.
Staging is concurrency-safe: a file lock serializes it, each writer builds
into a private temp dir, and the finished tree is published with an atomic
rename (never populating a shared path another process might read).
- The root-level resources are shipped inside the wheel under
`bot_bottle/_resources/` by a `setup.py` `build_py` step (kept in sync
with `resources.BUNDLED_RESOURCES`); `MANIFEST.in` includes them in the
sdist.
- Every former `_REPO_ROOT` / `_REPO_DIR` call site now derives from
`resources`: the docker/macos agent-image launch, each backend's
`orchestrator` / `gateway` / `infra` service, firecracker `infra_vm` /
`infra_artifact` / `setup`, and the shared `gateway` build context. So
checkout and wheel installs share one downstream path.
Verification: `test_resources` exercises both layouts — including the staged
wheel context, a re-stage when package content changes at the same version,
and a rebuild of a partial (crashed) stage. `test_wheel_install` builds the
wheel, installs it into an isolated venv, and asserts `bot-bottle doctor`
runs and `build_root()` produces a valid context; `build` is in
`requirements-dev.txt` so it runs in CI, and a build/install failure fails
the test (it does not skip). Running `start` end-to-end still needs a
Docker/KVM host (CI), not a source checkout.
### `install.sh`
A POSIX `sh` bootstrapper that:
1. Checks `python3` is present and ≥ 3.11; exits with a clear message
otherwise.
2. Checks `git` when installing a `git+` spec, and — when falling back to
pip — that pip is usable and the interpreter isn't externally managed
(PEP 668), pointing at pipx otherwise.
3. Creates `~/.bot-bottle/{agents,bottles,contrib}`.
4. Installs via `pipx` if available, else `python3 -m pip install --user`.
The spec defaults to the git URL and is overridable via
`BOT_BOTTLE_INSTALL_SPEC` (used by tests / local installs).
5. Locates the `bot-bottle` entry point: PATH first, else the
interpreter's own user-scheme scripts dir resolved via `sysconfig`
(`~/.local/bin` on Linux, `~/Library/Python/<X.Y>/bin` on a python.org
macOS interpreter — not hardcoded).
6. Runs `bot-bottle doctor` and reports the result.
It is idempotent and never calls `sudo`.
### `bot-bottle doctor`
A new store-free subcommand (no DB migration required) that checks and
reports:
- **python** — interpreter version (hard requirement: ≥ 3.11).
- **backend** — at least one backend *ready* on this host
(macos-container / firecracker / docker), via `is_backend_ready()` — a
full backend `status()` probe (daemon reachable, network pool present,
KVM usable), not a PATH-only check: a stopped daemon or half-configured
backend must not report `ok` when `start` can't work. Each not-ready
backend prints its own diagnostics. Hard requirement.
- **config** — whether `~/.bot-bottle/` exists (advisory only; `start`
provisions on first run).
Exits 0 when both hard requirements pass, non-zero otherwise.
## Testing strategy
- Unit test `bot-bottle doctor` success/failure paths with backend
readiness (`is_backend_ready`) and Python version mocked, including the
available-but-not-ready → fail case.
- Unit test that `pyproject.toml` parses, declares the entry point and an
empty `dependencies` list, and that every `package-data` glob resolves
to a file that exists on disk (guards against drift).
- Unit test that `install.sh` is executable, POSIX-ish (`set -eu`), never
calls `sudo`, and runs `doctor` after install.
## Open questions
- Should `version` be derived from a git tag at build time (e.g.
`hatch-vcs`) or kept static? Static (`0.1.0`) is simpler for now.
- Publishing target (PyPI vs. a self-hosted index) is deferred.
@@ -1,6 +1,6 @@
# PRD prd-new: Per-service signing keys for control-plane auth
# PRD 0079: Per-service signing keys for control-plane auth
- **Status:** Draft
- **Status:** Active
- **Author:** claude
- **Created:** 2026-07-26
- **Issue:** #476
@@ -1,4 +1,4 @@
# PRD prd-new: Encrypted at-rest egress secrets (SecretProvider, interim slice)
# PRD 0080: Encrypted at-rest egress secrets (SecretProvider, interim slice)
- **Status:** Draft
- **Author:** didericis
+8 -5
View File
@@ -7,10 +7,13 @@ document vs. a research note or a decision record).
## Naming and numbering
New PRDs use a `prd-new-<kebab-title>.md` placeholder name while the PR
is open. On merge to `main` a CI workflow assigns the next sequential
number (`0024-…`, `0025-…`), renames the file, and updates the title
header. Numbers are never reused; gaps are fine.
New PRDs may use a `prd-new-<kebab-title>.md` placeholder name while the
design is being drafted. Before merge, assign the next sequential number
after the highest-numbered PRD on `main`, rename the file to
`NNNN-<kebab-title>.md`, and update the title header. CI blocks merging
while any `prd-new-*.md` placeholder remains. If concurrent PRs select the
same number, the later PR must take the next available number before it
merges. Numbers are never reused; gaps are fine.
Once numbered, the filename stays fixed for the life of the doc.
@@ -26,7 +29,7 @@ The `Status:` line near the top tracks the PRD's lifecycle:
## Format
```markdown
# PRD prd-new: <short title> ← placeholder; CI fills in the number on merge
# PRD prd-new: <short title> ← replace with the final number before merge
- **Status:** Draft
- **Author:** <who>
+732
View File
@@ -0,0 +1,732 @@
# PRD prd-new: Canonical tamper-evident audit-event schema and local query contract
- **Status:** Draft
- **Author:** didericis-claude
- **Created:** 2026-07-26
- **Issue:** #487
## Summary
bot-bottle already emits security- and provenance-relevant events from
several producers — supervise operator decisions (PRD 0013's
`AuditStore`), egress allow/block enforcement, git-gate push decisions,
control-plane token minting, and (next) host-controller lifecycle
transitions — but each writes its own shape to its own sink. There is no
shared envelope, no tamper-evidence, and no single place to search. Local
incident reconstruction means grepping several stores that don't agree on
field names, timestamps, or how a bottled agent is identified.
This PRD defines **one canonical audit-event contract** every producer
emits into:
1. A **versioned envelope** — schema version, event id, event/observed
timestamps, host-attributed identity (`bottle`/`bottled_agent`/
`activation`), provenance (`manifest_digest` — the manifest is the
policy — and `engine` = bot-bottle version/SHA),
host-observed `actor`/`action`/`resource`/`outcome`, correlation/
causation ids, a sensitivity class, a typed payload, and an explicit
**trust boundary** between host-supplied and agent-claimed fields.
2. **Canonical JSON serialization + a per-writer hash chain** (with
normative test vectors), so any deletion, edit, or reorder of a past
record breaks the chain and is detectable offline.
3. An **append-only JSONL journal as the source of truth**, with a
**rebuildable SQLite index** and a local `audit query`/`verify` surface
— no paid platform, no network dependency.
4. An **initial event registry** covering lifecycle, host-controller,
supervise decision, egress (request/decision/cutoff/anomaly), git-gate
and signed-commit (#480), auth/authz, and audit self-events — each with
its trusted-vs-claimed fields and redaction rules.
5. A **stable export projection** (CloudEvents / OpenTelemetry Logs) and
the **#324 delivery contract** (payload, per-chain
`(host, epoch, seq)` cursor, dedup,
backpressure, retention ordering).
It is explicitly scheduled to land **immediately after the host
controller (#468)** so the host controller's lifecycle transitions are the
first producer wired onto the new contract (per the directive on #487).
## Problem
Audit infrastructure is fragmented across #468, #324, and #480 with no
shared schema. Concretely:
- **No shared envelope.** `supervise_audit_entries` (PRD 0013) has
`timestamp, bottle_slug, component, operator_action, ...`. The egress
proxy and git-gate log their own ad-hoc lines. There is no common
`event_id`, `event_type`, or version, so cross-producer correlation
("what did bottled agent X do between its start and this rejected push?") is
manual and lossy.
- **No tamper-evidence.** The audit store is a plain SQLite table. Anyone
who can write the DB can delete or rewrite a row and leave no trace.
Audit that an attacker (or a buggy agent) can silently rewrite is not
audit.
- **Trusted and untrusted data are mixed.** A bottled agent is attributed by
**source IP → slug** at the gateway (host-supplied, trustworthy). An
agent can also *claim* things about itself in a tool call
(agent-claimed, adversarial). Today nothing in the record marks which is
which, so a reader can be misled by an agent-supplied field that looks
authoritative.
- **No local search.** Reconstructing an incident means reading multiple
sinks with different schemas. There is no query contract and no promise
that the index can be rebuilt from the journal if it drifts or is lost.
- **No redaction rule.** Nothing prohibits a producer from writing a raw
token or secret into an audit record, which would turn the audit log
itself into a credential store.
## Goals / Success Criteria
- A single `AuditEvent` envelope type, versioned, that every producer
emits. The trust boundary is **one `untrusted` region**: everything
outside it is host-established and trusted (source-IP → bottled-agent
attribution, host wall-clock, producer identity, chain metadata);
`untrusted` is the sole place anything an agent or a remote claimed may
go. The boundary is structural, not a convention.
- **Canonical serialization** (`sort_keys`, `(",", ":")` separators,
UTF-8, `ensure_ascii=False`) is defined once and reused, so the same
logical event always hashes identically across producers and hosts.
- Each writer maintains a **hash chain**: `hash = sha256(prev_hash ||
canonical(event))`. Deleting or editing any past record breaks every
subsequent link; a standalone verifier detects the break offline with no
secret material.
- The **JSONL journal is the source of truth**; the **SQLite index is
fully rebuildable** from it (`audit rebuild` reconstructs the DB and
re-verifies the chain).
- **Local query** works with no paid platform and no egress: filter by
bottled agent, event type, time range, and producer, and follow a bottled
agent's events in order.
- A **redaction rule** is enforced at the envelope boundary: known
credential-shaped fields are rejected/redacted before a record is
written; the writer refuses raw secrets rather than storing them.
- The envelope **projects onto the OpenTelemetry Logs data model and a
CloudEvents JSON envelope** by field re-mapping alone (no reformat),
preserving the trust boundary and carrying the integrity fields — per
#487's export/interop requirement. (The export adapters are follow-up;
the *schema* must make them a re-map.)
- The **host controller (#468)** emits `lifecycle.*` events through this
contract as the first consumer; existing supervise/egress producers are
migrated behind the same envelope without changing operator-facing
behavior.
## Non-goals
- **Cross-host aggregation / shipping.** This PRD makes each host's journal
canonical and correlatable *by construction* (stable ids, hash chain),
but the transport that merges multiple hosts into one timeline is a
follow-up (#324). The schema is designed so that merge is a later append,
not a reformat.
- **Cryptographic signing / external anchoring.** Hash-chaining gives
tamper-**evidence** (you can detect edits), not tamper-**resistance**
against an attacker who can rewrite the whole chain. Per-writer signing
keys and periodic external anchoring are a follow-up; the chain-head hash
is the seam they attach to.
- **Real-time alerting / SIEM rules.** Query is local and pull-based here.
- **Retention / rotation policy.** Journal rotation and TTL are operator
policy, tracked separately; the format must survive rotation (chain head
carried across segments) but this PRD does not set the schedule.
- **Replacing PRD 0013's operator queue.** The supervise proposal/response
queue is unchanged; only its terminal *audit* record is re-emitted onto
the new envelope.
## Design
### The envelope
One dataclass, `AuditEvent`, serialized to a JSON object with a small,
stable top level:
```
{
// ---- schema + integrity (host-owned) ----
"v": 1, // schema version — bumped only on a breaking change
"id": "<uuid4>", // globally unique event id; stable across export/replay (dedup key)
"type": "egress.decision", // dotted event type from the registry
"epoch": 7, // writer-boot counter, bumped once per host-controller (writer) start
"seq": 1287, // monotonic sequence within this epoch (gap-detectable)
"segment": "20260726T000000Z", // journal segment id (rotation boundary); chain continues across segments
"prev": "<hex>", // prior record hash ("" only for the first record of a new chain)
"hash": "<hex>", // sha256(prev + canonical(this event with hash=""))
// ---- timestamps (host-owned) ----
"ts_event": "2026-07-26T18:22:04.061Z", // when the underlying event occurred at the boundary
"ts_recorded": "2026-07-26T18:22:04.113Z", // when the single writer appended it (authoritative)
"ts_mono": 90142.55, // monotonic secs since this epoch's boot (intra-epoch ordering only)
// ---- attribution + provenance (host-established) ----
"producer": "egress", // host component that emitted the event
"host": "mac-studio-1",
"engine": "bot-bottle/0.1.0+abc1234", // bot-bottle version + git SHA of the enforcing host code
"bottle": "amber-fox", // bottle (container/VM) identity
"bottled_agent": "amber-fox-12", // bottled-agent slug from source-IP attribution (null for host-level events)
"activation": "01J8Z...", // activation id: one run/session of the bottled agent (null if n/a)
"manifest_digest": "sha256:9f2…", // digest of the manifest — which IS the policy (egress routes etc.); null if n/a
// ---- semantics: host-observed facts of what happened ----
"actor": "bottled-agent:amber-fox-12", // who acted, as a host-attributed identity
"action": "egress.connect", // what was attempted / done
"resource": "registry.npmjs.org:443", // what it acted on, as observed at the boundary
"outcome": "blocked", // host-decided result: allowed|blocked|deferred|success|failure
"sensitivity": "security", // classification: normal|security|restricted (drives redaction + export)
// ---- correlation (host-assigned) ----
"correlation_id": "flow-9c2a…", // groups a related flow (request → decision → cutoff)
"causation_id": "<event id>", // the event that directly caused this one ("" if root)
// ---- typed, trusted, event-specific payload (shape fixed per type in the registry) ----
"payload": {
"route_id": 4,
"detector": "token_patterns"
},
// ---- the ONLY untrusted region: agent- or remote-claimed data ----
"untrusted": {
"reason": "npm install needs registry.npmjs.org" // the agent's stated justification
}
}
```
The **trust boundary is a single region, not a split.** Everything outside
`untrusted` is trusted by construction — the host established it: schema and
chain metadata, both timestamps, the attribution/provenance fields
(source-IP → `bottled_agent`/`bottle`/`activation`, `manifest_digest`,
`engine`), the host-observed semantics
(`actor`/`action`/`resource`/`outcome`), the host-assigned correlation ids,
and the typed `payload`. `untrusted` is the **one** place anything an agent
or a remote claimed may go (e.g. the agent's free-text `reason`). A reader
(or a future policy engine) trusts every field outside `untrusted` for
attribution and treats `untrusted.*` — and only `untrusted.*` — as
adversarial claims.
Framing it as "one untrusted region, everything else trusted" removes the
mistake where a producer forgets to mark a claimed field: a field is
trusted unless it is deliberately placed inside `untrusted`. The
construction API enforces this — producers pass trusted fields explicitly
and hand all agent/remote-claimed data as the single `untrusted` mapping,
so there is no way to emit a top-level field that *looks* authoritative but
isn't.
**Two timestamps** because they answer different questions and can diverge
under backpressure: `ts_event` is when the thing happened at the boundary
(the proxy saw the connect, the gate saw the push); `ts_recorded` is when
the single writer durably appended it. Ordering and the chain use
`(epoch, seq)`, never either wall clock. Both are host-set — a bottled
agent never supplies a timestamp.
**The manifest *is* the policy.** bot-bottle has no separate policy
artifact — a bottled agent's egress routes and other constraints are
declared in its manifest (`bot_bottle/manifest/egress.py`), so
`manifest_digest` already pins the ruleset in force; there is no distinct
`policy_version`. Given a fixed manifest, the only other thing that can
change a decision's outcome is the enforcing code — captured by `engine`
(bot-bottle version + git SHA). So two `egress.decision` records with the
same `resource` but different `outcome` are explained by exactly one of:
different `manifest_digest` (the rules changed) or different `engine` (the
enforcer changed). Runtime operator overrides (a supervise `egress-allow`)
are themselves audit events, so the effective ruleset at any instant is
`manifest_digest` plus the logged, approved deltas — reconstructable from
the chain, not from a version stamp.
**Optionality.** `bottle`/`bottled_agent`/`activation`, `manifest_digest`,
and `payload`/`untrusted` are `null`/absent for events that have no such
subject (a host-level `hostctl.*` or `audit.*` event has no bottled agent).
Absent ≠ empty: a reader distinguishes "no subject" from "unknown". `id`,
`type`, the chain fields, both timestamps, `producer`, `host`, `engine`,
`actor`, `action`, `outcome`, and `sensitivity` are always present.
#### Trust provenance of every common field
| Field | Trust | Set by |
|---|---|---|
| `v` `id` `type` `epoch` `seq` `segment` `prev` `hash` | trusted | the single writer |
| `ts_event` | trusted | emitting host component (boundary) |
| `ts_recorded` `ts_mono` | trusted | the single writer |
| `producer` `host` `engine` | trusted | the single writer |
| `bottle` `bottled_agent` `activation` | trusted | gateway source-IP → slug attribution |
| `manifest_digest` | trusted | control plane (the manifest = the policy in force) |
| `actor` `action` `resource` `outcome` | trusted | host component that observed/decided it |
| `sensitivity` | trusted | registry default for `type`, overridable up (never down) by the producer |
| `correlation_id` `causation_id` | trusted | the single writer (assigned as it threads the flow) |
| `payload.*` | trusted | emitting host component (shape fixed per `type`) |
| `untrusted.*` | **claimed** | copied verbatim from a bottle / gateway / forge / remote |
Every registry entry (below) restates, per event type, which `payload`
keys are required and names any `untrusted` keys it carries — so "trusted
vs claimed" is explicit for every event-specific attribute, not just the
common ones.
### Canonical serialization + hash chain
Serialization is defined once (extends the existing `sha256_hex` /
`util.py` helpers):
```
def canonical(event: dict) -> str:
return json.dumps(event, sort_keys=True, separators=(",", ":"),
ensure_ascii=False)
```
The `hash` field is computed over the canonical form of the event **with
`hash` set to `""`**, prefixed by the previous record's hash:
```
digest = sha256_hex(prev_hash + canonical({**event, "hash": ""}))
```
`prev` is the prior record's `hash`; only the first record of a brand-new
chain uses `prev = ""`. The first record of a rotated segment carries the
preceding segment's head, as specified under *Rotation* below.
Two exact rules pin the bytes so the chain is reproducible anywhere:
1. **Serialize the record with its own `hash` field set to `""`** (present,
empty), never omitted — the key set is identical before and after
hashing.
2. **Digest = `sha256_hex(prev + canonical(record_with_empty_hash))`**,
where `prev` is the previous record's `hash` string (`""` at genesis),
`+` is string concatenation, and `canonical` is the function above.
`ts_mono`, being a float, is serialized by Python's shortest-round-trip
`repr` via `json.dumps`; producers therefore emit it as a JSON number
they do not post-process. (All other fields are strings/ints/objects,
which serialize unambiguously.)
Editing or deleting record *n* changes its hash, so record *n+1*'s `prev`
no longer matches — the break is local and names the tampered record.
Verification needs only the journal itself (no keys), so it runs offline
and in CI.
#### Test vectors (normative)
Two records, reduced to the chain-relevant fields, demonstrate the exact
serialization and linkage. An implementation is conformant iff it
reproduces these bytes and hashes.
```
# Record 0 — brand-new chain genesis (prev = "")
canonical(record0, hash=""):
{"hash":"","id":"11111111-1111-4111-8111-111111111111","prev":"","seq":0,"type":"audit.segment_open"}
hash0 = sha256("" + canonical) =
942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83
# Record 1 — chains onto record 0 (prev = hash0)
canonical(record1, hash=""):
{"hash":"","id":"22222222-2222-4222-8222-222222222222","prev":"942ea5729bcac6efdbdea942396bfa574ab0d6ebf5615402595359422f2aeb83","seq":1,"type":"lifecycle.bottled_agent_start"}
hash1 = sha256(hash0 + canonical) =
bc082347680405fee50b60a9c304611aa026950b15d869b7e3ae56e1c451b856
# Tamper check: flip record0.type → recompute →
# 4553eda647f33f0c608cfea44be28efbbaca45ed30b873fcbd4405fa5ce737ed
# which no longer equals record1.prev (942ea5…) — the break is detected at record1.
```
The implementation PR ships these plus full-envelope vectors (every field
populated, and a redaction case) as committed fixtures, so a schema-version
bump that changes the bytes fails a golden test loudly.
### Ordering, idempotency, and duplicate handling
- **Ordering.** `(epoch, seq)` is a strict total order within one host's
native chain and, because
the writer is single, a strict order per bottle/activation within that
host — satisfying "at least strict causal order per activation/bottle".
`causation_id` records the explicit cause edges (a DAG) on top of the
total order, so a consumer can reconstruct request → decision → cutoff
even if unrelated events interleave between them. There is deliberately
no invented total order across imported host chains; a cross-host key is
`(host, epoch, seq)`, and consumers use correlation/causation edges where
causal ordering across hosts is known.
- **Idempotency.** `id` is the idempotency key. A producer that retries an
emit (e.g. after a writer restart mid-handoff) **reuses the same `id`**;
before appending, the writer checks a durable, host-wide id ledger that
spans every segment and survives restart/retention, and drops an id
already present. The ledger is operational metadata, not an audit source
of truth: after a crash it is reconciled from the journal before appends
resume, and retention preserves id tombstones after journal segments are
pruned. An in-memory set may cache the ledger but is never the authority.
The index additionally has a unique key on `id`; its `UPSERT` is
defensive and does not substitute for the pre-append check. Thus a
duplicate never enters the source-of-truth journal, double-counts, or
forks the chain.
- **Deduplication downstream.** Because `id` is stable across export and
replay, #324's cursor replay and any cross-host merge dedup on `id` — no
consumer needs to invent a second identity.
### Behavior across rotation, restart, import, truncation
- **Rotation.** At a segment boundary the writer opens a new segment file,
sets its `segment` id, and carries the rotated-out segment's head as the
new segment's first `prev` — so the chain is continuous *across* segments
(`prev = ""` is reserved for the first record of a brand-new chain) while
each file stays independently openable. `verify` walks segments in order
and checks the preceding-head-to-first-record link at each seam.
- **Restart.** Covered above: read last line → adopt its `hash` as `prev`,
bump `epoch`, reset `seq`. The chain never restarts even though the
counters do.
- **Import.** `audit import <segment>` appends an externally supplied
segment (e.g. recovered from another host or a backup). Import verifies
the incoming chain in isolation first. A continuation whose first `prev`
matches a known head extends that chain. A foreign chain is registered as
a separate immutable chain namespace rather than rewriting or grafting
its records (which would invalidate their hashes). Imported records keep
their original `host`, `id`, `epoch`, and `seq`; the index keys their
native order by `(host, epoch, seq)` so attribution is not laundered and
tuples from different hosts cannot collide.
- **Truncation.** A crash can leave a partial final line; `verify` reports
it as `truncated-tail` (recoverable — replay resumes from the last intact
record). A chain that ends before a persisted head, or a missing interior
`seq`, is reported as `gap`/`missing-suffix` (evidence of deletion, not a
clean crash). The two are distinguished so an operator can tell "power
loss" from "someone trimmed the log".
### Single writer; ordering across restarts
**Decided: one writer per host** (reviewed — the host controller owns it).
Producers hand events to the host controller, which is the sole appender,
so the chain has one well-defined total order and one `seq`/`epoch`
counter. This ties audit availability to the host controller being up,
which is acceptable because the host controller already gates every
lifecycle transition; per-producer chains are noted only as a future
scaling path, not built now.
**Restarts** are handled by the chain, not the clock. `ts_mono` resets to
~0 on every writer start, so it orders events only *within* one boot. On
start the writer:
1. reads the last line of the journal, adopts its `hash` as the next
record's `prev` (the chain is continuous across the restart), and
2. bumps `epoch` (persisted alongside the chain head) and resets `seq` to
0 for the new boot.
Total order within this host's native chain is therefore `(epoch, seq)`
monotonic across restarts by construction — with
`ts_event`/`ts_recorded` for human reading and `ts_mono` for sub-second
ordering inside an epoch. Imported chains retain their own
`(host, epoch, seq)` order and do not acquire a fictional order relative to
the local chain. A crash mid-append truncates at most the last (partial)
line; the verifier flags it and replay resumes from the last intact record.
### Journal (source of truth) + SQLite index (rebuildable)
- **Journal:** one append-only JSONL file per host (path from `paths.py`,
alongside `host_db_path()`), one canonical event per line, opened
`O_APPEND`. This is authoritative.
- **Index:** a new `audit_events` table via the existing `DbStore` /
`TableMigrations` machinery. It is a **derived cache, not a second source
of truth**: `audit rebuild` truncates and replays the journal,
re-verifying the chain as it goes, so a deleted or drifted DB is
regenerated from the journal with no data loss. On a `verify` failure
during rebuild it stops and reports rather than indexing past a break.
(This supersedes the free-standing `supervise_audit_entries` table, which
becomes a producer onto the new index.)
**Indexable fields** (columns + indices): `ts_event`, `ts_recorded`,
`type`, `host`, `bottle`, `bottled_agent`, `activation`, `actor`,
`outcome`, `sensitivity`, `correlation_id`, `causation_id`, plus two
event-specific projections promoted out of `payload` for query —
`repository` and `commit_sha` (populated for `forge.*`/`commit.*`, null
otherwise). The unique event id and native-order index are respectively
`id` and `(host, epoch, seq)`; a local-only `ingest_seq` provides stable
display order when a query intentionally mixes chains without pretending
that it is causal order. The full canonical record is stored verbatim in a
`raw` column so the index never loses fidelity to the journal.
**Local query surface** — `audit query`, no egress, no paid platform:
```
audit query \
[--since T] [--until T] [--type egress.*] [--host H] [--bottle B] \
[--activation A] [--agent SLUG] [--actor ID] [--outcome blocked] \
[--repository R] [--correlation-id C] [--commit SHA] \
[--follow BOTTLE] # one host chain's events in (epoch, seq) order
[--json | --table]
audit verify [--segment S] # offline chain check; exit non-zero on any break
audit rebuild # drop + replay journal → index
audit import <segment> # graft an external segment (see above)
```
Type filters accept a `group.*` glob. A read-only local HTTP endpoint
mirrors the same filters for the future review console; both are pure reads
over the index and can never mutate the journal.
### Event registry (initial)
Dotted `type` names, grouped. The registry is a table mapping each type to
its required `payload` keys, its `untrusted` keys (if any), a default
`sensitivity`, and its correlation behavior — so producers and the verifier
agree on shape and "trusted vs claimed" is pinned per type. Initial
coverage (the issue's mandated set):
| Group / type | Producer | Required `payload` (trusted) | `untrusted` | Default sensitivity |
|---|---|---|---|---|
| **lifecycle.*** — `bottled_agent_start` / `_stop` / `_crash` | host-controller (#468) | `manifest_digest`, `exit` (for stop/crash) | — | normal |
| **hostctl.*** — `broker_launch`, `broker_teardown`, `broker_reject` | host-controller (#468) | `op`, `request_digest` | — | security |
| **decision.*** — `proposed`, `resolved` | supervise | `tool`, `operator_action`, `justification`, `diff_digest` | `agent_rationale` | security |
| **egress.*** — `request`, `decision`, `cutoff`, `anomaly` | egress proxy | `route_id`, `detector` (on match), `bytes` (cutoff) | `reason`, `target_claimed` | security |
| **forge.*** — `push_accepted`, `push_rejected`, `pr_opened` | git-gate | `repository`, `ref`, `gitleaks_result` | `title`, `description` | security |
| **commit.signed** (#480) | git-gate | `repository`, `commit_sha`, `activation_key_id`, `signature_ref` | `commit_message` | security |
| **auth.*** — `token_minted`, `token_rejected`, `authz_denied` | control plane | `role`, `token_id`, `reason_code` | — | security |
| **audit.*** — `segment_open`, `verify_failed`, `truncation_detected`, `export_failed` | audit writer/verifier | `segment`, `detail` | — | security |
Notes:
- **`egress.request` vs `egress.decision`** share a `correlation_id`; the
`decision`'s `causation_id` points at the `request`, and a later `cutoff`
chains onto the `decision` — so a flow is reconstructable.
- **`audit.*` self-events** make the audit subsystem audit itself: a failed
verification, a detected truncation, or a dropped export is itself a
chained, tamper-evident record — you cannot silence the alarm without
breaking the chain that carries it.
- **Free-text and remote-echoed fields are always `untrusted`** (`reason`,
`agent_rationale`, PR `title`/`description`, `target_claimed`), because
they originate in the bottle or a remote response; the host-observed
counterpart (`resource`, `outcome`, `gitleaks_result`) is the trusted
fact.
**Sensitivity + redaction per type.** Every type's default `sensitivity`
is listed above; a producer may raise it (never lower it). `restricted`
events keep their `payload` in the journal but the export projection ships
only the envelope + a payload digest unless the consumer is authorized —
so a `security`/`restricted` record is still counted and correlated
downstream without leaking its body. The credential-shape redaction rules
(next) apply to **every** type regardless of sensitivity.
**Schema evolution & backward-compatible readers.** The registry is
append-only: **adding** a type, an optional `payload` key, or an
`untrusted` key does **not** bump `v`; readers ignore unknown fields
(forward-compatible) and treat absent optional fields as `null`.
**Removing** or **re-typing** a field, or making an optional field
required, bumps `v`. A reader declares the max `v` it understands and
refuses to *interpret* a higher-`v` record, but the **verifier is
version-agnostic** — the hash covers whatever fields exist, so chain
integrity is checkable across versions without understanding semantics.
Every `v` bump ships a migration note and updated golden vectors.
### Redaction rule
Redaction runs at the envelope boundary, before a record is written, in two
layers:
1. **Key deny-list (structural).** A field whose *key* matches a known
credential shape (`token`, `secret`, `password`, `authorization`,
`*_key`) is refused — the producer must pass a reference (a token *id*
or `sha256` fingerprint), never the raw value. `auth.token_minted`
therefore records the token id and role, not the JWT. This is the
primary guard: it is cheap, deterministic, and catches the intended
mistake (a producer stuffing a credential into a named field).
2. **Value scan — reuse the egress DLP detectors.** Per review, the value
layer reuses the *same* deterministic credential-shape detectors the
egress proxy already ships:
`bot_bottle/gateway/egress/dlp_detectors.py`
`scan_token_patterns` / `redact_tokens` (and `scan_known_secrets` for
host-known secret material). They are pure-Python, mitmproxy-free, and
already the project's source of truth for "what a leaked credential
looks like," so a single detector set governs both what may leave over
the wire and what may land in the journal — they can't drift apart.
**Scoped deliberately:** only the pattern/known-secret detectors are
reused, **not** `scan_entropy`. Entropy scoring is tuned for large
streamed request bodies; on the short, high-entropy structured values an
audit event legitimately carries (hashes, uuids, base64 ids) it would
false-positive and start redacting the very fingerprints the log needs.
So the shared layer is the deterministic detectors; entropy stays an
egress-only concern. (This is the "evaluate how reasonable that is" from
review: reuse the deterministic detectors — yes; share the entropy
heuristic — no.)
On a value-layer match the default is **redact** (scrub to a placeholder
and keep the event) rather than drop, so a producer bug can never make an
audit event vanish; the key deny-list stays a hard refusal because a
credential in a named field is always a producer bug worth surfacing.
**No raw-payload capture by default.** The envelope carries *decisions and
metadata*, not traffic. Prompts, model responses, request/response bodies,
and file contents are **not** recorded unless a producer opts a specific,
reviewed field in — and such a field is `untrusted` and subject to both
redaction layers. This keeps the audit log from becoming a covert copy of
the very data the sandbox exists to contain (the issue's "unsafe payload
capture" non-goal).
### Export / interoperability (CloudEvents, OpenTelemetry Logs)
#487 requires the envelope to map onto the **OpenTelemetry Logs data
model** and/or a **CloudEvents JSON** envelope *without losing integrity or
attribution semantics*. The flattened shape (one `untrusted` region,
everything else trusted at top level) does **not** conflict with either — it
maps *more* cleanly than a nested `trusted`/`untrusted` pair would, because
both target models expect a flat set of top-level fields plus one payload
subtree.
**CloudEvents.** Context attributes MUST be scalar simple types — a map
cannot be a context attribute — so a nested `trusted` block would have had
to be flattened for CloudEvents anyway. Our flat top level maps directly:
`id``id`, `type``type`, `producer`+`host``source`,
`bottled_agent``subject`, `ts_event``time` (`ts_recorded` as an extension); the integrity/chain fields
(`epoch`, `seq`, `prev`, `hash`, `v`) ride as **extension attributes**
(scalars — legal). The `untrusted` map goes in `data`. Only mechanical
transform needed: extension attribute names must be lowercase-alphanumeric,
so `bottled_agent`/`ts_mono`/etc. are renamed at export (e.g. a
`botbottle`-prefixed form) — a naming rule, not a schema conflict.
**OpenTelemetry Logs.** `ts_event``Timestamp`, `ts_recorded``ObservedTimestamp`; `type`→the `event.name`
attribute; the flat trusted fields → `Attributes` under a `botbottle.*`
namespace (`botbottle.bottled_agent`, `botbottle.producer`,
`botbottle.chain.hash`, …); `untrusted.*``Attributes` under
`botbottle.untrusted.*` (or `Body`). OTel attributes are a dotted map that
happily carries the nested subtree.
**Attribution is preserved** precisely because the boundary is now
structural: on export, top-level fields become trusted context/attributes
and the `untrusted` subtree stays a single, clearly-named region — so a
downstream consumer still sees exactly which fields an agent claimed.
Nothing agent-claimed is promoted to a trusted-looking position.
**Integrity has one deliberate caveat.** CloudEvents/OTel are
representation envelopes with their own (or no) canonicalization; `hash`
and `prev` are computed over **our** canonical JSON, not over the exported
form. So the chain fields travel *as data* for reference, but
tamper-evidence is always verified against the **native journal** (the
source of truth) — never re-derived from an exported CloudEvents/OTel
record, whose key ordering / number formatting the exporter may change.
Export is thus a lossless-for-attribution **projection** that carries the
integrity fields along; verification stays on the canonical journal. This
satisfies "without losing integrity or attribution semantics": both are
carried, neither is *relied upon* in the foreign format.
The export adapters themselves are follow-up implementation — this PRD
fixes the *schema* so that projection is a field re-map, never a reformat.
#### The #324 delivery contract (payload, cursor, backpressure)
#324 transports events off-box; it must not invent a second envelope. This
PRD fixes the contract it depends on:
- **Payload.** #324 ships the **native canonical record verbatim** (the
exact bytes the hash covers), optionally wrapped in the CloudEvents
projection whose `data` *is* that record. Either way the integrity fields
travel intact and the receiver can verify against the same bytes.
- **Cursor.** Export maintains one cursor per native host chain:
`(host, epoch, seq)` (equivalently that chain's last exported `hash`).
It advances **only on acknowledgement**, so delivery is at-least-once and
gap-free; a crash re-sends from the last acked cursor. Imported foreign
chains use independent cursors and never share a bare `(epoch, seq)`
namespace with the local chain.
- **Idempotency / replay.** Dedup is on `id` (stable across replay), so
at-least-once delivery is safe — the receiver collapses re-sends.
- **Backpressure.** The outbox is the journal itself plus a cursor; when
the endpoint is slow the cursor simply lags — the writer never blocks on
export, and audit never applies backpressure to the data plane it
records.
- **Retention interaction.** Retention/rotation **must not** prune a
segment whose records are still behind the export cursor; the reaper
honors `min(cursor)` across all configured consumers. (The schedule
itself stays the retention follow-up; this is the *ordering* constraint
that follow-up must respect.)
#### #480 signed-commit attribution maps in without weakening it
#480 binds a commit's bytes to a per-activation signing key. It maps to the
`commit.signed` event: `payload` carries `repository`, `commit_sha`,
`activation_key_id`, and a `signature_ref` (the detached-signature
location or its digest) — **not** the private key and not a re-derived
signature. The audit event therefore *references and timestamps* #480's
existing byte-to-activation-key proof inside the tamper-evident chain; it
does not re-implement or replace it, so #480's guarantee is unweakened —
the signature still verifies against the commit bytes independently, and
the audit record adds only "this binding was observed at this point in the
chain". The trusted `actor`/`activation` fields and the `commit_sha`
payload are host-observed at the gate, so attribution cannot be forged by
the committing agent.
## Implementation chunks
1. **(this PR — PRD only.)** The contract above. No code; scheduled to land
right after #468.
2. **Envelope + canonical + chain core.** `AuditEvent` dataclass,
`canonical()`, chain hashing, and the single-writer journal appender in
`bot_bottle/store/` (reusing `sha256_hex`); redaction wired to the
existing `gateway/egress/dlp_detectors` (`scan_token_patterns` /
`redact_tokens`); embed the git SHA at build so `engine` is populated
(only `version = "0.1.0"` exists in `pyproject.toml` today — the build
must stamp the SHA); unit tests for determinism, chain-break detection,
`epoch`/`seq` continuity across a simulated restart, and redaction of
both a deny-listed key and a token-shaped value.
3. **SQLite index + `audit` CLI.** New `audit_events` migration (indexable
fields above); replay-from-journal; offline chain verifier
(`truncated-tail` vs `gap`); `query` / `verify` / `rebuild` / `import`;
idempotent `UPSERT` by `id`.
4. **Host controller as first producer (#468).** Wire
`lifecycle.bottled_agent_*` and `hostctl.*` emission into the host
controller; establish the `epoch` bump + chain-head carry + segment
rotation on writer restart here (it owns the single writer).
5. **Migrate existing producers.** Re-emit supervise `decision.*` (retiring
the standalone `supervise_audit_entries` shape behind the index), egress
`egress.*`, git-gate `forge.*` + `commit.signed` (#480), control-plane
`auth.*`; add the `audit.*` self-events (verify/truncation/export
failure).
6. **CloudEvents / OTel export adapters + #324 delivery.** Projection layer
(field re-map per *Export / interoperability*) plus the outbox cursor,
ack-driven advance, and retention-ordering guard the #324 contract
specifies.
7. **(follow-up.)** Cross-host merge transport; per-writer signing +
external anchoring on the chain head; retention/rotation *schedule*.
## Acceptance-criteria coverage (#487)
The issue defines the contract; implementation is explicitly split into
follow-up PRs. This PRD is the durable decision record; each acceptance box
maps to a section:
| #487 acceptance criterion | Where |
|---|---|
| Durable PRD defines versioned envelope + initial registry | *The envelope*, *Event registry* |
| Canonical JSON + hash-chain rules, unambiguous, with test vectors | *Canonical serialization + hash chain**Test vectors* |
| Trust provenance explicit for every common + event-specific field | *Trust provenance of every common field*; per-type `untrusted` in *Event registry* |
| Redaction prohibits credentials / raw secrets / unsafe capture by default | *Redaction rule*; `untrusted`-only claims; sensitivity classes |
| JSONL journal canonical; SQLite index fully rebuildable | *Journal + SQLite index* (`audit rebuild`) |
| Minimum local search/query contract | *Journal + SQLite index**Local query surface* |
| #324 can transport/replay without a second envelope | *The #324 delivery contract* |
| #480 maps in without weakening its byte-to-activation-key guarantee | *#480 signed-commit attribution maps in…* |
| Schema evolution + backward-compatible readers | *Schema evolution & backward-compatible readers* |
| Integrity detects modification / deletion / reorder / bad continuation | *Test vectors* (tamper), *Behavior across rotation…truncation*, `audit verify` |
Two acceptance items are **specified here, implemented later** by design
(the issue permits this): the concrete test-vector *fixtures* and the
`audit` CLI land in impl chunks 23; the #324 outbox lands in chunk 6.
Nothing in the contract is left undefined — only its code is deferred.
## Resolved in review (#495)
- **Single writer per host — decided.** The host controller owns the sole
appender; per-producer chains are a future scaling path only. (Design →
*Single writer; ordering across restarts*.)
- **Restarts — decided.** An `epoch` counter (bumped per writer boot) plus
carrying the last chain head as the next `prev` gives a total order of
`(epoch, seq)` that survives restarts; `ts_mono` orders only within an
epoch. (Design → *ordering across restarts*.)
- **Flatten to one `untrusted` region — decided.** Everything outside
`untrusted` (chain metadata, `producer`/`host`, `bottled_agent`, `ts_*`)
is trusted by construction, so the separate `trusted` sub-block is
removed; a field is trusted unless deliberately placed under `untrusted`.
(Design → *The envelope*.)
- **Subject term is `bottled_agent` everywhere** — the top-level field and
the `lifecycle.bottled_agent_*` leaf names. (Design → *The envelope* /
*Event registry*.)
- **Retention head-carry — yes.** When a journal segment is rotated out,
the new segment's first record carries the rotated-out head as `prev`, so
the verifier still trusts the current head across a rotation. (Folds into the
retention follow-up.)
- **Redaction reuses the egress detectors — yes, scoped.** Reuse the
deterministic `dlp_detectors` (`scan_token_patterns` / `redact_tokens` /
`scan_known_secrets`); exclude `scan_entropy` as brittle on the short,
high-entropy structured values audit records carry. (Design → *Redaction
rule*.)
## Open questions
- **Value-scan cost on the hot path.** The single writer runs the reused
detectors on every event's `untrusted` block inline. Is that cheap enough
at lifecycle-event volume, or should the value scan move to index-build
time (journal stays raw, index stores the redacted view)? Leaning inline
so the raw journal never contains a leaked value in the first place.
- **`epoch` persistence location.** Store the per-writer `epoch` + chain
head in the SQLite index (rebuildable, but then the writer needs the DB
at boot) or in a tiny sidecar file next to the journal (independent of
the index)? Leaning sidecar, so the writer can start and append without
the index present.
@@ -4,7 +4,7 @@ Spike branch: `spike/rootless-docker-macos` (`a4d8461`)
**Outcome:** the podman recommendation below shipped as the
`nested_containers` bottle flag — see
[`docs/prds/prd-new-nested-containers.md`](../prds/prd-new-nested-containers.md).
[`docs/prds/0075-nested-containers.md`](../prds/0075-nested-containers.md).
The `docker_access` name used throughout the spike text was renamed on the
way in; it granted no access to anything on the host.
Executable
+123
View File
@@ -0,0 +1,123 @@
#!/bin/sh
# bot-bottle quick installer.
#
# Usage:
# curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh
#
# Python-native users can skip this entirely:
# pipx install bot-bottle # from a checkout or a published index
# uv tool install bot-bottle
#
# This script is a thin bootstrapper: it checks prerequisites, installs the
# package with pipx (falling back to pip --user), creates the config dir, and
# runs `bot-bottle doctor`. It is idempotent (safe to re-run) and never uses
# sudo. It does NOT install Docker or a VM backend for you — `doctor` reports
# what's missing after install.
set -eu
PACKAGE_SPEC="${BOT_BOTTLE_INSTALL_SPEC:-git+https://gitea.dideric.is/didericis/bot-bottle.git}"
MIN_PYTHON_MAJOR=3
MIN_PYTHON_MINOR=11
say() {
printf 'bot-bottle install: %s\n' "$*" >&2
}
die() {
say "error: $*"
exit 1
}
# --- prerequisites -----------------------------------------------------------
command -v python3 >/dev/null 2>&1 \
|| die "python3 ${MIN_PYTHON_MAJOR}.${MIN_PYTHON_MINOR}+ is required but was not found"
python3 - "$MIN_PYTHON_MAJOR" "$MIN_PYTHON_MINOR" <<'PY' || die "python3 ${MIN_PYTHON_MAJOR}.${MIN_PYTHON_MINOR} or newer is required"
import sys
want = (int(sys.argv[1]), int(sys.argv[2]))
raise SystemExit(0 if sys.version_info[:2] >= want else 1)
PY
# Installing a `git+` spec (the default) shells out to git under the hood,
# whether via pipx or pip. Fail early with a clear message rather than deep
# inside the installer's output.
case "${PACKAGE_SPEC}" in
git+*|*.git)
command -v git >/dev/null 2>&1 || die \
"git is required to install from '${PACKAGE_SPEC}'. Install git, or set "\
"BOT_BOTTLE_INSTALL_SPEC to a non-git spec (e.g. a wheel path or a package index name)."
;;
esac
# The pip fallback needs a usable pip. Externally-managed interpreters
# (PEP 668, common on Debian/Ubuntu/Homebrew) reject `pip install --user`;
# pipx sidesteps that, so recommend it when pip can't be used.
if ! command -v pipx >/dev/null 2>&1; then
python3 -m pip --version >/dev/null 2>&1 || die \
"neither pipx nor a usable 'python3 -m pip' was found. Install pipx "\
"(recommended): 'python3 -m pip install --user pipx' or your OS package manager."
if python3 - <<'PY'
import os
import sys
import sysconfig
# PEP 668: an EXTERNALLY-MANAGED marker in the stdlib dir means pip refuses
# to install into this interpreter without --break-system-packages.
marker = os.path.join(sysconfig.get_path("stdlib"), "EXTERNALLY-MANAGED")
raise SystemExit(0 if os.path.exists(marker) else 1)
PY
then
die "this Python is externally managed (PEP 668), so 'pip install --user' is "\
"blocked. Install pipx and re-run: 'python3 -m pip install --user --break-system-packages pipx', "\
"then 'pipx ensurepath'."
fi
fi
# --- config directories ------------------------------------------------------
mkdir -p \
"${HOME}/.bot-bottle/agents" \
"${HOME}/.bot-bottle/bottles" \
"${HOME}/.bot-bottle/contrib"
# --- install -----------------------------------------------------------------
if command -v pipx >/dev/null 2>&1; then
say "installing with pipx"
pipx install --force "${PACKAGE_SPEC}"
else
say "pipx not found; installing with 'python3 -m pip install --user'"
python3 -m pip install --user --upgrade "${PACKAGE_SPEC}"
fi
# --- locate the entry point --------------------------------------------------
# The pip --user scripts directory is platform-specific: ~/.local/bin on Linux,
# but ~/Library/Python/<X.Y>/bin on a python.org macOS interpreter. Ask the
# interpreter for its own user-scheme scripts dir instead of hardcoding.
USER_SCRIPTS="$(python3 - <<'PY'
import sysconfig
print(sysconfig.get_path("scripts", sysconfig.get_preferred_scheme("user")))
PY
)"
if command -v bot-bottle >/dev/null 2>&1; then
BOT_BOTTLE_BIN="bot-bottle"
elif [ -n "${USER_SCRIPTS}" ] && [ -x "${USER_SCRIPTS}/bot-bottle" ]; then
BOT_BOTTLE_BIN="${USER_SCRIPTS}/bot-bottle"
say "note: add ${USER_SCRIPTS} to your PATH to run 'bot-bottle' directly"
else
die "bot-bottle was installed but is not on PATH; add ${USER_SCRIPTS:-your user scripts dir} to PATH and re-run"
fi
# --- verify ------------------------------------------------------------------
say "running '${BOT_BOTTLE_BIN} doctor'"
if "${BOT_BOTTLE_BIN}" doctor; then
say "done. Run '${BOT_BOTTLE_BIN} --help' to get started."
else
say "install completed, but 'doctor' reported unmet prerequisites (see above)."
say "resolve them, then re-run '${BOT_BOTTLE_BIN} doctor'."
fi
+38 -1
View File
@@ -4,5 +4,42 @@ build-backend = "setuptools.build_meta"
[project]
name = "bot-bottle"
version = "0.0.0"
version = "0.1.0"
description = "Self-hosted sandbox for running AI coding agents with egress controls"
readme = "README.md"
requires-python = ">=3.11"
license = { text = "Apache-2.0" }
authors = [{ name = "didericis" }]
keywords = ["ai", "agents", "sandbox", "security", "egress"]
classifiers = [
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3 :: Only",
"Operating System :: POSIX :: Linux",
"Operating System :: MacOS",
]
# The package itself has no runtime pip dependencies (stdlib-only); the
# only language runtime is the Python interpreter. Keep this empty.
dependencies = []
[project.urls]
Homepage = "https://gitea.dideric.is/didericis/bot-bottle"
Source = "https://gitea.dideric.is/didericis/bot-bottle"
[project.scripts]
bot-bottle = "bot_bottle.cli:main"
[tool.setuptools.packages.find]
include = ["bot_bottle*"]
# Non-Python assets the runtime reads from inside the package (container
# build contexts, entrypoints, netpool defaults). Keep in sync with the
# files shipped under bot_bottle/; test_pyproject.py asserts they exist.
[tool.setuptools.package-data]
bot_bottle = [
"gateway/egress/entrypoint.sh",
"contrib/claude/Dockerfile",
"contrib/codex/Dockerfile",
"contrib/pi/Dockerfile",
"backend/firecracker/netpool.defaults.env",
"backend/macos_container/nested-containers-init.sh",
]
+3
View File
@@ -5,3 +5,6 @@
pylint>=3.0.0
pyright>=1.1.411
coverage>=7.0.0
# PEP 517 build front-end used by tests/unit/test_wheel_install.py to build and
# install a real wheel (proves the installed distribution is self-contained).
build>=1.0.0
+11 -9
View File
@@ -54,18 +54,20 @@ def check_pull_request(event: dict[str, Any], api: GiteaApi) -> list[str]:
pull = event["pull_request"]
errors: list[str] = []
labels = pull.get("labels") or []
if labels:
errors.append(
"PRs must be unlabeled; put tracker metadata on the linked issue "
f"(found: {', '.join(label['name'] for label in labels)})."
)
numbers = deliberate_issue_numbers(pull.get("title", ""), pull.get("body", ""))
if not numbers:
if labels and numbers:
errors.append(
"PR must reference an issue with Closes/Fixes/Resolves #N, "
"Part of #N, Related to #N, Refs #N, or References #N."
"PR must use exactly one tracking mode: remove PR labels when "
"linking an issue, or remove the issue reference when labels "
"belong on the PR."
)
if not numbers:
if not labels:
errors.append(
"PR must either have a label or reference an issue with "
"Closes/Fixes/Resolves #N, Part of #N, Related to #N, "
"Refs #N, or References #N."
)
return errors
real_issues = 0
+46
View File
@@ -0,0 +1,46 @@
"""Build shim. Project metadata lives in ``pyproject.toml``; this only adds a
build step that copies the root-level build resources (the Dockerfiles, the
nix netpool module, the netpool script, and ``pyproject.toml``) into
``bot_bottle/_resources/`` so an installed wheel is self-contained and can
build its gateway/infra/orchestrator images without a source checkout.
Kept in sync with ``bot_bottle.resources.BUNDLED_RESOURCES`` the
``test_resources`` suite guards against drift between the two lists.
"""
from __future__ import annotations
import shutil
from pathlib import Path
from setuptools import setup
from setuptools.command.build_py import build_py
_ROOT = Path(__file__).resolve().parent
# Must match bot_bottle.resources.BUNDLED_RESOURCES (paths relative to root).
_BUNDLED_RESOURCES = (
"pyproject.toml",
"Dockerfile.gateway",
"Dockerfile.orchestrator",
"Dockerfile.orchestrator.fc",
"nix/firecracker-netpool.nix",
"scripts/firecracker-netpool.sh",
)
class _BundleResources(build_py):
"""Copy the root-level build resources into the built package tree so they
ship inside the wheel under ``bot_bottle/_resources/``."""
def run(self) -> None:
super().run()
pkg_resources = Path(self.build_lib) / "bot_bottle" / "_resources"
for rel in _BUNDLED_RESOURCES:
src = _ROOT / rel
dst = pkg_resources / rel
dst.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(src, dst)
setup(cmdclass={"build_py": _BundleResources})
+98
View File
@@ -0,0 +1,98 @@
"""Unit: `bot-bottle doctor` host prerequisite checks (ADR 0004).
`doctor` is a store-free diagnostic it must run on a fresh install
before any DB migration, and its exit code gates only the two hard
prerequisites (Python and at least one *ready* backend). The config-dir
check is advisory and never affects the exit code.
Backend readiness is probed with `is_backend_ready()` (a full status()
check), not the cheap PATH-only `is_backend_available()` a host with a
stopped daemon or half-configured backend must not report `ok`.
"""
from __future__ import annotations
import io
import tempfile
import unittest
from contextlib import redirect_stdout
from pathlib import Path
from unittest.mock import patch
from bot_bottle.cli.commands import doctor
def _run(argv: list[str] | None = None) -> tuple[int, str]:
buf = io.StringIO()
with redirect_stdout(buf):
code = doctor.cmd_doctor(argv or [])
return code, buf.getvalue()
class TestDoctor(unittest.TestCase):
def test_passes_when_python_and_backend_ready(self):
with patch.object(doctor, "known_backend_names", return_value=("docker",)), \
patch.object(doctor, "is_backend_ready", return_value=True):
code, out = _run()
self.assertEqual(0, code)
self.assertIn("ok: python", out)
self.assertIn("ok: backend: docker: ready", out)
def test_fails_when_no_backend_ready(self):
# The regression the reviewer flagged: a backend whose binary is on PATH
# but whose daemon/pool isn't ready must NOT pass. is_backend_ready is
# the full status() check, so returning False here means "not ready".
with patch.object(doctor, "known_backend_names", return_value=("docker", "firecracker")), \
patch.object(doctor, "is_backend_ready", return_value=False):
code, out = _run()
self.assertEqual(1, code)
self.assertIn("fail: backend", out)
self.assertIn("warn: backend: docker: not ready", out)
def test_passes_when_at_least_one_backend_ready(self):
# docker not ready, firecracker ready → overall pass, mixed report.
def ready(name: str, *, quiet: bool = False) -> bool:
del quiet
return name == "firecracker"
with patch.object(doctor, "known_backend_names", return_value=("docker", "firecracker")), \
patch.object(doctor, "is_backend_ready", side_effect=ready):
code, out = _run()
self.assertEqual(0, code)
self.assertIn("warn: backend: docker: not ready", out)
self.assertIn("ok: backend: firecracker: ready", out)
def test_fails_when_python_too_old(self):
# Force the version gate to fail without touching the interpreter.
with patch.object(doctor, "MIN_PYTHON", (99, 0)), \
patch.object(doctor, "known_backend_names", return_value=("docker",)), \
patch.object(doctor, "is_backend_ready", return_value=True):
code, out = _run()
self.assertEqual(1, code)
self.assertIn("fail: python", out)
def test_missing_config_dir_is_advisory_not_fatal(self):
# A missing ~/.bot-bottle warns but must not fail. Point home at a
# fresh empty dir so the shared suite HOME (which other tests may
# populate) can't turn this into an "ok: config".
with tempfile.TemporaryDirectory() as tmp, \
patch.object(doctor.Path, "home", return_value=Path(tmp)), \
patch.object(doctor, "known_backend_names", return_value=("docker",)), \
patch.object(doctor, "is_backend_ready", return_value=True):
code, out = _run()
self.assertEqual(0, code)
self.assertIn("warn: config", out)
def test_present_config_dir_reports_ok(self):
with tempfile.TemporaryDirectory() as tmp, \
patch.object(doctor.Path, "home", return_value=Path(tmp)), \
patch.object(doctor, "known_backend_names", return_value=("docker",)), \
patch.object(doctor, "is_backend_ready", return_value=True):
(Path(tmp) / ".bot-bottle").mkdir()
code, out = _run()
self.assertEqual(0, code)
self.assertIn("ok: config", out)
if __name__ == "__main__":
unittest.main()
+101
View File
@@ -0,0 +1,101 @@
"""Unit: install.sh bootstrapper contract.
The installer is a thin, sudo-free, idempotent bootstrapper. These are
static checks on the script text (no network / no real install) so CI can
run them anywhere: it must be executable, fail-fast, never call sudo,
create the config tree, install the package, and verify with `doctor`.
"""
from __future__ import annotations
import os
import sysconfig
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
INSTALL_SH = REPO_ROOT / "install.sh"
class TestInstallScript(unittest.TestCase):
@classmethod
def setUpClass(cls):
cls.text = INSTALL_SH.read_text()
def test_exists_and_executable(self):
self.assertTrue(INSTALL_SH.is_file())
self.assertTrue(os.access(INSTALL_SH, os.X_OK), "install.sh must be executable")
def test_posix_shebang_and_failfast(self):
first = self.text.splitlines()[0]
self.assertEqual("#!/bin/sh", first)
self.assertIn("set -eu", self.text)
def test_never_uses_sudo(self):
# Only executable lines matter; the header comment may mention sudo.
code = [
ln for ln in self.text.splitlines()
if ln.strip() and not ln.lstrip().startswith("#")
]
self.assertNotIn("sudo", "\n".join(code))
def test_creates_config_tree(self):
self.assertIn(".bot-bottle/agents", self.text)
self.assertIn(".bot-bottle/bottles", self.text)
def test_installs_via_pipx_with_pip_fallback(self):
self.assertIn("pipx install", self.text)
self.assertIn("pip install --user", self.text)
def test_runs_doctor_after_install(self):
self.assertIn("doctor", self.text)
def test_install_spec_is_overridable(self):
# Tests / local installs point BOT_BOTTLE_INSTALL_SPEC at a checkout.
self.assertIn("BOT_BOTTLE_INSTALL_SPEC", self.text)
def test_requires_git_for_git_specs(self):
# A git+ / .git spec (the default) shells out to git; the script must
# gate on it rather than failing opaquely inside pipx/pip.
self.assertIn("command -v git", self.text)
self.assertIn("git+*|*.git", self.text)
def test_checks_pip_usable_before_fallback(self):
self.assertIn("python3 -m pip --version", self.text)
def test_detects_externally_managed_python(self):
# PEP 668: 'pip install --user' is blocked on externally-managed
# interpreters; the script must detect this and point at pipx.
self.assertIn("EXTERNALLY-MANAGED", self.text)
self.assertIn("pipx", self.text)
def test_resolves_user_scripts_dir_not_hardcoded(self):
# The pip --user scripts dir differs by platform; the script must ask
# the interpreter (sysconfig + the preferred *user* scheme) rather than
# hardcoding Linux's ~/.local/bin (which is wrong on macOS python.org).
self.assertIn("get_preferred_scheme", self.text)
self.assertIn("sysconfig", self.text)
# No hardcoded Linux path in executable lines (a comment may mention it).
code = "\n".join(
ln for ln in self.text.splitlines()
if ln.strip() and not ln.lstrip().startswith("#")
)
self.assertNotIn(".local/bin", code)
def test_macos_user_scheme_is_not_dot_local_bin(self):
# The case the fix exists for: a python.org macOS interpreter uses the
# osx_framework_user scheme, whose scripts land under
# ~/Library/Python/<X.Y>/bin — NOT ~/.local/bin. Drive the same
# sysconfig lookup install.sh uses, with a mac-like userbase, to prove
# it resolves a non-~/.local/bin directory.
self.assertIn("osx_framework_user", sysconfig.get_scheme_names())
scripts = sysconfig.get_path(
"scripts", "osx_framework_user",
vars={"userbase": "/Users/dev/Library/Python/3.11"},
)
self.assertEqual("/Users/dev/Library/Python/3.11/bin", scripts)
self.assertNotIn("/.local/bin", scripts)
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -283,7 +283,7 @@ class TestBuildOrLoadImages(unittest.TestCase):
images = launch_mod.build_or_load_images(plan)
build.assert_called_once_with(
"agent:base", launch_mod._REPO_DIR, # pylint: disable=protected-access
"agent:base", str(launch_mod.resources.build_root()),
dockerfile="/repo/Dockerfile",
)
derived.assert_called_once_with("agent:base", build)
+1 -1
View File
@@ -1,4 +1,4 @@
"""Unit tests for per-bottle egress secret encryption (PRD prd-new-secret-provider)."""
"""Unit tests for per-bottle egress secret encryption (PRD 0080)."""
from __future__ import annotations
+45
View File
@@ -0,0 +1,45 @@
"""Unit: pyproject.toml packaging contract.
Guards the install/distribution surface: the console-script entry point,
the stdlib-only (empty) dependency list, and that every package-data glob
still points at a file that exists (so an installed wheel isn't missing a
Dockerfile or entrypoint the runtime reads).
"""
from __future__ import annotations
import tomllib
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
PYPROJECT = REPO_ROOT / "pyproject.toml"
class TestPyproject(unittest.TestCase):
@classmethod
def setUpClass(cls):
with PYPROJECT.open("rb") as fh:
cls.data = tomllib.load(fh)
def test_entry_point_targets_cli_main(self):
scripts = self.data["project"]["scripts"]
self.assertEqual("bot_bottle.cli:main", scripts["bot-bottle"])
def test_no_runtime_dependencies(self):
# AGENTS.md: the package has no runtime pip dependencies.
self.assertEqual([], self.data["project"]["dependencies"])
def test_requires_python_311(self):
self.assertEqual(">=3.11", self.data["project"]["requires-python"])
def test_package_data_files_exist(self):
pkg_data = self.data["tool"]["setuptools"]["package-data"]["bot_bottle"]
self.assertTrue(pkg_data, "expected package-data entries")
for rel in pkg_data:
path = REPO_ROOT / "bot_bottle" / rel
self.assertTrue(path.is_file(), f"package-data missing: {path}")
if __name__ == "__main__":
unittest.main()
+184
View File
@@ -0,0 +1,184 @@
"""Unit: bot_bottle.resources — build-resource resolution for both a source
checkout and an installed wheel.
The checkout path is what the whole test suite already runs under; the wheel
path is exercised here by faking an installed layout (a package dir with a
bundled ``_resources/`` and no sibling Dockerfiles) and asserting that
``build_root()`` stages a repo-root-shaped context. See
``test_wheel_install.py`` for the end-to-end build+install check.
"""
from __future__ import annotations
import tempfile
import unittest
from contextlib import contextmanager
from pathlib import Path
from unittest.mock import patch
from bot_bottle import resources
from tests.unit import use_bottle_root
class TestCheckoutMode(unittest.TestCase):
"""The environment the suite runs in: a real source checkout."""
def test_is_source_checkout(self):
self.assertTrue(resources.is_source_checkout())
def test_build_root_is_repo_root(self):
root = resources.build_root()
self.assertTrue((root / "bot_bottle").is_dir())
self.assertTrue((root / "pyproject.toml").is_file())
self.assertTrue((root / "Dockerfile.gateway").is_file())
def test_resource_helpers_resolve(self):
self.assertTrue(resources.dockerfile("Dockerfile.gateway").is_file())
self.assertTrue(resources.nix_netpool_module().is_file())
self.assertTrue(resources.netpool_script().is_file())
def test_bundled_resources_all_exist_at_root(self):
# Drift guard: every path setup.py bundles must exist in the checkout.
root = resources.build_root()
for rel in resources.BUNDLED_RESOURCES:
self.assertTrue((root / rel).is_file(), f"missing bundled resource: {rel}")
class TestWheelMode(unittest.TestCase):
"""Fake an installed wheel: a package dir with _resources/ and no
checkout Dockerfiles beside it."""
def _fake_install(self, tmp: Path) -> Path:
pkg = tmp / "site-packages" / "bot_bottle"
(pkg / "cli").mkdir(parents=True)
(pkg / "__init__.py").write_text("")
(pkg / "cli" / "__init__.py").write_text("# module\n")
bundled = pkg / "_resources"
for rel in resources.BUNDLED_RESOURCES:
dst = bundled / rel
dst.parent.mkdir(parents=True, exist_ok=True)
dst.write_text(f"# fake {rel}\n")
return pkg
@contextmanager
def _wheel(self):
"""Point `resources` at a fake installed wheel with an isolated
app-data dir; yields the package dir so a test can mutate it."""
with tempfile.TemporaryDirectory() as tmpname:
tmp = Path(tmpname)
pkg = self._fake_install(tmp)
self.addCleanup(use_bottle_root(tmp / "appdata"))
with patch.object(resources, "_PKG", pkg), \
patch.object(resources, "_BUNDLED", pkg / "_resources"), \
patch.object(resources, "_CHECKOUT_ROOT", tmp / "no-checkout"):
yield pkg
def test_stage_and_resolve(self):
with self._wheel():
self.assertFalse(resources.is_source_checkout())
root = resources.build_root()
# Staged context looks like a repo root.
self.assertTrue((root / "bot_bottle" / "__init__.py").is_file())
self.assertTrue((root / "bot_bottle" / "cli" / "__init__.py").is_file())
self.assertTrue((root / "pyproject.toml").is_file())
self.assertTrue((root / "Dockerfile.gateway").is_file())
self.assertTrue((root / "nix" / "firecracker-netpool.nix").is_file())
self.assertTrue((root / "scripts" / "firecracker-netpool.sh").is_file())
# The bundled-resource copies are NOT re-nested under the staged
# package (keeps it byte-identical to a checkout package).
self.assertFalse((root / "bot_bottle" / "_resources").exists())
# Helpers resolve off the staged root.
self.assertEqual(root / "Dockerfile.gateway",
resources.dockerfile("Dockerfile.gateway"))
# Idempotent: second call returns the same completed dir.
self.assertEqual(root, resources.build_root())
def test_refreshes_when_content_changes_at_same_version(self):
# Regression for the stale-cache bug: `pipx install --force` of a newer
# commit keeps version 0.1.0, so keying on version would reuse the old
# tree. Keying on content must re-stage when a package file changes.
with self._wheel() as pkg:
root1 = resources.build_root()
self.assertTrue((root1 / ".complete").is_file())
(pkg / "cli" / "__init__.py").write_text("# new commit, same version\n")
root2 = resources.build_root()
self.assertNotEqual(root1, root2)
self.assertEqual(
"# new commit, same version\n",
(root2 / "bot_bottle" / "cli" / "__init__.py").read_text(),
)
def test_failed_stage_cleans_up_temp_dir(self):
# A failure mid-stage must not leave a half-written temp dir behind.
with self._wheel():
base = resources.bot_bottle_root() / "build-root"
with patch.object(resources.shutil, "copytree", side_effect=OSError("boom")):
with self.assertRaises(OSError):
resources.build_root()
self.assertEqual([], list(base.glob(".staging-*")))
def test_rebuilds_when_stage_incomplete(self):
# A crash mid-stage can leave a dir without its `.complete` marker; the
# next call must rebuild it rather than trust the partial tree.
with self._wheel():
root = resources.build_root()
(root / ".complete").unlink()
(root / "sentinel").write_text("stale")
again = resources.build_root()
self.assertEqual(root, again) # same content digest → same dir
self.assertTrue((again / ".complete").is_file())
self.assertFalse((again / "sentinel").exists()) # rebuilt clean
def test_reuses_peer_stage_after_lock_wait(self):
# Regression for the staging race: a caller that loses the lock must,
# once it wins, see the peer's completed tree and reuse it — never
# re-clobber a shared path. Drive it deterministically: hold the lock,
# let a worker block after its fast-path miss, publish a complete tree
# as the "peer", then release so the worker takes the reuse path.
import fcntl
import threading
import time
with self._wheel():
base = resources.bot_bottle_root() / "build-root"
base.mkdir(parents=True, exist_ok=True)
dest = base / resources._content_digest() # pylint: disable=protected-access
result: dict[str, Path] = {}
with open(base / ".stage.lock", "w", encoding="utf-8") as held:
fcntl.flock(held, fcntl.LOCK_EX)
def worker() -> None:
result["root"] = resources.build_root()
t = threading.Thread(target=worker)
t.start()
# Let the worker miss the fast path (dest not yet complete) and
# block on the held lock, then publish a complete tree as a peer
# would have and release the lock.
time.sleep(0.3)
dest.mkdir(parents=True)
(dest / ".complete").write_text("")
fcntl.flock(held, fcntl.LOCK_UN)
t.join(timeout=10)
self.assertEqual(dest, result["root"])
self.assertFalse(t.is_alive())
def test_missing_bundle_raises(self):
with tempfile.TemporaryDirectory() as tmpname:
tmp = Path(tmpname)
pkg = tmp / "bot_bottle"
pkg.mkdir()
restore = use_bottle_root(tmp / "appdata")
self.addCleanup(restore)
with patch.object(resources, "_PKG", pkg), \
patch.object(resources, "_BUNDLED", pkg / "_resources"), \
patch.object(resources, "_CHECKOUT_ROOT", tmp / "no-checkout"):
with self.assertRaises(resources.ResourceError):
resources.build_root()
if __name__ == "__main__":
unittest.main()
+1 -1
View File
@@ -728,7 +728,7 @@ class TestResolvedRoutesPayload(unittest.TestCase):
class TestNonBlockingSupervise(unittest.TestCase):
"""PRD prd-new / issue #412: pending responses carry the proposal id, and
"""PRD 0072 / issue #412: pending responses carry the proposal id, and
`check-proposal` polls a queued proposal without blocking or re-proposing."""
_ROUTES = "routes:\n - host: example.com\n"
+36 -2
View File
@@ -27,7 +27,41 @@ class TestCheckPullRequest(unittest.TestCase):
event = {"pull_request": {"title": "Change", "body": "Part of #12", "labels": []}}
self.assertEqual(check_pull_request(event, api), [])
def test_rejects_labels_and_pr_reference(self):
def test_accepts_labelled_pr_without_issue(self):
api = Mock()
event = {
"pull_request": {
"title": "Change",
"body": "",
"labels": [{"name": "Kind/Documentation"}],
}
}
self.assertEqual(check_pull_request(event, api), [])
api.request.assert_not_called()
def test_rejects_unlabelled_pr_without_issue(self):
api = Mock()
event = {"pull_request": {"title": "Change", "body": "", "labels": []}}
errors = check_pull_request(event, api)
self.assertEqual(len(errors), 1)
self.assertIn("either have a label or reference an issue", errors[0])
api.request.assert_not_called()
def test_rejects_labelled_pr_linked_to_real_issue(self):
api = Mock()
api.request.return_value = {"number": 12, "pull_request": None}
event = {
"pull_request": {
"title": "Change",
"body": "Closes #12",
"labels": [{"name": "Kind/Documentation"}],
}
}
errors = check_pull_request(event, api)
self.assertEqual(len(errors), 1)
self.assertIn("exactly one tracking mode", errors[0])
def test_still_validates_issue_reference_when_both_modes_are_used(self):
api = Mock()
api.request.return_value = {"number": 12, "pull_request": {}}
event = {
@@ -39,7 +73,7 @@ class TestCheckPullRequest(unittest.TestCase):
}
errors = check_pull_request(event, api)
self.assertEqual(len(errors), 2)
self.assertIn("unlabeled", errors[0])
self.assertIn("exactly one tracking mode", errors[0])
self.assertIn("not an issue", errors[1])
+119
View File
@@ -0,0 +1,119 @@
"""Integration: build the wheel, install it into an isolated venv, and prove
the installed distribution is self-contained.
This is the boundary a source-tree existence test can't reach (issue #197
review): under an installed wheel the package lives in ``site-packages`` with
no repo root above it, so anything resolving Dockerfiles / nix / scripts from
``__file__``'s parents would break. Here we install for real and assert that
``bot-bottle doctor`` runs from the console script and that
``bot_bottle.resources`` stages a valid, repo-root-shaped build context.
It does NOT run `start` building images needs a Docker/KVM host (CI). It
skips cleanly when the build/venv toolchain isn't available.
"""
from __future__ import annotations
import subprocess
import sys
import tempfile
import unittest
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parents[2]
class TestWheelInstall(unittest.TestCase):
"""`build` is a declared dev dependency (requirements-dev.txt), so this runs
in CI. A build/install failure is a real packaging regression and FAILS
only genuinely-unsupported infra (no `venv`/`ensurepip`) skips."""
_tmp: "tempfile.TemporaryDirectory[str]"
venv_py: Path
app_root: Path
@classmethod
def setUpClass(cls):
cls._tmp = tempfile.TemporaryDirectory( # pylint: disable=consider-using-with
prefix="bb-wheel-")
tmp = Path(cls._tmp.name)
dist = tmp / "dist"
# A failed wheel build is exactly the regression this test guards — fail,
# don't skip. `build` is installed via requirements-dev.txt.
built = subprocess.run(
[sys.executable, "-m", "build", "--wheel", "--outdir", str(dist), str(REPO_ROOT)],
capture_output=True, text=True, check=False,
)
if built.returncode != 0:
raise AssertionError(f"wheel build failed:\n{built.stderr[-2000:]}")
wheels = list(dist.glob("*.whl"))
if not wheels:
raise AssertionError(f"no wheel produced:\n{built.stdout[-2000:]}")
# A missing `venv`/`ensurepip` is unsupported optional infra, not a
# packaging bug — skip only here.
venv = tmp / "venv"
made = subprocess.run([sys.executable, "-m", "venv", str(venv)],
capture_output=True, text=True, check=False)
if made.returncode != 0:
raise unittest.SkipTest(f"venv/ensurepip unavailable:\n{made.stderr[-1500:]}")
cls.venv_py = venv / "bin" / "python"
# Installing the freshly-built wheel must succeed — fail if it doesn't.
install = subprocess.run(
[str(cls.venv_py), "-m", "pip", "install", "--quiet", str(wheels[0])],
capture_output=True, text=True, check=False,
)
if install.returncode != 0:
raise AssertionError(f"pip install of the wheel failed:\n{install.stderr[-2000:]}")
# Isolate the staged build root the wheel writes under the app-data dir.
cls.app_root = tmp / "appdata"
@classmethod
def tearDownClass(cls):
cls._tmp.cleanup()
def _run(self, tail: "list[str]") -> "subprocess.CompletedProcess[str]":
"""Run the installed venv's python with `tail` appended, from a neutral
cwd so the source checkout isn't on sys.path — we must import the
*installed* package, not the repo we built from."""
env = {"BOT_BOTTLE_ROOT": str(self.app_root), "PATH": "/usr/bin:/bin"}
return subprocess.run(
[str(self.venv_py), *tail],
capture_output=True, text=True, env=env, cwd=self._tmp.name, check=False,
)
def test_console_entry_point_installed(self):
# The `bot-bottle` script the wheel declares must exist in the venv.
script = self.venv_py.parent / "bot-bottle"
self.assertTrue(script.is_file(), "bot-bottle console script not installed")
def test_doctor_runs_from_installed_package(self):
proc = self._run(["-m", "bot_bottle.cli", "doctor"])
# doctor exits non-zero here (no backend), but it must RUN and report.
self.assertIn("python", proc.stdout)
self.assertIn(proc.returncode, (0, 1))
def test_installed_wheel_is_self_contained(self):
# From the installed layout (not a checkout), resources must resolve
# Dockerfiles and stage a repo-root-shaped build context.
script = (
"import bot_bottle.resources as r\n"
"assert not r.is_source_checkout(), 'should not look like a checkout'\n"
"assert r.dockerfile('Dockerfile.gateway').is_file()\n"
"assert r.nix_netpool_module().is_file()\n"
"assert r.netpool_script().is_file()\n"
"root = r.build_root()\n"
"assert (root / 'bot_bottle' / '__init__.py').is_file(), 'no package in context'\n"
"assert (root / 'pyproject.toml').is_file(), 'no pyproject in context'\n"
"assert (root / 'Dockerfile.gateway').is_file(), 'no Dockerfile in context'\n"
"print('SELF_CONTAINED_OK')\n"
)
proc = self._run(["-c", script])
self.assertIn("SELF_CONTAINED_OK", proc.stdout, msg=proc.stderr)
if __name__ == "__main__":
unittest.main()