Compare commits

...

56 Commits

Author SHA1 Message Date
didericis 9bf2961d13 fix: name an absolute path in the sudo re-run hint
test / unit (push) Successful in 57s
test / image-input-builds (push) Successful in 1m2s
Update Quality Badges / update-badges (push) Successful in 1m8s
test / integration-docker (push) Successful in 58s
test / coverage (push) Successful in 15s
lint / lint (push) Failing after 10m9s
The firecracker setup message told users to run

    sudo bot-bottle backend setup --backend=firecracker

which fails for exactly the users who followed the documented install. sudo
replaces PATH with sudoers' secure_path — /usr/local/sbin:/usr/local/bin:
/usr/sbin:/usr/bin:/sbin:/bin on Debian and Ubuntu — which deliberately
excludes user-writable directories. Both supported install paths land in one:
pipx uses ~/.local/bin, and install.sh's venv fallback symlinks there. So the
hint works for anyone who installed system-wide and breaks with "command not
found" for everyone else, which is how it survives a read-through.

Add bot_bottle/invocation.py: self_path() resolves the running entry point to
an absolute path, and sudo_command() builds the copy-pasteable form. The one
sudo recommendation in the tree now uses it. Non-sudo hints keep the bare
`bot-bottle`, which is correct — the user reached them by running it.

self_path() falls back to the bare name when argv[0] cannot be resolved (a
`python -m` style invocation), because a slightly wrong hint beats a traceback
raised while reporting some unrelated problem.

Tested behaviourally rather than by scanning source: the first version of the
test grepped the module and failed on the comment explaining why the bare form
is wrong. It now drives _setup_systemd() as non-root and asserts what is
actually printed. Verified the guard bites by restoring the bare form and
watching it fail.

Not verified end to end: this message only prints on the systemd path, so it
is unreachable on macOS, where the rest of this work was tested.
2026-07-27 12:35:51 -04:00
didericis 6385752040 feat: get Alpine and NixOS cells to green
Completes the matrix: all five distros now pass both `test` and `test-ready`,
validated end-to-end on the delphi KVM host (QEMU 11.0.2).

Alpine — the generic_ cloud image silently ignores a NoCloud seed, so cloud-init
never ran. Fixes, all confirmed by booting:
- switch to the nocloud_ image variant (built for a local seed),
- start sshd from a runcmd (OpenRC doesn't auto-start it after key injection),
- give the account a throwaway password — Alpine's non-PAM sshd refuses pubkey
  auth for a cloud-init-locked ('!') account, unlike the UsePAM=yes distros,
- install sudo via cloud-init packages: (the minimal image has none, so the
  test-ready `sudo apk add` prereq failed).
Also add instance-id to the NoCloud meta-data, which the nocloud image requires.

NixOS — publishes no downloadable cloud qcow2 (Hydra builds AMIs), so the harness
now BUILDS one with nixos-generators (new scripts/linux-install-test-nixos.nix:
cloud-init for the key, flakes enabled, deliberately no python/git/pipx).
ensure_base_image branches to build_nixos_image for the nixos distro. The
test-ready prereq uses `nix profile install` pinned to nixpkgs/nixos-24.11,
because the guest's default unstable registry builds pipx from source and its
test suite currently fails to build.

Validation: 10/10 cells pass, clean teardown, no leaked VMs or run dirs.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 12:24:31 -04:00
didericis 496608fc25 fix: correct harness defects found running the matrix
First full run on the KVM host surfaced three harness bugs and two stale
image URLs; all fixed here.

Harness bugs:
- Liveness probe used `bot-bottle --version`, which the CLI does not implement
  (unknown args die non-zero) — so every SUCCESSFUL install was misreported as
  "no runnable entry point". Switched to `bot-bottle --help`, which exits 0
  before any DB/migration/network work.
- cmd_down never removed serial.log, so its rmdir failed and every run left an
  orphan scratch dir behind. Added serial.log to the cleanup.
- The teardown trap was armed AFTER cmd_up, but cmd_up's wait_for_ssh can fail
  with QEMU already running (a guest that never opens SSH) — leaking the VM.
  Arm the trap before cmd_up.

Stale image URLs:
- Fedora 41 is EOL and 404s; bumped to Fedora 44 (44-1.7).
- Alpine bumped to 3.21.7; its cloud images ship a .sha512 only (this verifier
  is sha256), so SUM_URL is now empty (skip) with a note.

Validated on delphi (QEMU 11.0.2): ubuntu/fedora/arch pass BOTH test and
test-ready. Alpine (cloud-init seed not applied → no SSH) and NixOS (no
upstream cloud qcow2) remain blocked on image provisioning, documented in the
research note as follow-ups.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 12:24:31 -04:00
didericis 218f29cb05 refactor: adopt macOS test/test-ready convention for variants
Replaces the BB_TEST_PREREQS env toggle with the two subcommands the macOS
harness established, so the two repos read the same:

  test        A bare host — prerequisites NOT set up (the default state of a
              stock cloud image). Exercises install.sh's own prerequisite-guard
              logic. SOUND (PASS) when install.sh either installs cleanly or
              declines with one of its own recognized, actionable errors; a
              crash or unrecognized failure fails.
  test-ready  Prerequisites satisfied — the harness installs python3/git/pipx
              first, then runs install.sh, which must actually land: entry point
              runnable, doctor reporting a usable python and config.

Structure now mirrors scripts/macos-install-test.sh: a shared _test_cycle
driving up -> (prereqs) -> run -> verdict -> down, thin cmd_test / cmd_test_ready
wrappers setting _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL, a standalone `prereqs`
subcommand, and a _test_teardown that prints the PASS/FAIL claim.

The doctor check is now the macOS-style classifier rather than a bare exit-code
read: a Traceback is an install defect (fail), a missing `ok: python:` /
`ok: config:` line is a fail, and a not-ready backend is reported, not fatal
(BB_TEST_REQUIRE_BACKEND=1 makes it fatal for a nested-virt host). This is where
Linux diverges from macOS test-ready — a plain VM has no nested KVM for
Firecracker and the harness doesn't provision the Docker backend, so backend
readiness is never the Linux criterion.

test-all now runs every distro × {test, test-ready}. Research note updated.

Validated with `bash -n` and `shellcheck`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 12:24:31 -04:00
didericis 1518f73de5 feat: add bare-host install variant to the Linux harness
Splits the Linux clean-install harness into two variants, selected with
BB_TEST_PREREQS:

  with    (default) — install python3 + git + pipx first, then install.sh.
                      The ready-host happy path; PASS = install.sh exits 0 and
                      leaves a runnable bot-bottle entry point.
  without           — run install.sh on the BARE cloud image. Exercises
                      install.sh's own prerequisite-guard logic (python gate,
                      git-for-git-specs gate, pipx/pip PEP-668 handling). PASS =
                      install.sh either fully succeeds OR declines with one of
                      its own recognized, actionable prerequisite errors; a
                      crash or unrecognized failure is a FAIL.

cmd_run now captures install.sh's exit code + full output (install.rc /
install.log) instead of aborting on non-zero, so the verdict step applies the
variant's criterion. The old assert_installed is replaced by a quiet
entry_point_runnable helper plus classify_outcome, which matches the bare-host
declines against the exact die() messages install.sh prints. doctor still runs
for visibility, but only when an entry point exists.

test-all now runs the full matrix -- every distro x both variants -- each cell
in its own throwaway VM, and can be pinned to one variant via BB_TEST_PREREQS.
Research note updated to document both variants and their pass criteria.

Validated with `bash -n` and `shellcheck`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 12:24:31 -04:00
didericis 9d82535390 feat: add Linux clean-install test harness
Adds scripts/linux-install-test.sh, a throwaway-VM harness that exercises
install.sh the way a brand-new user would across a Linux distro matrix
(Ubuntu, Fedora, Arch, Alpine, NixOS), plus the research note
docs/research/testing-clean-install-on-linux.md that motivates the approach.

This is the Linux counterpart to the macOS clean-install harness. On Linux
the boundary of choice flips from a throwaway user account to a disposable
KVM VM: it's a genuine kernel + userland + package-manager boundary that
wipes to nothing on teardown, and a single harness can swap distro cloud
images to cover the several package-management regimes Linux fragments into.
The host already needs KVM for the Firecracker backend, so a per-run,
copy-on-write VM (qemu-img create -b base) is cheap here -- the equivalent
of `docker run --rm`, for a whole machine.

Per run the harness caches one read-only base image, boots a throwaway
overlay via QEMU/KVM with user-mode networking (no root, no bridge), injects
an ephemeral SSH key + passwordless login through a cloud-init seed ISO,
installs the distro's prerequisites (python3 + git + pipx), pipes THIS
checkout's install.sh into the guest exactly as `curl ... | sh` would, and
asserts the CLI installed cleanly. The overlay is deleted on teardown, so
even the OS-level prerequisites are wiped -- unlike a throwaway user, the
reset is total.

Subcommands mirror the macOS harness (up/run/status/down/test) plus test-all
(the matrix) and ssh (an interactive guest shell). test arms an EXIT/INT/TERM
trap the moment the VM exists, so a failure or Ctrl-C still tears it down.

Scope is installer correctness, not runtime: there is no nested KVM/Docker in
the VM, so `bot-bottle doctor` correctly reports every backend not-ready and
exits non-zero by design. The pass criterion is therefore install.sh exiting
0, the bot-bottle entry point being present on the fresh user's PATH, and
`bot-bottle --version` running -- not a green doctor. doctor's output is still
printed so a real installer regression (broken shim, import error) stays
visible.

Validated with `bash -n` and `shellcheck`. Runtime is host-only (needs
/dev/kvm, qemu, cloud-localds), so like the macOS harness it isn't exercised
by the Linux PR CI. The cloud-image URLs in the DISTRO table are the one
place to bump when a distro cuts a newer build.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 12:24:31 -04:00
didericis c89847b626 fix(macos-container): tag the pinned base from its ref, not its ID
lint / lint (push) Successful in 59s
Update Quality Badges / update-badges (push) Successful in 1m6s
test / coverage (push) Successful in 22s
test / image-input-builds (push) Successful in 58s
test / integration-docker (push) Successful in 58s
test / unit (push) Successful in 2m51s
`container image tag` only accepts image-name[:tag] as its source and
rejects a bare 64-hex image ID ("cannot specify 64 byte hex string as
reference"), so pinning the agent image for the nested-containers layer
died before the build could start. Tag from the ref instead; the
existing post-tag inspection is what keeps the handoff fail-closed if
the ref moves between the two commands.

The unit test mocked subprocess and asserted the ID-as-source call
shape, so it never saw the CLI's rejection. It now pins the ref as the
source, and covers the bare-hex ID that `container image inspect`
actually reports plus the mid-flight tag move.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
2026-07-27 10:56:59 -04:00
didericis cd9f023f3d docs: name bot-bottle, not ./cli.py, in every instruction
prd-number-check / require-numbered-prds (pull_request) Successful in 11s
tracker-policy-pr / check-pr (pull_request) Successful in 14s
test / unit (pull_request) Successful in 59s
test / integration-docker (pull_request) Successful in 1m6s
test / coverage (push) Successful in 21s
test / image-input-builds (push) Successful in 44s
test / image-input-builds (pull_request) Successful in 1m15s
test / unit (push) Successful in 57s
test / coverage (pull_request) Successful in 20s
lint / lint (push) Successful in 1m3s
Update Quality Badges / update-badges (push) Successful in 1m12s
test / integration-docker (push) Successful in 1m0s
`doctor` told users to run `./cli.py backend setup`. cli.py is a four-line
wrapper at the repo root that calls bot_bottle.cli:main, and it does not ship —
[tool.setuptools.packages.find] includes only bot_bottle*, so anyone who
installed rather than cloned has no such file. Confirmed against a real
install: the user's tree contains exactly one executable, `bot-bottle`, and no
cli.py anywhere, while doctor recommended `./cli.py` three times. Invisible in
development, where ./cli.py works fine from a checkout, which is why only a
clean-install test surfaced it.

The two entry points are the same code, so the fix is to name the one that
always exists. 141 replacements across 45 files: the runtime messages that
caused this, plus README, docs, PRDs, research notes and test prose, so nothing
teaches the invocation a user cannot run.

scripts/demo.sh is deliberately untouched — it *executes* ./cli.py from a
checkout, where that is the correct and available path.

Verified end to end: a sandbox install now reports
"Run: bot-bottle backend setup --backend=firecracker", and bot-bottle is on
that user's PATH. Unit suite unchanged against the pre-existing baseline.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 10:20:17 -04:00
didericis 5c499d290b docs: record what the passing runs established
test / image-input-builds (pull_request) Successful in 47s
test / unit (pull_request) Successful in 59s
test / integration-docker (pull_request) Failing after 2m50s
test / coverage (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 12m32s
prd-number-check / require-numbered-prds (pull_request) Failing after 12m45s
Both variants now pass, and the with-prerequisites run settled several things
that were guesses before:

* The Apple `container` service is per-user, confirmed directly rather than
  inferred: the throwaway account's appRoot is its own
  (/Users/bbtest/Library/Application Support/com.apple.container/), and
  starting it left the admin's service running and its agents intact.
* Setting up a new account is two steps, not one. The guest kernel lives in
  that same per-user app root, so a fresh account has none and
  `container system start` prompts to download it — and only prompts, since
  the flags default to asking. Headless callers need
  --enable-kernel-install.
* Neither step is performed by `bot-bottle backend setup
  --backend=macos-container`, which checks and then defers to `container
  system start`. So the real path for a new account is
  `container system start --enable-kernel-install`.
* Any of it requires entering the user's launchd domain with `launchctl
  asuser`, because the apiserver is a per-user agent reached over XPC.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 10:12:21 -04:00
didericis 0500ae5f4c fix(harness): make the prereq step non-interactive
prd-number-check / require-numbered-prds (pull_request) Successful in 16s
tracker-policy-pr / check-pr (pull_request) Successful in 11s
test / image-input-builds (pull_request) Successful in 42s
test / unit (pull_request) Successful in 53s
test / integration-docker (pull_request) Successful in 1m1s
test / coverage (pull_request) Failing after 15s
The launchd fix worked. `container system start` now gets past "Testing access
to container-apiserver" and "Verifying machine API server is running" — the two
steps it could not reach before — so the XPC domain problem is solved. It then
failed somewhere new:

    No default kernel configured.
    Error: failed to read user input
    Install the recommended default kernel from [...kata-static...]? [Y/n]:

`container system start` PROMPTS for the default Linux kernel when given
neither --enable-kernel-install nor --disable-kernel-install ("default: prompt
user"). A throwaway account always hits that prompt, because the kernel lives
in the per-user app root and a fresh home has none — and with no TTY the
prompt dies immediately.

Pass --enable-kernel-install. That is also the honest choice for what this
variant claims: the backend cannot run a bottle without a guest kernel, so
"prerequisites satisfied" has to include it. The cost is a kernel download per
run, since the previous run's copy went with the deleted home;
BB_TEST_KERNEL_INSTALL=0 switches to --disable-kernel-install when you only
care that the service comes up.

This also means the per-user prerequisite is not one step but two — start the
service, install a kernel — which is worth knowing for anyone setting up a
second account by hand.

The rig's stub now models the prompt: an unflagged start fails the way the real
CLI does. Verified that guard bites by removing the flag and watching the
scenario fail, so a future edit cannot silently reintroduce an interactive
prereq step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 10:00:49 -04:00
didericis 5614a56ccd fix(harness): run as the user inside their launchd domain
prd-number-check / require-numbered-prds (pull_request) Successful in 11s
test / image-input-builds (pull_request) Successful in 44s
test / unit (pull_request) Successful in 57s
test / integration-docker (pull_request) Failing after 3m4s
test / coverage (pull_request) Has been skipped
tracker-policy-pr / check-pr (pull_request) Failing after 13m12s
`test` passes. `test-ready` got as far as the service start and failed:

    Launching container-apiserver...
    Error: failed to get a response from apiserver: invalidState: "unauthorized request"

which is the limitation this script's own header predicted — "may need a full
launchd user session (`launchctl asuser`)".

`container system start` registers com.apple.container.apiserver as a per-USER
launchd agent and then talks to it over XPC. `launchctl list` confirms the
shape: the apiserver and every container-network/runtime job are agents in the
invoking user's domain. Under plain `sudo -u` the caller stays in root's
bootstrap namespace, so the lookup crosses domains and the apiserver rejects it
as unauthorized — the agent launched fine, the client just could not reach it.

run_as_user now enters the target user's domain with `launchctl asuser <uid>`,
which is also what a real user gets from Terminal, so it is the more faithful
way to run everything here rather than a special case for the service start.

A never-GUI-logged-in account may have no bootstrappable domain at all, so
availability is probed once and cached, and everything falls back to plain
`sudo -u` when it is missing. Without that fallback a missing domain would
break `test` — which passes today and does not need a session — for a reason
unrelated to what it tests. The rig covers that path explicitly.

If the start still fails, the error now names the cause and the way out
(log into the account once to get it a domain) instead of just relaying the
CLI's message.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:56:51 -04:00
didericis 71c0f8ed88 feat(harness): add test-ready, the with-prerequisites variant
prd-number-check / require-numbered-prds (pull_request) Successful in 11s
tracker-policy-pr / check-pr (pull_request) Successful in 21s
test / image-input-builds (pull_request) Successful in 48s
test / integration-docker (pull_request) Successful in 1m4s
test / unit (pull_request) Successful in 2m52s
test / coverage (pull_request) Failing after 20s
"Does the installer work" and "can a new user actually run a bottle" are
different questions, and collapsing them is what made the previous run
ambiguous. Split them into two cycles over the same throwaway account:

  test        a macOS system WITHOUT the prerequisites set up for this user,
              which is the default state of every new account since the Apple
              `container` service is per-user. Asserts the install is sound;
              reports backend readiness without failing on it, because
              install.sh provides no backend and cannot regress one.

  test-ready  the same system WITH them. Runs `container system start` for the
              throwaway user, then demands doctor go fully green, backend
              included — the end-to-end claim.

The new `prereqs` step runs `container system start` rather than
`bot-bottle backend setup --backend=macos-container`, because that subcommand
does not start anything: it checks, then tells you to run `container system
start` yourself (backend/macos_container/setup.py, "no privileged host setup
required"). The harness runs what the product actually asks for.

test-ready sets BB_TEST_REQUIRE_BACKEND, so it demands exactly what `test`
merely reports. Both share one cycle function; the step count and the PASS
claim are the only differences beyond the extra step, so neither variant can
drift from the other's teardown or dirty-account guarantees.

Rig covers the distinguishing case directly: identical host state where `test`
passes on install soundness and `test-ready` starts the service and reaches a
green backend, plus the service-start failure and an install failure under
test-ready. 22 scenarios, all passing.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:51:35 -04:00
didericis 556217ae7b fix(harness): judge the install separately from backend readiness
prd-number-check / require-numbered-prds (pull_request) Successful in 9s
test / image-input-builds (pull_request) Successful in 46s
test / integration-docker (pull_request) Successful in 1m3s
test / unit (pull_request) Successful in 2m58s
test / coverage (pull_request) Failing after 18s
tracker-policy-pr / check-pr (pull_request) Failing after 14m1s
The first run against the branch's own code confirmed the doctor fix — no
traceback, the firecracker probes now report "TAP pool: 0/8" instead of dying
on PermissionError — and then failed for a reason the installer cannot fix.

The Apple `container` service is per-USER. `container system status` reports an
appRoot under ~/Library/Application Support/com.apple.container/, and bbtest
sees "container system service: NOT running" while the creating admin's is
running. A brand-new account therefore has no backend until it runs
`container system start` once, doctor rightly fails, and `test` would be
permanently red for host state that install.sh neither creates nor can regress.

So stop treating doctor's exit code as one verdict. `test` now fails when the
install is broken — no entry point, doctor crashed, or doctor could not report
a usable python and config dir — and reports backend readiness as a note.
BB_TEST_REQUIRE_BACKEND=1 restores the strict behaviour.

A traceback is checked for explicitly rather than inferred from the exit code,
because those are the same value. The PermissionError bug exited non-zero
exactly like a missing prerequisite does; only the traceback distinguishes a
defect from an environment fact, and that distinction is the whole point of
this change.

The research doc's footprint table had the service as a host-wide launchd
service that survives user deletion. It does not. The state lives in the home
and goes with it, so the reset is more complete than claimed — but the
corollary is that a fresh account cannot run a bottle until the service is
started for it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:45:57 -04:00
didericis 7461802b62 fix(harness): install the branch under test, and stop mangling snippets
prd-number-check / require-numbered-prds (pull_request) Successful in 9s
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / unit (pull_request) Successful in 1m3s
test / integration-docker (pull_request) Successful in 1m2s
test / image-input-builds (pull_request) Successful in 1m6s
test / coverage (pull_request) Failing after 20s
lint / lint (push) Successful in 3m30s
Two harness bugs, both of which made the last run lie about what it verified.

The run installed main. install.sh's default spec is the repo's default
branch, so `run` piped *this checkout's* installer into the throwaway user and
then had it install a package that does not contain the branch's changes —
which is why the doctor PermissionError fix appeared not to work: it was never
in the code under test. The clone line said so plainly (commit 420184b, not
this branch's HEAD) and I read past it. The spec now pins to the current
branch, over https because the throwaway user has neither our SSH key nor read
access to this mode-700 checkout. Since what it clones is whatever the remote
has, a dirty tree or an unpushed branch now prints a warning rather than
quietly testing something other than what is on disk.

The doctor probe also died with "sh: -c: line 1: syntax error: unexpected end
of file". `sudo -i` joins its argv into one string for the login shell's -c, so
an argument's quoting is not preserved and a newline terminates the command.
The multi-line snippet added in the previous commit could not survive that.
run_as_user now feeds snippets on stdin to `sh -s`, which is the same trick
install.sh already arrives by, and is immune to the joining. The install spec
moves into the stream as an export for the same reason — as an argument it had
the identical latent quoting bug, unnoticed only because the default spec has
no shell metacharacters in it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:38:39 -04:00
didericis 5c12c6bf4e fix(doctor): survive a PATH entry the user can't execute
The install itself now works end to end on a fresh macOS account — venv built,
package installed from git, entry point linked — and `doctor` then died with an
unhandled traceback:

    PermissionError: [Errno 13] Permission denied: 'ip'

netpool's probe helpers caught only FileNotFoundError. That is not the only way
a probe binary can be unavailable: when a name on PATH exists but this user
cannot execute it, exec fails with EACCES, and CPython reports that in
preference to the ENOENT from the other PATH entries. So `except
FileNotFoundError` misses it and the crash propagates all the way out of
`doctor`. Reproduced directly: a mode-000 file named `ip` on PATH yields
exactly the error above.

_run_ok's docstring already stated the intent — treat an unavailable binary as
failure rather than crashing — so this widens the catch to OSError to match
what it says. Any OSError means the probe could not run, which for a
fail-closed check is indistinguishable from "not present". The same narrow
catch is fixed in overlapping_routes and in the two docker probes
(compose ls, docker ps), which are the same shape and equally reachable.

Deliberately not touched: the FileNotFoundError catches around file I/O in
bottle_state and orchestrator/service, where the narrow exception is correct.

The harness also required `bot-bottle` on PATH before running doctor, which
could never be true: install.sh prints the PATH line rather than editing a
shell profile, by design, so on a fresh account the entry point is installed
and working but not on PATH. It now looks where the installer actually puts
it (~/.local/bin, then the venv) and notes when it's running by absolute path.
That was the harness failing a run for a reason the installer intends.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:38:39 -04:00
didericis 4096409495 fix(install): name the profile the user's shell actually reads
The PATH hint hardcoded ~/.zprofile, which is right for macOS's zsh and wrong
for the Linux users this installer also serves. Pick from $SHELL instead.
~/.profile is the default for non-zsh rather than ~/.bash_profile, because
creating the latter would shadow an existing ~/.profile.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:38:20 -04:00
didericis 8d0782d1c7 fix(install): install into a private venv when pipx is absent
The harness's second run hit the wall the first one predicted: a fresh account
has no pipx, so install.sh fell to `pip install --user`, and every Python a Mac
offers — Homebrew and python.org alike — is externally managed, so PEP 668
blocked it. That fallback was never a fallback on macOS; it was a dead end that
printed instructions.

Replace it with a venv at ~/.bot-bottle/venv (BOT_BOTTLE_VENV to move it),
with the console script symlinked into ~/.local/bin. PEP 668 does not apply
inside a venv, and venv is stdlib, so unlike pipx there is nothing to bootstrap
first. pipx stays the preferred path when present, so anyone already managing
their Python apps that way is unaffected — and the post-install PATH check now
asks pipx for PIPX_BIN_DIR instead of assuming ~/.local/bin.

Keeping the venv under ~/.bot-bottle rather than ~/.local/share means the whole
footprint stays in one directory, which is what lets the throwaway-account
teardown remain a complete reset.

This removes the PEP 668 pre-flight and the sysconfig user-scheme lookup, both
of which existed only to serve the --user path. Their tests go with them:

* `detects_externally_managed_python` asserted the check that is now moot;
  replaced by one asserting pipx is still preferred when present.
* `checks_pip_usable_before_fallback` pinned a pip probe that no longer runs;
  replaced by one asserting the venv's own pip does the install, since using
  the base interpreter's would install outside the venv.
* `resolves_user_scripts_dir_not_hardcoded` and
  `macos_user_scheme_is_not_dot_local_bin` guarded the ~/Library/Python
  scripts-dir lookup. Nothing installs there now. The surviving "don't
  hardcode" concern is pipx's bin dir, which has its own test.

Five tests are added for the new path: the venv fallback exists, no --user path
survives, the venv is under the config dir, venv creation failure names
python3-venv (Debian ships it separately), and the entry point is exposed
outside the venv.

Verified end to end in a sandbox HOME with a fresh-account PATH and no pipx:
venv built, package installed, symlink created, `doctor` reached and green
(python 3.14.5, macos-container ready), exit 0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:38:20 -04:00
didericis 6793a09e2b docs: correct two claims the live harness run disproved
The PATH story was wrong in an instructive way. I said a Homebrew Python is on
PATH only because of a shell-profile line; on this host /etc/paths.d/homebrew
puts /opt/homebrew/bin on every login shell's PATH, fresh accounts included.
The stub still wins, because path_helper appends /etc/paths.d/* *after*
/etc/paths and /usr/bin is in the latter. Ordering, not absence, is what makes
bare `python3` the 3.9.6 stub — which is also why the versioned `python3.14`
candidate is the one that matched during the real run, rather than the
/opt/homebrew/bin/python3 fallback I expected.

The harness header also credited install.sh with writing a PATH line into the
login shell. It does not write one. The reset argument is unaffected (such a
line would live in the deleted home either way), but the claim was untrue.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:38:20 -04:00
didericis af293d7036 fix(install): find a usable python instead of dead-ending on PATH's
The macOS install harness caught this on its first real run: a throwaway
account gets `bot-bottle install: error: python3 3.11 or newer is required`
and stops. A fresh account's PATH is just /etc/paths, which excludes
/opt/homebrew/bin, so `python3` resolves to the Command Line Tools stub —
still 3.9.6 on macOS 26. Homebrew's shellenv line lives in the *installing*
user's ~/.zprofile and is inherited by nobody. The documented `curl … | sh`
path therefore dead-ends for anyone whose profile isn't already set up, which
is every new user, launchd job, and CI runner.

So look past PATH before giving up: try `python3`, then python3.11-3.14, then
/opt/homebrew/bin, /usr/local/bin, ~/.local/bin, and python.org framework
builds, and say which one was picked when it isn't the obvious one. On this
host that turns the failure into a successful install.

The chosen interpreter is now threaded through everything downstream — the pip
probe, the PEP 668 check, the pip --user fallback, and the user-scripts-dir
lookup — which were all still hardcoding `python3` and would otherwise have
run against the 3.9 stub we just rejected. `pipx install` gains `--python`,
since pipx otherwise builds the venv with whichever interpreter pipx itself
was installed with, not the one that passed the version check.

When nothing usable is found the error is now actionable: what was found and
why it's insufficient, where else it looked, a platform-appropriate install
command, and BOT_BOTTLE_PYTHON to point at an interpreter directly. An
explicit BOT_BOTTLE_PYTHON that is too old or unusable is an error rather than
a silent fallback to a different interpreter than the caller asked for.

Three existing tests asserted on incidental literals rather than the behaviour
they describe, and are narrowed to their actual intent:

* `never_uses_sudo` matched the word anywhere, including the new "sudo apt
  install python3.12" remediation *advice*. It now strips string literals and
  comments first, so it still catches sudo as a bare command, in a pipeline,
  and in a command substitution — verified by mutation — while allowing the
  script to print the word.
* `checks_pip_usable_before_fallback` pinned the literal `python3 -m pip`.
* `resolves_user_scripts_dir_not_hardcoded` banned ".local/bin" script-wide;
  it's now scoped to the USER_SCRIPTS assignment it exists to guard, since
  ~/.local/bin is a legitimate place to *find an interpreter*.

README grows the install command and a Requirements section it never had,
leading with the Python floor and why a working `python3` in your own shell
says nothing about what a fresh account sees.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:38:20 -04:00
didericis d415581adc feat: add a one-shot test cycle to the macOS install harness
`test` chains up -> run -> status -> down, which is the loop you actually want
when verifying a clean install. Three things make it more than a convenience
wrapper:

* It refuses to start against an existing account. A reused home is not a clean
  install, so testing one silently would defeat the harness.
* It tears the account down from an EXIT/INT trap armed the moment the account
  exists, so a failed run — or a Ctrl-C mid-install — still leaves the machine
  clean. BB_TEST_KEEP=1 opts out to poke at a failure.
* Its verdict is stricter than the installer's. install.sh exits 0 when it
  finishes but `doctor` reports unmet prerequisites, so "the installer
  succeeded" is not a useful assertion; `test` fails if the install fails, if
  bot-bottle never reached the new user's PATH, or if doctor is unhappy. That
  meant giving cmd_status a real exit status instead of swallowing doctor's.

Also fixes two bugs in `run`'s installer staging, by removing the staging
entirely and feeding install.sh in on stdin:

* `mktemp /tmp/bb-install.XXXXXX.sh` did not do what it looks like. BSD mktemp
  only substitutes trailing Xs, so every run wrote the *same* predictable path,
  as root, mode 644, in a world-writable directory.
* The cleanup only ran on the normal and failure returns, so an interrupted run
  leaked the file.

Piping on stdin sidesteps both: root opens the redirect before sudo drops
privileges, so the mode-700 home that motivated the staging is a non-issue,
there is no file to leak, and `sh -s` matches the documented `curl … | sh`
shape more closely than executing a staged copy did.

The command-level `exit 1`s become `return 1` so the steps compose under the
trap, and the "next, run this" hints are suppressed inside `test`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WEfZZhakx13bxTfXcZCoS5
2026-07-27 09:38:20 -04:00
didericis-claude 27b9ba5247 feat: add macOS clean-install test harness
Add scripts/macos-install-test.sh, a throwaway-user harness for exercising
install.sh the way a brand-new user would on macOS, plus the research note
that motivates the approach.

The harness has up/run/status/down/deep-reset subcommands. Because install.sh
writes only to the user home (pipx venv, ~/.bot-bottle, a PATH line) and never
installs the backend, deleting the account is a complete, deterministic reset
of the install surface. A disposable macOS VM can't stand in on M1/M2: the
Apple `container` backend needs Virtualization.framework, and running it inside
a guest VM requires nested virtualization (M3+ only), so a throwaway user is
the only way to reach the real host backend from a clean $HOME.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 09:38:20 -04:00
didericis-codex 420184b874 fix: close consolidated quality review findings
test / image-input-builds (push) Successful in 44s
lint / lint (push) Successful in 1m3s
test / unit (push) Successful in 1m4s
test / integration-docker (push) Successful in 1m6s
test / coverage (push) Successful in 20s
Update Quality Badges / update-badges (push) Successful in 2m57s
2026-07-27 09:26:25 -04:00
didericis-codex 7938b90d19 fix: enforce shared storage permissions 2026-07-27 09:26:25 -04:00
didericis-codex d879258f62 fix: bound heavy gateway operations 2026-07-27 09:26:25 -04:00
didericis-codex 78d3b43061 fix: enforce cleanup and secret integrity 2026-07-27 09:26:25 -04:00
didericis-codex cdc5c8203d fix(cleanup): use authoritative resource identities 2026-07-27 09:26:25 -04:00
didericis-codex d04bbf1454 fix(gateway): contain output pump shutdown races 2026-07-27 09:26:25 -04:00
didericis-codex c00097d998 fix(gateway): bound stdlib HTTP request work 2026-07-27 09:26:25 -04:00
didericis-codex 6630a1f701 fix(cleanup): revalidate destructive backend plans 2026-07-27 09:26:25 -04:00
didericis-codex a7ad82f03a refactor(supervisor): separate MCP dispatch from transport 2026-07-27 09:26:25 -04:00
didericis-codex 9b8f126b8f refactor(egress): extract outbound DLP request stage 2026-07-27 09:26:25 -04:00
didericis-codex d167c03215 refactor(egress): extract request policy stages 2026-07-27 09:26:25 -04:00
didericis-codex 479b93bd0b fix(orchestrator): bound streamed request bodies 2026-07-27 09:26:25 -04:00
didericis-codex 2de61eeb17 fix(orchestrator): keep host client dependency-free 2026-07-27 09:26:25 -04:00
didericis-codex a4bc5d4508 refactor(orchestrator): replace manual HTTP dispatch with FastAPI 2026-07-27 09:26:25 -04:00
didericis-codex 7ebd067d2c build(orchestrator): pin FastAPI runtime dependencies 2026-07-27 09:26:25 -04:00
didericis-codex be50354100 docs(prd): require shared storage permissions 2026-07-27 09:26:25 -04:00
didericis-codex e1163184c9 docs(prd): bound heavy gateway operations 2026-07-27 09:26:25 -04:00
didericis-codex 70f6a9be20 docs(prd): require cleanup execution integrity 2026-07-27 09:26:25 -04:00
didericis-codex aa43fd032f docs(prd): extend authoritative cleanup boundaries 2026-07-27 09:26:25 -04:00
didericis-codex be5c803e8e docs(prd): define authoritative failure boundaries 2026-07-27 09:26:25 -04:00
didericis-codex 38ff4a5e88 fix(orchestrator): satisfy adapter type contracts 2026-07-27 09:26:25 -04:00
didericis-codex 8df76f6126 refactor(egress): split request policy pipeline stages 2026-07-27 09:26:25 -04:00
didericis-codex b63c41db9c feat(firecracker): enumerate running bottles 2026-07-27 09:26:25 -04:00
didericis-codex 6c52686069 fix(orchestrator): bound unauthenticated HTTP requests 2026-07-27 09:26:25 -04:00
didericis-codex a52245af95 fix(security): authenticate persisted egress secrets 2026-07-27 09:26:25 -04:00
didericis-codex d4e48a4169 fix(docker): fail closed on network address scan errors 2026-07-27 09:26:25 -04:00
didericis-codex 53cc3ca492 fix(firecracker): abort cleanup on scan errors 2026-07-27 09:26:25 -04:00
didericis-codex bafb73fdb9 refactor(backend): type enumeration failures 2026-07-27 09:26:25 -04:00
didericis-codex b0f317c499 fix(docker): surface enumeration failures 2026-07-27 09:26:25 -04:00
didericis-codex 57eb4394dc fix(security): prohibit unauthenticated orchestrator 2026-07-27 09:26:25 -04:00
didericis-codex 82e1a353bd docs: analyze Sandbox Agent SDK architecture 2026-07-27 03:53:02 +00:00
didericis-codex a6e1aebda1 docs: update agent sandbox competitor landscape 2026-07-27 03:44:41 +00:00
didericis-claude 74ec9843f0 chore(coverage): relax thresholds to cut low-value churn
prd-number-check / require-numbered-prds (pull_request) Successful in 9s
tracker-policy-pr / check-pr (pull_request) Successful in 10s
test / unit (pull_request) Successful in 49s
test / image-input-builds (pull_request) Successful in 50s
test / integration-docker (pull_request) Successful in 56s
test / coverage (pull_request) Successful in 15s
test / image-input-builds (push) Successful in 46s
Update Quality Badges / update-badges (push) Successful in 56s
test / coverage (push) Successful in 22s
test / integration-docker (push) Successful in 1m3s
test / unit (push) Successful in 48s
lint / lint (push) Successful in 2m59s
Lower the diff-coverage gate from 90% to 80% and the critical-module
target from 90% to 85%. The 90% diff gate forced back-fill tests on
nearly every changed line; 80% keeps new code honest without the churn.
Global coverage stays informational per ADR 0004 (no new gate added).

Updates scripts/diff_coverage.py, scripts/coverage.sh,
scripts/critical-modules.txt, .gitea/workflows/test.yml, and records the
change as a dated revision in ADR 0004.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-26 21:45:08 -04:00
didericis-claude ed9fc76f97 fix(docker): only heal on the ParseAddr poison signature, not any inspect error
prd-number-check / require-numbered-prds (pull_request) Successful in 5s
test / unit (pull_request) Successful in 50s
tracker-policy-pr / check-pr (pull_request) Successful in 6s
test / image-input-builds (pull_request) Successful in 59s
test / integration-docker (pull_request) Successful in 1m6s
test / coverage (pull_request) Successful in 42s
Update Quality Badges / update-badges (push) Successful in 49s
lint / lint (push) Successful in 1m2s
test / coverage (push) Successful in 24s
test / integration-docker (push) Successful in 1m3s
test / unit (push) Successful in 48s
test / image-input-builds (push) Successful in 2m44s
Address review: the self-heal classified every non-absence `network inspect`
failure as a poisoned network and force-removed the shared gateway. But
inspect also fails on transient daemon/API errors, permission failures,
timeouts, or a bad context — destroying a healthy gateway on that guess would
tear the network out from under every live bottle.

Now the destructive path runs only for the known poison signature (docker's
`ParseAddr` error from the malformed `::1/64` IPv6 gateway). The absent case
(`No such network`) still just creates; any other inspect failure raises a
clear GatewayError without mutating shared state.

Adds a regression test asserting a generic inspect error issues neither
`docker rm --force` nor `docker network rm` and surfaces the error.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 01:41:09 +00:00
didericis-claude bd8a146a46 fix(docker): heal a poisoned IPv6 gateway network, not just avoid creating one
test / integration-docker (pull_request) Successful in 1m15s
test / image-input-builds (pull_request) Successful in 1m14s
lint / lint (push) Successful in 3m28s
tracker-policy-pr / check-pr (pull_request) Failing after 10m28s
test / unit (pull_request) Failing after 10m39s
prd-number-check / require-numbered-prds (pull_request) Failing after 10m45s
test / coverage (pull_request) Has been skipped
PR #515 stopped bot-bottle from *creating* a gateway network with a
malformed IPv6 subnet (--ipv6=false), but it can't recover a network that
is *already* poisoned. On a daemon that default-enables IPv6, the fixed-name
`bot-bottle-gateway` network gets an `fdd0::/64` subnet whose `::1/64`
gateway trips docker's own netip.ParseAddr, so `docker network inspect`/`ls`
exit non-zero — poisoning every command that reads networks.

Such a network can survive on a shared runner from a pre-fix or concurrent
launch. `_ensure_network` never healed it: its migrate/recreate branch only
ran when `network inspect` *succeeded*, but a poisoned network makes inspect
*fail*, so the code fell through to `network create`, which no-ops on
"already exists" — leaving the poison in place. The next subnet read then
failed with ConsolidatedLaunchError (test_multitenant_isolation), and a bare
`docker network ls` failed too (test_orphan_cleanup).

Fix, two parts:
- gateway.py: when `network inspect` fails for a reason other than "no such
  network", treat the network as poisoned and force-remove + recreate it
  IPv4-only. Absent-vs-poisoned is distinguished by the inspect stderr.
- test.yml: add an integration-docker preflight that drops the leftover
  gateway network (and its container) before the suite, so direct `network
  ls`/`inspect` calls in tests are clean even on a daemon where the in-code
  heal can't run because inspect itself is what's broken.

Adds unit coverage for the poisoned-heal and the absent-create split.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 01:23:25 +00:00
137 changed files with 6438 additions and 1166 deletions
+16 -2
View File
@@ -102,6 +102,20 @@ jobs:
python3 --version
python3 cli.py backend status --backend=docker
- name: Preflight — clear any leftover poisoned gateway network
run: |
# The gateway network has a fixed name and persists across jobs on
# this shared runner. A pre-fix or concurrent launch can leave it with
# a malformed IPv6 subnet that trips docker's own ParseAddr in
# `network inspect` (see PR #515); the code now self-heals it, but the
# heal can't run if `network inspect` is what's broken on some daemon
# versions. Drop the network here so this run recreates it IPv4-only.
# Remove the attached gateway container first (else `network rm` fails
# on active endpoints); both are recreated by ensure_running. Harmless
# when absent.
docker rm --force bot-bottle-orch-gateway 2>/dev/null || true
docker network rm bot-bottle-gateway 2>/dev/null || true
- name: Run integration tests (docker) with coverage
env:
BOT_BOTTLE_BACKEND: docker
@@ -284,7 +298,7 @@ jobs:
- name: Combined coverage (unit + docker integration)
run: PYTHON=python3 bash scripts/coverage.sh aggregate critical
- name: Diff-coverage gate (changed lines >= 90%)
- name: Diff-coverage gate (changed lines >= 80%)
run: |
git fetch --no-tags origin main:refs/remotes/origin/main
python3 scripts/diff_coverage.py --base origin/main --min 90
python3 scripts/diff_coverage.py --base origin/main --min 80
+8 -6
View File
@@ -9,12 +9,9 @@
# Keeping the content in one place means future orchestrator deps (e.g.
# iroh) are added here once, not duplicated per backend.
#
# It stays deliberately lean: the control plane is **stdlib-only** today, so
# no third-party payload — none of the gateway's mitmproxy/git/gitleaks
# (that's Dockerfile.gateway) and no buildah (that's the firecracker
# builder, and lives only in Dockerfile.orchestrator.fc). Keeping the
# secret-dense control plane on a minimal dependency surface is the point
# (PRD 0070's "secret concentration").
# It stays deliberately lean: only the pinned FastAPI/Uvicorn control-plane
# stack is installed here — none of the gateway's mitmproxy/git/gitleaks
# (that's Dockerfile.gateway) and no buildah (that's firecracker-only).
#
# Shares an exact multi-architecture Python/trixie manifest with the gateway
# image. The version-qualified tag keeps the human-readable upstream version;
@@ -25,6 +22,11 @@ FROM ${PYTHON_BASE_IMAGE}
WORKDIR /app
COPY requirements.orchestrator.lock /tmp/requirements.orchestrator.lock
RUN pip install --no-cache-dir --require-hashes \
-r /tmp/requirements.orchestrator.lock \
&& rm /tmp/requirements.orchestrator.lock
# The orchestrator content. Baked so the image is self-contained (runs from
# a built image, no runtime bind-mount); the docker backend may still
# bind-mount /app for dev live-reload, which simply overlays this copy.
+22 -8
View File
@@ -15,7 +15,7 @@
## Features
- **Per-bottle egress allowlist** — TLS-bumped HTTP/HTTPS chokepoint with a per-manifest host allowlist; per-route path/method/header `matches` filtering; outbound DLP scanning for known tokens and secrets, inbound DLP scanning for prompt-injection attempts; DoH and arbitrary hosts blocked by default.
- **Per-route token-match policy** — each egress route picks what happens when the outbound DLP catches a token via `dlp.outbound_on_match`: `supervise` (default) holds the request and surfaces it in `./cli.py supervise` for approval (an approved value is remembered for the life of the proxy); `redact` scrubs the value and forwards; `block` is a hard `403`. Cuts false-positive friction without weakening default-deny.
- **Per-route token-match policy** — each egress route picks what happens when the outbound DLP catches a token via `dlp.outbound_on_match`: `supervise` (default) holds the request and surfaces it in `bot-bottle supervise` for approval (an approved value is remembered for the life of the proxy); `redact` scrubs the value and forwards; `block` is a hard `403`. Cuts false-positive friction without weakening default-deny.
- **Tokens the agent never sees** — host secrets live in a gateway; the agent dials `http://gateway:9099/<path>` and the proxy strips inbound `Authorization` and injects the real token before forwarding. `printenv` in the agent shows proxy URLs only.
- **Gitleaks-scanned push (git-gate)** — `bottle.git` remotes route through a per-bottle `git daemon` that gitleaks-scans incoming refs pre-receive and forwards clean refs upstream over SSH. The agent never holds the upstream credential.
- **Manifest-scoped skills + secrets** — each bottle declares its skills, env, git identity, remotes, and egress routes; unknown keys die at load.
@@ -39,7 +39,7 @@ On the legacy Docker backend, the same logical bottle is two containers per agen
The Docker topology looks like this:
```
host ( ./cli.py )
host ( bot-bottle )
starts │ stops
@@ -71,9 +71,23 @@ When the agent exits, `cli.py` tears down every gateway and both networks; nothi
## Quickstart
On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The Firecracker backend (Linux) requires Docker on the host for the gateway plus the `firecracker` binary and KVM. The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
```sh
curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh
```
Use `BOT_BOTTLE_BACKEND=docker ./cli.py start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
The installer is a bootstrapper: it finds a suitable Python, installs bot-bottle with `pipx` (falling back to `pip --user`), creates `~/.bot-bottle`, and runs `bot-bottle doctor`. It is idempotent and never uses `sudo`. Python-native users can skip it entirely with `pipx install bot-bottle` or `uv tool install bot-bottle`.
### Requirements
**Python ≥ 3.11**, and this is the one that trips people up on macOS: the `python3` Apple ships at `/usr/bin/python3` is **3.9.6**, which is too old. Bare `python3` resolves to that stub far more often than people expect. `path_helper` builds a login shell's `PATH` from `/etc/paths` and then appends `/etc/paths.d/*`, and `/usr/bin` sits in the former — so even when `/opt/homebrew/bin` *is* on the `PATH` (via `/etc/paths.d/homebrew`), it comes after `/usr/bin` and loses. Prepending a newer Python is something your shell profile does, and a fresh account, a launchd job, or a CI runner has no such profile. So the installer looks past bare `python3` before giving up: it tries `python3`, then the versioned `python3.11``python3.14` names, then `/opt/homebrew/bin`, `/usr/local/bin`, `~/.local/bin`, and python.org framework builds — and tells you which one it picked when it isn't the obvious one. Point it somewhere specific with `BOT_BOTTLE_PYTHON=/path/to/python3`.
**No `pipx` required.** If `pipx` is present the installer uses it and stays out of the way. If it isn't, bot-bottle installs into a private venv at `~/.bot-bottle/venv` (override with `BOT_BOTTLE_VENV`) and symlinks the entry point into `~/.local/bin`. There is deliberately no `pip install --user` path: Homebrew, python.org and Debian/Ubuntu interpreters are all externally managed (PEP 668), which blocks `--user` outright — so on a Mac it is never the fallback it appears to be. A venv is exempt from PEP 668, and `venv` is stdlib, so unlike `pipx` there is nothing to bootstrap first.
**`git`**, because the default install spec is a `git+` URL. Set `BOT_BOTTLE_INSTALL_SPEC` to a wheel path or index name to avoid it.
**A backend**, which the installer deliberately does *not* install for you — `doctor` reports what's missing afterwards. On compatible macOS hosts, the default backend requires Apple's `container` CLI and does not require Docker. The Firecracker backend (Linux) requires Docker on the host for the gateway plus the `firecracker` binary and KVM. The legacy Docker backend requires Docker. Claude bottles also need a long-lived Claude Code OAuth token (`claude setup-token`) exported as `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`.
Use `BOT_BOTTLE_BACKEND=docker bot-bottle start <agent>` on hosts where neither Apple Container nor KVM is available and Docker is the desired backend.
> **CI (macOS Apple Container):** the advisory `integration-macos` job in `.gitea/workflows/pre-release-test.yml` runs only on manual dispatch. It targets a self-hosted host-mode runner labelled `macos`; Apple Container cannot run inside the Linux pull-request runner. Provision an Apple Silicon host with the `container` CLI running and Python ≥ 3.11 plus `coverage` on the launchd service's explicit `PATH`. The infra container is a singleton (`bot-bottle-mac-infra`), so keep runner concurrency at 1. Its coverage is reported separately and never feeds the required pull-request gate.
@@ -166,10 +180,10 @@ On Linux, a KVM-capable host defaults to the Firecracker backend. It needs:
- **`/dev/kvm`** present and accessible. Load `kvm-intel` or `kvm-amd` (and enable virtualization in BIOS/firmware). The invoking user must be in the `kvm` group: `sudo usermod -aG kvm "$USER"` then re-login. bot-bottle preflights this and reports exactly what's missing.
- **`firecracker`** on `PATH`: grab a release from <https://github.com/firecracker-microvm/firecracker/releases>. Start flows print this pointer when the binary is missing.
- **Docker** for the gateway and image build.
- **A one-time privileged network setup** — the per-bottle TAP pool plus the fail-closed `nftables` isolation table. Run `./cli.py backend setup --backend=firecracker` for the host-appropriate config (a NixOS module, a `sudo` script elsewhere); `./cli.py backend status --backend=firecracker` reports what's present, including whether the pool range collides with an existing route. The pool defaults to `10.243.0.0/16` (an obscure RFC-1918 block that dodges docker/libvirt/LAN and, deliberately, Tailscale's `100.64.0.0/10` CGNAT range); override with `BOT_BOTTLE_FC_IP_BASE` if it clashes on your host.
- **A one-time privileged network setup** — the per-bottle TAP pool plus the fail-closed `nftables` isolation table. Run `bot-bottle backend setup --backend=firecracker` for the host-appropriate config (a NixOS module, a `sudo` script elsewhere); `bot-bottle backend status --backend=firecracker` reports what's present, including whether the pool range collides with an existing route. The pool defaults to `10.243.0.0/16` (an obscure RFC-1918 block that dodges docker/libvirt/LAN and, deliberately, Tailscale's `100.64.0.0/10` CGNAT range); override with `BOT_BOTTLE_FC_IP_BASE` if it clashes on your host.
```sh
BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
BOT_BOTTLE_BACKEND=firecracker bot-bottle start <agent>
```
> **NixOS:** enable `virtualisation.docker`, ensure the KVM module is loaded (`boot.kernelModules = [ "kvm-intel" ];` or `kvm-amd`), and add your user to the `kvm` and `docker` groups. For the network pool, consume the flake module — `imports = [ inputs.bot-bottle.nixosModules.firecracker-netpool ]; services.bot-bottle-firecracker = { enable = true; owner = "you"; };` — then `nixos-rebuild switch` (imperative nft/TAP rules don't survive a rebuild; channel users can `imports = [ <bot-bottle>/nix/firecracker-netpool.nix ]`). `firecracker` isn't in nixpkgs by default as a user binary — install the release binary (pin the version) and put it on `PATH`.
@@ -177,7 +191,7 @@ BOT_BOTTLE_BACKEND=firecracker ./cli.py start <agent>
> **CI:** Firecracker integration runs in the manually dispatched `.gitea/workflows/pre-release-test.yml` on a self-hosted runner labelled `kvm`; privileged KVM hosts never execute unreviewed PR code automatically. Provision it like a normal Firecracker host: `firecracker` on `PATH`, `/dev/kvm`, the cached guest kernel and static dropbear, and the persistent TAP/nft pool. The required pull-request workflow runs unit plus the complete Docker integration suite on `ubuntu-latest`; see `docs/ci.md`.
```sh
./cli.py start <agent> # builds the image on first run, drops you into claude
bot-bottle start <agent> # builds the image on first run, drops you into claude
```
## Manifest
@@ -253,7 +267,7 @@ You help maintain Gitea-hosted projects.
| `dlp.outbound_on_match` | no | What to do when an outbound token is detected: `supervise` (default for manifest routes — hold for operator approval), `redact` (scrub the value and forward), or `block` (hard 403). Agent-provider routes (e.g. `api.anthropic.com`) default to `redact`. |
| `git.fetch` | no | `true` permits smart HTTP clone/fetch (`git-upload-pack`) for this host. Push (`git-receive-pack`) remains blocked. |
When an outbound DLP detector matches a token, the route's `dlp.outbound_on_match` policy decides what happens. Under the default `supervise`, the proxy queues an `egress-token-allow` proposal for the operator's `./cli.py supervise` TUI and holds the request open until it is answered (or `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS`, default 300s, elapses — after which it fails closed). The operator never sees the raw token, only the host, method, path, and a redacted snippet; approving adds the value to an in-memory safelist for the life of the egress proxy. Under `redact`, the matched value is scrubbed from the body, headers, and path and the request is forwarded (failing closed if a match lands somewhere unredactable, like the hostname). Under `block` it stays a hard `403`. Structural blocks (CRLF injection) and not-in-allowlist host blocks are always hard `403`s regardless of policy.
When an outbound DLP detector matches a token, the route's `dlp.outbound_on_match` policy decides what happens. Under the default `supervise`, the proxy queues an `egress-token-allow` proposal for the operator's `bot-bottle supervise` TUI and holds the request open until it is answered (or `EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS`, default 300s, elapses — after which it fails closed). The operator never sees the raw token, only the host, method, path, and a redacted snippet; approving adds the value to an in-memory safelist for the life of the egress proxy. Under `redact`, the matched value is scrubbed from the body, headers, and path and the request is forwarded (failing closed if a match lands somewhere unredactable, like the hostname). Under `block` it stays a hard `403`. Structural blocks (CRLF injection) and not-in-allowlist host blocks are always hard `403`s regardless of policy.
More examples in `examples/`. Full design lives under `docs/prds/`; the trust-boundary rationale is in `docs/prds/0011-per-file-md-manifest.md`.
+1 -1
View File
@@ -226,7 +226,7 @@ class AgentProvider(ABC):
initial task in a non-interactive (headless) session.
Called only when ``--prompt`` is passed to
``./cli.py start --headless``; the returned args are appended
``bot-bottle start --headless``; the returned args are appended
after the provider's ``bypass_args`` and ``startup_args``."""
def provision_ca(self, bottle: "Bottle", plan: "BottlePlan") -> None:
+3
View File
@@ -30,6 +30,7 @@ if TYPE_CHECKING:
BottleImages,
BottlePlan,
BottleSpec,
EnumerationError,
ExecResult,
)
from .selection import (
@@ -59,6 +60,7 @@ _LAZY_MODULES: dict[str, str] = {
"BottleImages": "base",
"BottleBackend": "base",
"BackendStatus": "base",
"EnumerationError": "base",
"get_bottle_backend": "selection",
"known_backend_names": "selection",
"has_backend": "selection",
@@ -100,6 +102,7 @@ __all__ = [
"BottlePlan",
"BottleSpec",
"ExecResult",
"EnumerationError",
"CommitCancelled",
"Freezer",
"get_freezer",
+11 -3
View File
@@ -42,6 +42,10 @@ class BackendStatus(enum.IntEnum):
READY = 0
class EnumerationError(RuntimeError):
"""A backend could not produce an authoritative live-resource snapshot."""
@dataclass(frozen=True)
class BottleSpec:
"""CLI-supplied intent. Backend-agnostic — each backend's prepare
@@ -168,6 +172,10 @@ class BottleCleanupPlan(ABC):
"""True iff there is nothing to clean up; the CLI uses this to
short-circuit before showing the y/N."""
@abstractmethod
def intersect(self, current: "BottleCleanupPlan") -> "BottleCleanupPlan":
"""Resources both displayed to the operator and currently removable."""
@dataclass(frozen=True)
class ExecResult:
@@ -523,7 +531,7 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
host-appropriate config or commands. Prints to stdout/stderr and
returns a shell exit code (0 = nothing to report / success). A
backend that needs no host setup prints a short note and returns
0. Invoked generically by `./cli.py backend setup [--backend=…]`
0. Invoked generically by `bot-bottle backend setup [--backend=…]`
so operators can provision any backend without a
backend-specific command. Classmethod (like `is_available`) —
it's a host query, not per-bottle state."""
@@ -540,14 +548,14 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT]):
stderr. When quiet=True returns the status code silently —
useful for cheap programmatic checks.
Invoked by `./cli.py backend status [--backend=…]` (quiet=False)
Invoked by `bot-bottle backend status [--backend=…]` (quiet=False)
and by is_backend_ready() (caller-controlled)."""
@classmethod
@abstractmethod
def teardown(cls) -> int:
"""Undo `setup()` — the inverse operation, surfaced as
`./cli.py backend teardown [--backend=…]` (uninstall). Symmetric
`bot-bottle backend teardown [--backend=…]` (uninstall). Symmetric
with setup: where setup is advisory (prints the privileged
commands / declarative config to apply), teardown prints the
commands / config change to remove the host prerequisites. A
+60
View File
@@ -0,0 +1,60 @@
"""Shared destructive-cleanup execution and failure accounting."""
from __future__ import annotations
import os
import shutil
import subprocess
from collections.abc import Sequence
from pathlib import Path
class CleanupError(RuntimeError):
"""One or more approved cleanup mutations did not complete."""
class CleanupFailures:
"""Attempt every approved mutation, then fail with complete diagnostics."""
def __init__(self) -> None:
self._messages: list[str] = []
def run(self, argv: Sequence[str], description: str) -> None:
raw_timeout = os.environ.get(
"BOT_BOTTLE_CLEANUP_COMMAND_TIMEOUT_SECONDS", "120",
)
try:
timeout = float(raw_timeout)
except ValueError:
timeout = 120.0
try:
result = subprocess.run(
list(argv), capture_output=True, text=True, check=False,
timeout=max(timeout, 1.0),
)
except (OSError, subprocess.SubprocessError) as exc:
self._messages.append(f"{description}: {exc}")
return
if result.returncode != 0:
detail = (result.stderr or result.stdout).strip()
self._messages.append(
f"{description}: {detail or f'exit {result.returncode}'}"
)
def remove_tree(self, path: Path, description: str) -> None:
try:
shutil.rmtree(path)
except FileNotFoundError:
return
except OSError as exc:
self._messages.append(f"{description}: {exc}")
def record(self, message: str) -> None:
self._messages.append(message)
def raise_if_any(self) -> None:
if self._messages:
raise CleanupError("; ".join(self._messages))
__all__ = ["CleanupError", "CleanupFailures"]
@@ -46,6 +46,22 @@ class DockerBottleCleanupPlan(BottleCleanupPlan):
and not self.orphan_state_dirs
)
def intersect(self, current: BottleCleanupPlan) -> "DockerBottleCleanupPlan":
if not isinstance(current, DockerBottleCleanupPlan):
raise TypeError("cleanup plans must have the same backend type")
return DockerBottleCleanupPlan(
projects=tuple(x for x in self.projects if x in current.projects),
stray_containers=tuple(
x for x in self.stray_containers if x in current.stray_containers
),
stray_networks=tuple(
x for x in self.stray_networks if x in current.stray_networks
),
orphan_state_dirs=tuple(
x for x in self.orphan_state_dirs if x in current.orphan_state_dirs
),
)
def print(self) -> None:
print(file=sys.stderr)
for name in self.projects:
+38 -38
View File
@@ -23,11 +23,12 @@ Active-agent enumeration lives in `backend/docker/enumerate.py`.
from __future__ import annotations
import shutil
import subprocess
from ...paths import bot_bottle_root
from ...log import info, warn
from ...log import info
from .. import EnumerationError
from ..cleanup_control import CleanupFailures
from . import util as docker_mod
from .bottle_cleanup_plan import DockerBottleCleanupPlan
from ...bottle_state import bottle_state_dir, is_preserved
@@ -36,15 +37,17 @@ from .compose import COMPOSE_PROJECT_PREFIX, list_compose_projects
def _list_prefixed_containers() -> list[str]:
"""All bot-bottle-prefixed containers, running or stopped."""
result = subprocess.run(
["docker", "ps", "-a",
"--filter", f"name=^{COMPOSE_PROJECT_PREFIX}",
"--format", "{{.Names}}\t{{.Label \"com.docker.compose.project\"}}"],
capture_output=True, text=True, check=False,
)
try:
result = subprocess.run(
["docker", "ps", "-a",
"--filter", f"name=^{COMPOSE_PROJECT_PREFIX}",
"--format", "{{.Names}}\t{{.Label \"com.docker.compose.project\"}}"],
capture_output=True, text=True, check=False,
)
except OSError as exc:
raise EnumerationError(f"docker ps failed: {exc}") from exc
if result.returncode != 0:
warn(f"docker ps failed: {result.stderr.strip()}")
return []
raise EnumerationError(f"docker ps failed: {result.stderr.strip()}")
out: list[str] = []
for line in (result.stdout or "").splitlines():
if not line:
@@ -63,15 +66,19 @@ def _list_prefixed_networks() -> list[str]:
to a compose project. Compose-managed networks have a
`com.docker.compose.project` label; bare ones (from pre-compose
code paths) don't."""
result = subprocess.run(
["docker", "network", "ls",
"--filter", f"name={COMPOSE_PROJECT_PREFIX}",
"--format", "{{.Name}}\t{{.Label \"com.docker.compose.project\"}}"],
capture_output=True, text=True, check=False,
)
try:
result = subprocess.run(
["docker", "network", "ls",
"--filter", f"name={COMPOSE_PROJECT_PREFIX}",
"--format", "{{.Name}}\t{{.Label \"com.docker.compose.project\"}}"],
capture_output=True, text=True, check=False,
)
except OSError as exc:
raise EnumerationError(f"docker network ls failed: {exc}") from exc
if result.returncode != 0:
warn(f"docker network ls failed: {result.stderr.strip()}")
return []
raise EnumerationError(
f"docker network ls failed: {result.stderr.strip()}"
)
out: list[str] = []
for line in (result.stdout or "").splitlines():
if not line:
@@ -120,7 +127,10 @@ def prepare_cleanup() -> DockerBottleCleanupPlan:
`enumerate_active_agents()` so the orphan-state-dir bucket
doesn't include slugs whose non-docker bottle is still up."""
docker_mod.require_docker()
projects = list_compose_projects()
projects = list_compose_projects(
warn_on_error=False,
raise_on_error=True,
)
project_set = set(projects)
# Late import to avoid a circular at module-load time —
# the backend package's __init__ imports this module.
@@ -140,40 +150,30 @@ def cleanup(plan: DockerBottleCleanupPlan) -> None:
"""Remove everything in the plan. Projects first (whose `compose
down` reaps their containers + networks atomically), then stray
legacy resources, then orphan state dirs."""
failures = CleanupFailures()
for project in plan.projects:
info(f"docker compose down ({project})")
result = subprocess.run(
failures.run(
["docker", "compose", "-p", project, "down", "--volumes"],
capture_output=True, text=True, check=False,
f"docker compose down failed for {project}",
)
if result.returncode != 0:
warn(
f"compose down failed for {project}: "
f"{result.stderr.strip()}"
)
for name in plan.stray_containers:
info(f"removing stray container {name}")
subprocess.run(
failures.run(
["docker", "rm", "-f", name],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
f"removing stray container {name}",
)
for name in plan.stray_networks:
info(f"removing stray network {name}")
subprocess.run(
failures.run(
["docker", "network", "rm", name],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
f"removing stray network {name}",
)
for identity in plan.orphan_state_dirs:
path = bottle_state_dir(identity)
info(f"removing orphan state dir {path}")
try:
shutil.rmtree(path, ignore_errors=True)
except OSError as e:
warn(f"failed to remove {path}: {e}")
failures.remove_tree(path, f"removing orphan state dir {path}")
failures.raise_if_any()
+30 -13
View File
@@ -16,6 +16,7 @@ from pathlib import Path
from typing import Any
from ...log import die, warn
from ..base import EnumerationError
# --- Lifecycle helpers (PRD 0018 chunk 3) ----------------------------------
@@ -52,19 +53,20 @@ def slug_from_compose_project(project: str) -> str:
def list_compose_projects(
*, include_stopped: bool = True, warn_on_error: bool = True,
*,
include_stopped: bool = True,
warn_on_error: bool = True,
raise_on_error: bool = False,
) -> list[str]:
"""All compose project names starting with `bot-bottle-`.
`include_stopped=True` (default) runs `docker compose ls --all`
so exited projects appear too; pass False to get only projects
with at least one running container.
Returns [] on docker daemon errors or malformed output rather
than raising — callers should treat the empty list as "no
projects discoverable", not "no projects exist". `warn_on_error`
stays true for explicit operator commands like cleanup, but active
discovery paths set it false so dashboard refreshes don't spam
stderr while Docker Desktop is stopped."""
Best-effort callers get ``[]`` on Docker errors or malformed output.
Enumeration callers pass ``raise_on_error=True`` so a failed query is not
reported as an authoritative empty result.
"""
argv = ["docker", "compose", "ls", "--format", "json"]
if include_stopped:
argv.insert(3, "--all")
@@ -72,19 +74,30 @@ def list_compose_projects(
result = subprocess.run(
argv, capture_output=True, text=True, check=False,
)
except FileNotFoundError:
# docker binary not on PATH — same shape as a daemon-down
# error from the caller's POV: no projects discoverable.
except OSError as exc:
# Not only "not found": docker on PATH but not executable by this
# user raises PermissionError. Either way the query never ran, so an
# enumeration caller must not read the empty result as authoritative.
if raise_on_error:
raise EnumerationError(
f"docker compose ls failed: docker unavailable ({exc})"
) from exc
return []
if result.returncode != 0:
message = f"docker compose ls failed: {result.stderr.strip()}"
if raise_on_error:
raise EnumerationError(message)
if warn_on_error:
warn(f"docker compose ls failed: {result.stderr.strip()}")
warn(message)
return []
try:
projects = json.loads(result.stdout or "[]")
except json.JSONDecodeError as e:
message = f"docker compose ls returned malformed JSON: {e}"
if raise_on_error:
raise EnumerationError(message) from e
if warn_on_error:
warn(f"docker compose ls returned malformed JSON: {e}")
warn(message)
return []
names: list[str] = []
for p in projects:
@@ -97,7 +110,10 @@ def list_compose_projects(
def list_active_slugs(
*, include_stopped: bool = False, warn_on_error: bool = True,
*,
include_stopped: bool = False,
warn_on_error: bool = True,
raise_on_error: bool = False,
) -> list[str]:
"""Slugs (project name minus prefix) of currently-running
bottles. Used by the dashboard's operator-edit verbs to choose
@@ -108,6 +124,7 @@ def list_active_slugs(
for p in list_compose_projects(
include_stopped=include_stopped,
warn_on_error=warn_on_error,
raise_on_error=raise_on_error,
)
) if slug
)
@@ -68,6 +68,11 @@ def _network_container_ips(network: str) -> list[str]:
"docker", "network", "inspect", "--format",
"{{range .Containers}}{{.IPv4Address}} {{end}}", network,
])
if proc.returncode != 0:
detail = proc.stderr.strip() or f"exit {proc.returncode}"
raise ConsolidatedLaunchError(
f"could not inspect addresses on gateway network {network}: {detail}"
)
ips: list[str] = []
for entry in proc.stdout.split():
ips.append(entry.split("/", 1)[0])
+13 -12
View File
@@ -1,9 +1,8 @@
"""Active-agent enumeration for the docker backend.
Returns `ActiveAgent` records the CLI `active` command and the
dashboard agents pane consume. Empty when docker isn't reachable
— gated by `has_backend('docker')` at the cross-backend caller
so this module trusts that docker is available when called.
dashboard agents pane consume. Docker query failures raise rather
than masquerading as an authoritative empty result.
The parser (`_parse_services_by_project`) is exposed for direct
unit testing; the docker `docker ps` invocation is in
@@ -13,17 +12,18 @@ from __future__ import annotations
import subprocess
from .. import ActiveAgent
from .. import ActiveAgent, EnumerationError
from ...bottle_state import read_metadata
from .compose import compose_project_name, list_active_slugs
def enumerate_active() -> list[ActiveAgent]:
"""All currently-running docker-backed agents. Caller is
responsible for gating on `has_backend('docker')` if it
matters; if docker is missing the `docker ps` call below
returns an empty list silently."""
slugs = list_active_slugs(include_stopped=False, warn_on_error=False)
"""All currently-running docker-backed agents."""
slugs = list_active_slugs(
include_stopped=False,
warn_on_error=False,
raise_on_error=True,
)
if not slugs:
return []
services_by_project = _query_services_by_project()
@@ -74,8 +74,9 @@ def _query_services_by_project() -> dict[str, set[str]]:
],
capture_output=True, text=True, check=False,
)
except FileNotFoundError:
return {}
except OSError as exc:
# Missing, or on PATH but not executable by this user (PermissionError).
raise EnumerationError(f"docker ps failed: docker unavailable ({exc})") from exc
if r.returncode != 0:
return {}
raise EnumerationError(f"docker ps failed: {r.stderr.strip()}")
return _parse_services_by_project(r.stdout or "")
+35 -4
View File
@@ -140,12 +140,43 @@ class DockerGateway(Gateway):
marker = inspected.stdout.strip()
if marker in {"", self._subnet}:
return
if inspected.returncode == 0:
# Migrate the stale auto-IPAM network created by older releases.
# Removing the fixed gateway is safe here: this launch recreates it.
# Inspectable but mislabelled: the stale auto-IPAM network created
# by older releases. Replace it below.
stale = True
else:
# inspect failed. Classify by stderr — do NOT assume "not absent"
# implies "poisoned": a transient daemon/API error, permission
# failure, timeout, or bad context also fails here, and destroying
# the shared gateway on that guess would tear the network out from
# under every live bottle.
err = inspected.stderr.lower()
if "no such network" in err or "not found" in err:
# Absent: nothing to replace — create it below.
stale = False
elif "parseaddr" in err:
# Present but poisoned. A daemon that default-enables IPv6
# attaches an fdd0::/64 subnet whose `::1/64` gateway trips
# docker's own netip.ParseAddr in `network inspect`/`ls`, so the
# command exits non-zero with that signature. A fixed release
# never *creates* such a network, but one can survive on a
# shared host from an older or concurrent launch — and
# `--ipv6=false` alone can't heal it, since the create below only
# no-ops on "already exists". Force-replace it so later reads
# (e.g. `_network_cidr` pinning a source IP) stop failing.
stale = True
else:
# Unrecognized failure: no evidence the network is malformed.
# Surface it rather than mutate shared state on a guess.
raise GatewayError(
f"gateway network {self.network} could not be inspected: "
f"{inspected.stderr.strip()}"
)
if stale:
# Migrate the stale/poisoned network. Removing the fixed gateway is
# safe here: this launch recreates it.
run_docker(["docker", "rm", "--force", self.name])
removed = run_docker(["docker", "network", "rm", self.network])
if removed.returncode != 0:
if removed.returncode != 0 and "no such network" not in removed.stderr.lower():
raise GatewayError(
f"gateway network {self.network} needs explicit subnet "
f"{self._subnet} but could not be replaced: "
+3 -3
View File
@@ -8,7 +8,7 @@ pointer, and `status()` reports whether docker is usable.
This is intentionally minimal; a richer version (daemon config checks,
gVisor/runsc install guidance, rootless-docker hints) is tracked
separately. Reached via `DockerBottleBackend.setup` / `.status`, which
the generic `./cli.py backend {setup,status}` dispatches to.
the generic `bot-bottle backend {setup,status}` dispatches to.
"""
from __future__ import annotations
@@ -69,7 +69,7 @@ def teardown() -> int:
sys.stderr.write(
"Docker backend: nothing to undo — it provisions no privileged host "
"state (networks and the gateway are per-launch and are "
"removed by `./cli.py cleanup`). Docker itself is left installed.\n"
"removed by `bot-bottle cleanup`). Docker itself is left installed.\n"
)
return 0
@@ -89,5 +89,5 @@ def status() -> int:
runsc = _docker_on_path() and _util.runsc_available()
sys.stderr.write(f"gVisor runsc runtime: {'registered' if runsc else 'not registered (optional)'}\n")
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=docker\n")
sys.stderr.write("\nRun: bot-bottle backend setup --backend=docker\n")
return 0 if ok else 1
@@ -27,3 +27,11 @@ class FirecrackerBottleCleanupPlan(BottleCleanupPlan):
@property
def empty(self) -> bool:
return not (self.vm_pids or self.run_dirs)
def intersect(self, current: BottleCleanupPlan) -> "FirecrackerBottleCleanupPlan":
if not isinstance(current, FirecrackerBottleCleanupPlan):
raise TypeError("cleanup plans must have the same backend type")
return FirecrackerBottleCleanupPlan(
vm_pids=tuple(x for x in self.vm_pids if x in current.vm_pids),
run_dirs=tuple(x for x in self.run_dirs if x in current.run_dirs),
)
+101 -27
View File
@@ -11,10 +11,9 @@ Reaps *orphans* only — resources with no live VM behind them:
— a VMM left lingering after its dir was removed.
A run dir with a *live* firecracker process is a running bottle and is
left strictly alone: it is neither killed nor removed. (The backend's
`enumerate_active` registry is still a stub — #354 — so a live process
is the only reliable "this bottle is in use" signal we have. Once the
registry lands, registry-orphaned-but-running VMs can be reaped too.)
left strictly alone: it is neither killed nor removed. Active-agent
enumeration uses this same process snapshot, so cleanup and generic
backend consumers agree about which bottles are running.
TAP slots free themselves (the flock drops when the launcher exits), so
there is nothing to reclaim there.
@@ -22,14 +21,16 @@ there is nothing to reclaim there.
from __future__ import annotations
from collections.abc import Sequence
import os
import shutil
import signal
import subprocess
from pathlib import Path
from ...log import info
from . import util
from .. import EnumerationError
from ..cleanup_control import CleanupError, CleanupFailures
from . import lifecycle_lock, util
from .bottle_cleanup_plan import FirecrackerBottleCleanupPlan
@@ -37,7 +38,7 @@ def _run_root() -> Path:
return util.cache_dir() / "run"
def _run_dir_of(cmd: str, run_root: Path) -> Path | None:
def _run_dir_of(args: Sequence[str], run_root: Path) -> Path | None:
"""The bottle run dir a firecracker cmdline belongs to, or None.
A bottle VM is launched with `--config-file <run_root>/<slug>/config.json`,
@@ -45,15 +46,35 @@ def _run_dir_of(cmd: str, run_root: Path) -> Path | None:
the run root. Anything else (a builder VM, the infra VM elsewhere) is
not ours to reap here.
"""
toks = cmd.split()
for i, tok in enumerate(toks):
if tok == "--config-file" and i + 1 < len(toks):
parent = Path(toks[i + 1]).parent
for i, arg in enumerate(args):
if arg == "--config-file" and i + 1 < len(args):
parent = Path(args[i + 1]).parent
if parent.parent == run_root:
return parent
return None
def _decode_cmdline(raw: bytes) -> tuple[str, ...]:
"""Decode Linux's NUL-delimited argv without losing embedded spaces."""
return tuple(
value.decode(errors="surrogateescape")
for value in raw.split(b"\0") if value
)
def _process_args(pid: int) -> tuple[str, ...] | None:
"""Read one process's lossless argv, or None when it exited meanwhile."""
try:
raw = Path(f"/proc/{pid}/cmdline").read_bytes()
except FileNotFoundError:
return None
except OSError as exc:
raise EnumerationError(
f"could not inspect Firecracker pid {pid}: {exc}"
) from exc
return _decode_cmdline(raw)
def _scan_processes(run_root: Path) -> tuple[set[str], list[int]]:
"""Inspect running firecracker VMs under ``run_root``.
@@ -62,23 +83,34 @@ def _scan_processes(run_root: Path) -> tuple[set[str], list[int]]:
* ``orphan_pids`` — firecracker pids whose run dir no longer exists
(a lingering VMM to kill).
"""
result = subprocess.run(
["pgrep", "-a", "firecracker"],
capture_output=True, text=True, check=False,
)
if result.returncode != 0:
try:
result = subprocess.run(
["pgrep", "firecracker"],
capture_output=True, text=True, check=False,
)
except OSError as exc:
raise EnumerationError(
f"could not enumerate Firecracker processes: {exc}"
) from exc
if result.returncode == 1:
# pgrep's documented "no processes matched" result.
return set(), []
if result.returncode != 0:
detail = (result.stderr or "").strip() or f"exit {result.returncode}"
raise EnumerationError(
f"could not enumerate Firecracker processes: {detail}"
)
live: set[str] = set()
orphan_pids: list[int] = []
for line in result.stdout.splitlines():
parts = line.split(None, 1)
if len(parts) != 2:
continue
try:
pid = int(parts[0])
pid = int(line.strip())
except ValueError:
continue
run_dir = _run_dir_of(parts[1], run_root)
args = _process_args(pid)
if args is None:
continue
run_dir = _run_dir_of(args, run_root)
if run_dir is None:
continue
if run_dir.is_dir():
@@ -114,12 +146,54 @@ def prepare_cleanup() -> FirecrackerBottleCleanupPlan:
def cleanup(plan: FirecrackerBottleCleanupPlan) -> None:
for pid in plan.vm_pids:
"""Revalidate the preview under the launch lock, then remove its survivors."""
with lifecycle_lock.hold():
fresh = prepare_cleanup()
approved_pids = set(plan.vm_pids).intersection(fresh.vm_pids)
approved_dirs = set(plan.run_dirs).intersection(fresh.run_dirs)
failures = CleanupFailures()
for pid in sorted(approved_pids):
try:
_terminate_orphan(pid, _run_root())
except CleanupError as exc:
failures.record(str(exc))
for path in sorted(approved_dirs):
info(f"rm -rf {path}")
failures.remove_tree(Path(path), f"removing Firecracker run dir {path}")
failures.raise_if_any()
def _terminate_orphan(pid: int, run_root: Path) -> None:
"""Signal exactly the process identity that still owns an orphan config."""
try:
pidfd = os.pidfd_open(pid)
except ProcessLookupError:
return
except OSError as exc:
raise EnumerationError(
f"could not pin Firecracker pid {pid} for cleanup: {exc}"
) from exc
try:
try:
raw = Path(f"/proc/{pid}/cmdline").read_bytes()
except FileNotFoundError:
return
except OSError as exc:
raise EnumerationError(
f"could not revalidate Firecracker pid {pid}: {exc}"
) from exc
args = _decode_cmdline(raw)
run_dir = _run_dir_of(args, run_root)
if run_dir is None or run_dir.is_dir():
return
info(f"kill firecracker VM pid {pid}")
try:
os.kill(pid, signal.SIGTERM)
signal.pidfd_send_signal(pidfd, signal.SIGTERM)
except ProcessLookupError:
pass
for path in plan.run_dirs:
info(f"rm -rf {path}")
shutil.rmtree(path, ignore_errors=True)
return
except OSError as exc:
raise CleanupError(
f"could not signal Firecracker pid {pid}: {exc}"
) from exc
finally:
os.close(pidfd)
+22 -4
View File
@@ -1,14 +1,32 @@
"""Active-agent enumeration for the Firecracker backend.
The backend is disabled during the companion-container removal (#385) — it can't
launch bottles, so there are none to enumerate. Real enumeration returns
with the backend's consolidated relaunch (#354).
Running bottles are the Firecracker processes whose ``--config-file`` points
at an existing per-bottle run directory. The same authoritative process scan
protects cleanup from deleting live VMs; operational scan failures propagate
as ``EnumerationError`` instead of masquerading as an empty host.
"""
from __future__ import annotations
from ...bottle_state import read_metadata
from .. import ActiveAgent
from .cleanup import live_run_dirs
def enumerate_active() -> list[ActiveAgent]:
return []
out: list[ActiveAgent] = []
for run_dir in live_run_dirs():
slug = run_dir.name
metadata = read_metadata(slug)
out.append(ActiveAgent(
backend_name="firecracker",
slug=slug,
agent_name=metadata.agent_name if metadata else "?",
started_at=metadata.started_at if metadata else "",
# Firecracker uses the shared gateway, so there are no
# per-bottle gateway service containers to report.
services=(),
label=metadata.label if metadata else "",
color=metadata.color if metadata else "",
))
return out
@@ -41,6 +41,8 @@ from ... import resources
from ...log import die, info
from . import util
ARTIFACT_HTTP_TIMEOUT_SECONDS = 30.0
# Bump if the on-disk artifact *format* changes (compression, layout) so a new
# scheme can't collide with a cached/published artifact of the old one.
_ARTIFACT_FORMAT = "1"
@@ -54,6 +56,7 @@ _BUILD_INPUTS = {
"image-build-args.json",
"Dockerfile.orchestrator",
"Dockerfile.orchestrator.fc",
"requirements.orchestrator.lock",
),
"gateway": (
"image-build-args.json",
@@ -163,7 +166,9 @@ def _download(url: str, dest: Path) -> None:
"""Stream `url` to `dest` (atomic via a `.part` sibling)."""
tmp = dest.with_suffix(dest.suffix + ".part")
try:
with urllib.request.urlopen(_open(url)) as resp, open(tmp, "wb") as out:
with urllib.request.urlopen(
_open(url), timeout=ARTIFACT_HTTP_TIMEOUT_SECONDS,
) as resp, open(tmp, "wb") as out:
shutil.copyfileobj(resp, out, _CHUNK)
except urllib.error.HTTPError as e:
tmp.unlink(missing_ok=True)
+1 -1
View File
@@ -204,7 +204,7 @@ def boot_vm(
Records the PID."""
if not netpool.tap_present(slot.iface):
die(f"infra link {slot.iface} not present.\n"
f" ./cli.py backend setup --backend=firecracker")
f" bot-bottle backend setup --backend=firecracker")
run_dir.mkdir(parents=True, exist_ok=True)
rootfs = run_dir / "rootfs.ext4"
@@ -108,7 +108,7 @@ def verify_isolation(private_key: Path, guest_ip: str) -> None:
die(f"ISOLATION FAILURE: the VM reached the host canary "
f"{canary_ip}:{canary_port}. The egress boundary is not in "
f"force — refusing to run the agent (fail-closed). Verify the "
f"nft table with: ./cli.py backend setup --backend=firecracker")
f"nft table with: bot-bottle backend setup --backend=firecracker")
if result.returncode == 2:
die("isolation probe inconclusive: the guest has no python3/bash/nc "
"to run the connectivity test. Refusing to continue "
+23 -19
View File
@@ -46,7 +46,7 @@ from ...log import die, info, warn
from ...supervisor.types import SUPERVISE_PORT
from ..docker.egress import EGRESS_PORT
from ..util import AGENT_CA_BUNDLE, AGENT_CA_PATH
from . import firecracker_vm, image_builder, isolation_probe, netpool, util
from . import firecracker_vm, image_builder, isolation_probe, lifecycle_lock, netpool, util
from .bottle import FirecrackerBottle
from .bottle_plan import FirecrackerBottlePlan
from ...orchestrator.store.config_store import resolve_teardown_timeout
@@ -164,25 +164,29 @@ def launch(
)
# Step 6: build the per-bottle rootfs + SSH key, then boot.
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
# Remove the run dir on teardown so the per-bottle rootfs.ext4 (~1G)
# doesn't leak. Registered before vm.terminate below so it runs *after*
# it (ExitStack is LIFO): the VM is gone before we rm its rootfs.
stack.callback(lambda: shutil.rmtree(run_dir, ignore_errors=True))
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(agent_base, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
# Cleanup takes the same lock while refreshing its process snapshot.
# Hold it until the VMM exists so a newly-created run dir can never be
# mistaken for an orphan in the build-before-boot window.
with lifecycle_lock.hold():
run_dir = util.cache_dir() / "run" / plan.slug
run_dir.mkdir(parents=True, exist_ok=True)
# Remove the run dir on teardown so the per-bottle rootfs.ext4 (~1G)
# doesn't leak. Registered before vm.terminate below so it runs
# *after* it (ExitStack is LIFO).
stack.callback(lambda: shutil.rmtree(run_dir, ignore_errors=True))
rootfs = run_dir / "rootfs.ext4"
util.build_rootfs_ext4(agent_base, rootfs)
private_key, pubkey = util.generate_keypair(run_dir)
vm = firecracker_vm.boot(
name=plan.container_name,
rootfs=rootfs,
tap=slot.iface,
guest_ip=slot.guest_ip,
host_ip=slot.host_ip,
pubkey=pubkey,
run_dir=run_dir,
)
vm = firecracker_vm.boot(
name=plan.container_name,
rootfs=rootfs,
tap=slot.iface,
guest_ip=slot.guest_ip,
host_ip=slot.host_ip,
pubkey=pubkey,
run_dir=run_dir,
)
stack.callback(vm.terminate)
firecracker_vm.wait_for_ssh(vm, private_key)
persist_env_var_secret(private_key, slot.guest_ip, ctx.env_var_secret)
@@ -0,0 +1,30 @@
"""Serialize Firecracker run-directory creation with orphan cleanup."""
from __future__ import annotations
import fcntl
from contextlib import contextmanager
from pathlib import Path
from typing import Generator
from . import util
def _lock_path() -> Path:
return util.cache_dir() / "run.lifecycle.lock"
@contextmanager
def hold() -> Generator[None]:
"""Exclude cleanup while a launch directory lacks a visible VMM."""
path = _lock_path()
path.parent.mkdir(parents=True, exist_ok=True)
with path.open("a", encoding="utf-8") as handle:
fcntl.flock(handle, fcntl.LOCK_EX)
try:
yield
finally:
fcntl.flock(handle, fcntl.LOCK_UN)
__all__ = ["hold"]
+14 -6
View File
@@ -3,7 +3,7 @@ config renderers (shell command + NixOS module) shown to operators.
The Firecracker backend needs a privileged one-time network setup:
a pool of point-to-point TAP devices (owned by the invoking user, so
`./cli.py start` never needs root) and a dedicated nftables table that
`bot-bottle start` never needs root) and a dedicated nftables table that
isolates every VM. The pool parameters live in exactly one place —
`netpool.defaults.env`, a plain KEY=VALUE file next to this module —
and every consumer reads *that*: this module (below), the shell script
@@ -177,14 +177,20 @@ def gw_slot() -> Slot:
# --- fail-closed verification ---------------------------------------
def _run_ok(argv: list[str]) -> bool:
"""Run a probe command, treating a missing binary as failure
"""Run a probe command, treating an unavailable binary as failure
(rather than crashing) so callers can stay fail-closed."""
try:
return subprocess.run(
argv, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL,
check=False,
).returncode == 0
except FileNotFoundError:
except OSError:
# Not only "missing". A name on PATH that isn't executable by this
# user raises PermissionError, and CPython reports that EACCES in
# preference to the ENOENT from the other PATH entries — which is
# how `doctor` came to die with a traceback on a fresh macOS
# account. Any OSError means the probe couldn't run, which for a
# fail-closed check is indistinguishable from "not present".
return False
@@ -244,7 +250,9 @@ def overlapping_routes() -> list[RouteConflict]:
["ip", "-json", "route", "show", "table", "all"],
capture_output=True, text=True, check=False,
)
except FileNotFoundError:
except OSError:
# Missing, or present-but-not-executable for this user; either way
# there are no routes we can enumerate. See _run_ok.
return []
if proc.returncode != 0 or not proc.stdout.strip():
return []
@@ -307,11 +315,11 @@ def allocate(slug: str) -> tuple[Slot, IO[str]]:
return s, handle
die(f"Firecracker TAP pool exhausted ({pool_size()} slots, all in "
f"use). Stop a running bottle or raise BOT_BOTTLE_FC_POOL_SIZE "
f"and re-run `./cli.py backend setup --backend=firecracker`.")
f"and re-run `bot-bottle backend setup --backend=firecracker`.")
raise AssertionError("unreachable")
# --- config renderers (shown by `./cli.py backend setup`) -----------
# --- config renderers (shown by `bot-bottle backend setup`) -----------
# The persistent unit is the portable install: the same systemd oneshot
# on every systemd distro (Debian/Ubuntu/Fedora/RHEL/Arch/…).
@@ -34,6 +34,7 @@ from pathlib import Path
from . import infra_artifact, infra_vm, util
_CHUNK = 1 << 20
_REGISTRY_HTTP_TIMEOUT_SECONDS = 30.0
_GZ_NAME = "rootfs.ext4.gz"
_SHA_NAME = "rootfs.ext4.gz.sha256"
@@ -91,7 +92,9 @@ def _put(url: str, body: "bytes | Path", token: str) -> None:
req.add_header("Authorization", f"token {token}")
req.add_header("Content-Type", "application/octet-stream")
try:
with urllib.request.urlopen(req) as resp:
with urllib.request.urlopen(
req, timeout=_REGISTRY_HTTP_TIMEOUT_SECONDS,
) as resp:
print(f" uploaded {url} (HTTP {resp.status})")
except urllib.error.HTTPError as e:
if e.code == 409:
@@ -112,7 +115,9 @@ def _delete(url: str, token: str) -> None:
if token:
req.add_header("Authorization", f"token {token}")
try:
with urllib.request.urlopen(req):
with urllib.request.urlopen(
req, timeout=_REGISTRY_HTTP_TIMEOUT_SECONDS,
):
pass
except urllib.error.HTTPError as e:
if e.code != 404:
@@ -151,7 +156,10 @@ def _try_download_published(role: str, role_dir: Path) -> str | None:
version = _role_version(role)
sha_url = infra_artifact.artifact_url(version, _SHA_NAME, role=role)
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)):
with urllib.request.urlopen(
infra_artifact._open(sha_url),
timeout=_REGISTRY_HTTP_TIMEOUT_SECONDS,
):
pass
except urllib.error.HTTPError as e:
if e.code == 404:
@@ -195,7 +203,10 @@ def _publish_bundle(role: str, role_dir: Path, token: str) -> str:
# present, a re-publish is a no-op. Otherwise clear any partial upload left
# by an interrupted prior attempt and upload the complete set.
try:
with urllib.request.urlopen(infra_artifact._open(sha_url)) as resp:
with urllib.request.urlopen(
infra_artifact._open(sha_url),
timeout=_REGISTRY_HTTP_TIMEOUT_SECONDS,
) as resp:
remote_sha = resp.read().decode("utf-8").split()[0].strip().lower()
except urllib.error.HTTPError as e:
if e.code != 404:
+9 -6
View File
@@ -8,7 +8,7 @@ bundled setup script. `status()` reports what's present, including
whether the pool range collides with an existing route.
Called through `FirecrackerBottleBackend.setup` / `.status`, which the
generic `./cli.py backend {setup,status}` command dispatches to.
generic `bot-bottle backend {setup,status}` command dispatches to.
"""
from __future__ import annotations
@@ -20,6 +20,7 @@ import subprocess
import sys
from pathlib import Path
from ... import invocation
from ... import resources
from . import netpool
from . import util
@@ -34,7 +35,7 @@ _UNIT_PATH = Path("/etc/systemd/system") / netpool.SYSTEMD_UNIT
def _owner() -> str:
# Under `sudo`, USER is root but SUDO_USER is the real invoker — the
# TAPs must be owned by them so `./cli.py start` stays rootless.
# TAPs must be owned by them so `bot-bottle start` stays rootless.
return os.environ.get("SUDO_USER") or os.environ.get("USER") or "youruser"
@@ -161,7 +162,7 @@ def _setup_systemd() -> None:
if rc == 0:
sys.stderr.write(
f"Installed and started {netpool.SYSTEMD_UNIT}. Verify with "
f"`./cli.py backend status --backend=firecracker`.\n"
f"`bot-bottle backend status --backend=firecracker`.\n"
)
else:
sys.stderr.write(
@@ -179,9 +180,11 @@ def _setup_systemd() -> None:
f"sudo systemctl daemon-reload\n"
f"sudo systemctl enable --now {netpool.SYSTEMD_UNIT}\n"
)
# Absolute path, not `sudo bot-bottle`: sudo's secure_path drops
# ~/.local/bin, where both pipx and install.sh put the entry point.
sys.stderr.write(
f"\n(Or re-run this as root to install it directly: "
f"sudo ./cli.py backend setup --backend=firecracker)\n"
f"\n(Or re-run this as root to install it directly:\n"
f" {invocation.sudo_command('backend', 'setup', '--backend=firecracker')})\n"
)
@@ -334,7 +337,7 @@ def status() -> int:
sys.stderr.write(f"range overlap: none (base {netpool.ip_base()})\n")
_report_persistence()
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=firecracker\n")
sys.stderr.write("\nRun: bot-bottle backend setup --backend=firecracker\n")
return 0 if ok else 1
+4 -4
View File
@@ -9,7 +9,7 @@ generation.
The privileged network setup (TAP pool + nft table) is a one-time
operator step see `netpool.py`, `scripts/firecracker-netpool.sh`,
and `./cli.py backend setup --backend=firecracker`.
and `bot-bottle backend setup --backend=firecracker`.
"""
from __future__ import annotations
@@ -132,18 +132,18 @@ def _require_network_pool() -> None:
f"{netpool.pool_size()} slots) overlaps existing routes: "
f"{detail}. This can shadow or be shadowed by that route; "
f"set BOT_BOTTLE_FC_IP_BASE to a free range and re-run "
f"./cli.py backend setup --backend=firecracker.")
f"bot-bottle backend setup --backend=firecracker.")
missing = netpool.missing_taps()
if missing:
die(f"network pool incomplete — missing TAP devices: "
f"{', '.join(missing)}.\n ./cli.py backend setup --backend=firecracker")
f"{', '.join(missing)}.\n bot-bottle backend setup --backend=firecracker")
if shutil.which("nft") is not None and not netpool.nft_table_present():
# nft is queryable and says the table is absent — that's a
# definite, catchable misconfiguration; fail early.
warn(f"isolation table `inet {netpool.NFT_TABLE}` not found via nft. "
"If this is a permissions issue it will be re-checked "
"empirically after boot; otherwise run: "
"./cli.py backend setup --backend=firecracker")
"bot-bottle backend setup --backend=firecracker")
# --- rootfs pipeline (rootless) -------------------------------------
+2 -2
View File
@@ -43,7 +43,7 @@ class Freezer(ABC):
Calls _freeze for the backend-specific snapshot, then writes the
committed image reference to per-bottle state and marks the bottle
preserved so the next `./cli.py resume` boots from the snapshot.
preserved so the next `bot-bottle resume` boots from the snapshot.
Raises CommitCancelled if the user declines an interactive
confirmation prompt (e.g. the macos-container stop prompt).
@@ -51,7 +51,7 @@ class Freezer(ABC):
image_ref = self._freeze(agent)
write_committed_image(agent.slug, image_ref)
mark_preserved(agent.slug)
info(f"to resume from this snapshot: ./cli.py resume {agent.slug}")
info(f"to resume from this snapshot: bot-bottle resume {agent.slug}")
self._export_hint(agent.slug, image_ref)
@abstractmethod
@@ -25,3 +25,11 @@ class MacosContainerBottleCleanupPlan(BottleCleanupPlan):
@property
def empty(self) -> bool:
return not self.containers and not self.networks
def intersect(self, current: BottleCleanupPlan) -> "MacosContainerBottleCleanupPlan":
if not isinstance(current, MacosContainerBottleCleanupPlan):
raise TypeError("cleanup plans must have the same backend type")
return MacosContainerBottleCleanupPlan(
containers=tuple(x for x in self.containers if x in current.containers),
networks=tuple(x for x in self.networks if x in current.networks),
)
+13 -12
View File
@@ -4,7 +4,9 @@ from __future__ import annotations
import subprocess
from ...log import info, warn
from .. import EnumerationError
from ..cleanup_control import CleanupFailures
from ...log import info
from . import util as container_mod
from .bottle_cleanup_plan import MacosContainerBottleCleanupPlan
@@ -19,8 +21,8 @@ def _list_prefixed_containers() -> list[str]:
check=False,
)
if result.returncode != 0:
warn(f"container list failed: {result.stderr.strip()}")
return []
detail = result.stderr.strip() or f"exit {result.returncode}"
raise EnumerationError(f"container list failed: {detail}")
return sorted(
name for name in (line.strip() for line in result.stdout.splitlines())
if name.startswith(_PREFIX)
@@ -35,7 +37,8 @@ def _list_prefixed_networks() -> list[str]:
check=False,
)
if result.returncode != 0:
return []
detail = result.stderr.strip() or f"exit {result.returncode}"
raise EnumerationError(f"container network list failed: {detail}")
return sorted(
name for name in (line.strip() for line in result.stdout.splitlines())
if name.startswith(_PREFIX)
@@ -51,19 +54,17 @@ def prepare_cleanup() -> MacosContainerBottleCleanupPlan:
def cleanup(plan: MacosContainerBottleCleanupPlan) -> None:
failures = CleanupFailures()
for name in plan.containers:
info(f"container delete --force {name}")
subprocess.run(
failures.run(
["container", "delete", "--force", name],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
f"deleting container {name}",
)
for name in plan.networks:
info(f"container network delete {name}")
subprocess.run(
failures.run(
["container", "network", "delete", name],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
check=False,
f"deleting network {name}",
)
failures.raise_if_any()
+12 -11
View File
@@ -5,7 +5,7 @@ from __future__ import annotations
import subprocess
from ...bottle_state import read_metadata
from .. import ActiveAgent
from .. import ActiveAgent, EnumerationError
from .infra import INFRA_NAME, ORCHESTRATOR_NAME
# The name every agent container carries: `bot-bottle-<slug>`. Exported
@@ -20,17 +20,18 @@ CONTAINER_NAME_PREFIX = "bot-bottle-"
_INFRA_NAMES = frozenset({INFRA_NAME, ORCHESTRATOR_NAME})
class EnumerationError(RuntimeError):
"""container list failed; the resulting live set is not authoritative."""
def enumerate_active() -> list[ActiveAgent]:
result = subprocess.run(
["container", "list", "--quiet"],
capture_output=True,
text=True,
check=False,
)
try:
result = subprocess.run(
["container", "list", "--quiet"],
capture_output=True,
text=True,
check=False,
)
except FileNotFoundError as exc:
raise EnumerationError(
"container list failed: container CLI not found"
) from exc
if result.returncode != 0:
raise EnumerationError(
f"container list failed: "
+2 -2
View File
@@ -5,7 +5,7 @@ Like Docker, this backend needs no privileged network-pool provisioning
running. `setup()` points at the install/`container system start` steps;
`status()` reports readiness. Reached via
`MacosContainerBottleBackend.setup` / `.status`, dispatched from the
generic `./cli.py backend {setup,status}`.
generic `bot-bottle backend {setup,status}`.
"""
from __future__ import annotations
@@ -75,5 +75,5 @@ def status() -> int:
if not _service_running():
ok = False
if not ok:
sys.stderr.write("\nRun: ./cli.py backend setup --backend=macos-container\n")
sys.stderr.write("\nRun: bot-bottle backend setup --backend=macos-container\n")
return 0 if ok else 1
+9 -2
View File
@@ -689,6 +689,13 @@ def pinned_local_image_ref(ref: str) -> str:
nested-containers layer. A content-derived tag prevents another concurrent
build from moving the provider's ordinary ``:latest`` tag between those
two builds.
Unlike Docker, `container image tag` only accepts ``image-name[:tag]`` as
its source and rejects a bare image ID ("cannot specify 64 byte hex string
as reference"), so the mutable ``ref`` is what gets tagged here. The
post-tag inspection below is what keeps that fail-closed: if ``ref`` moved
between the two commands, the new tag will not resolve to the ID this call
derived its name from.
"""
image = image_id(ref)
digest = image.removeprefix("sha256:")
@@ -701,14 +708,14 @@ def pinned_local_image_ref(ref: str) -> str:
repository = repository[:last_colon]
pinned_ref = f"{repository}:sha256-{digest}"
result = subprocess.run(
[_CONTAINER, "image", "tag", image, pinned_ref],
[_CONTAINER, "image", "tag", ref, pinned_ref],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
die(
f"could not tag exact local image {image}: "
f"could not tag exact local image {image} via {ref!r}: "
f"{(result.stderr or result.stdout or '').strip() or '<no detail>'}"
)
if image_id(pinned_ref) != image:
+13 -2
View File
@@ -63,12 +63,23 @@ def provision_git_gate(
transport.exec(["chmod", "+x", "/etc/git-gate/access-hook"])
creds = _creds_dir(bottle_id)
transport.exec(["mkdir", "-p", creds])
transport.exec(["chmod", "700", creds])
credential_paths: list[str] = []
for u in plan.upstreams:
if u.identity_file:
transport.cp_into(u.identity_file, f"{creds}/{u.name}-key")
key_path = f"{creds}/{u.name}-key"
transport.cp_into(u.identity_file, key_path)
credential_paths.append(key_path)
known_hosts = str(u.known_hosts_file)
if known_hosts and known_hosts != ".":
transport.cp_into(known_hosts, f"{creds}/{u.name}-known_hosts")
known_hosts_path = f"{creds}/{u.name}-known_hosts"
transport.cp_into(known_hosts, known_hosts_path)
credential_paths.append(known_hosts_path)
# Copy-mode behavior differs across Docker, Apple Container, and SSH.
# Apply the security contract inside the gateway so every backend produces
# the same private credential namespace.
if credential_paths:
transport.exec(["chmod", "600", *credential_paths])
# Init the bare repos + per-repo credential config for this namespace.
script = git_gate_render_provision(bottle_id, plan.upstreams)
transport.exec(["sh", "-c", script])
+1 -1
View File
@@ -86,7 +86,7 @@ def _print_vm_install_instructions() -> None:
info("Then start the service: container system start")
else:
info("Install Firecracker: https://github.com/firecracker-microvm/firecracker/releases")
info("Configure the host: ./cli.py backend setup")
info("Configure the host: bot-bottle backend setup")
def _auto_select_backend(prompt: bool = True) -> str:
+3 -3
View File
@@ -1,9 +1,9 @@
"""`backend` CLI command — generic host setup/status across backends.
`./cli.py backend setup [--backend=NAME]` provisions (or points at how
`bot-bottle backend setup [--backend=NAME]` provisions (or points at how
to provision) the chosen backend's one-time host prerequisites.
`./cli.py backend status [--backend=NAME]` reports readiness.
`./cli.py backend teardown [--backend=NAME]` undoes setup (uninstall).
`bot-bottle backend status [--backend=NAME]` reports readiness.
`bot-bottle backend teardown [--backend=NAME]` undoes setup (uninstall).
All dispatch to the backend's `setup()` / `status()` / `teardown()`
classmethods, so there are no backend-specific commands swapping
+15 -4
View File
@@ -1,7 +1,7 @@
"""cleanup: stop and remove all orphaned bot-bottle resources.
Walks every registered backend (docker, firecracker, macos-container)
so a single `./cli.py cleanup` reaps every backend's leftovers — a
so a single `bot-bottle cleanup` reaps every backend's leftovers — a
firecracker bottle's VM processes and run dirs won't survive a
docker-only cleanup pass (issue addressed alongside #77).
@@ -22,6 +22,7 @@ from __future__ import annotations
import sys
from ...backend import get_bottle_backend, has_backend, known_backend_names
from ...backend.cleanup_control import CleanupError
from ...log import info
from ...util import read_tty_line
@@ -52,10 +53,20 @@ def cmd_cleanup(_argv: list[str]) -> int:
info("cleanup: skipped")
return 0
for name, backend, plan in prepared:
if plan.empty:
# Confirmation authorizes a fresh authoritative snapshot, not blind use of
# identities that may have changed while the operator reviewed the preview.
failures: list[str] = []
for name, backend, displayed in prepared:
current = backend.prepare_cleanup()
approved = displayed.intersect(current)
if approved.empty:
continue
backend.cleanup(plan)
try:
backend.cleanup(approved)
except CleanupError as exc:
failures.append(f"{name}: {exc}")
if failures:
raise CleanupError("cleanup incomplete: " + "; ".join(failures))
info("cleanup: done")
return 0
+2 -2
View File
@@ -4,7 +4,7 @@ Docker bottles are committed to a local Docker image. Macos-container
bottles are exported and rebuilt as a local Apple Container image.
Firecracker bottles stream the guest rootfs out over SSH and rebuild a
local Docker image. The resulting reference is stored in per-bottle
state so the next `./cli.py resume <slug>` boots from the snapshot
state so the next `bot-bottle resume <slug>` boots from the snapshot
instead of rebuilding from the Dockerfile.
"""
@@ -37,7 +37,7 @@ def cmd_commit(argv: list[str]) -> int:
if slug is None:
active = enumerate_active_agents()
if not active:
die("no active bottles; start one with `./cli.py start`")
die("no active bottles; start one with `bot-bottle start`")
choices = [a.slug for a in active]
slug = tui.filter_select(choices, title="Select bottle to commit")
if slug is None:
+1 -1
View File
@@ -8,7 +8,7 @@ override and transcript snapshot under the same state dir.
Use case: an interrupted or preserved bottle needs to be relaunched;
the operator runs
./cli.py resume <identity>
bot-bottle resume <identity>
to bring up the replacement from the recorded state.
"""
+6 -6
View File
@@ -203,19 +203,19 @@ def _start_headless(
if not os.isatty(stdin_fd):
die(
"--headless requires a PTY on stdin; run via:\n"
" script -q /dev/null ./cli.py start ..."
" script -q /dev/null bot-bottle start ..."
)
agent_name = args.name
if not agent_name:
die("--headless requires an agent name: ./cli.py start <agent> --headless")
die("--headless requires an agent name: bot-bottle start <agent> --headless")
manifest.require_agent(agent_name) # raises ManifestError if unknown
prompt = args.prompt
if not prompt:
die(
"--headless requires --prompt: "
"./cli.py start <agent> --headless --prompt 'Do the thing'"
"bot-bottle start <agent> --headless --prompt 'Do the thing'"
)
if args.bottle:
@@ -319,9 +319,9 @@ def attach_agent(
`resume=True` adds `--continue` so claude picks up its most
recent session non-interactively (no session-picker prompt).
First-attach paths (`./cli.py start`) leave it False.
First-attach paths (`bot-bottle start`) leave it False.
Used as the inner step of `./cli.py start`."""
Used as the inner step of `bot-bottle start`."""
runtime = runtime_for(agent_provider_template)
info(
f"attaching interactive {agent_provider_template} session "
@@ -354,7 +354,7 @@ def settle_state(identity: str) -> None:
if not identity:
return
if is_preserved(identity):
info(f"to resume this bottle: ./cli.py resume {identity}")
info(f"to resume this bottle: bot-bottle resume {identity}")
return
cleanup_state(identity)
+1 -1
View File
@@ -110,7 +110,7 @@ def discover_pending() -> list[QueuedProposal]:
def _approval_status(qp: QueuedProposal, verb: str) -> str:
"""Status-line text after a successful approval."""
base = f"{verb} {qp.proposal.tool} for [{qp.label}]"
return f"{base}; resume: ./cli.py resume {qp.label}"
return f"{base}; resume: bot-bottle resume {qp.label}"
def _detail_lines(
+12 -4
View File
@@ -136,10 +136,18 @@ def _pump(name: str, stream: IO[bytes]) -> None:
"""Read lines from `stream`, prefix with `[name]`, write to
stdout. Runs in its own thread per child; daemon=True so a
blocked read doesn't keep the process alive after main exits."""
for raw in iter(stream.readline, b""):
line = raw.decode("utf-8", errors="replace").rstrip("\n")
sys.stdout.write(f"[{name}] {line}\n")
sys.stdout.flush()
try:
for raw in iter(stream.readline, b""):
line = raw.decode("utf-8", errors="replace").rstrip("\n")
sys.stdout.write(f"[{name}] {line}\n")
sys.stdout.flush()
except (OSError, ValueError) as exc:
# The manager closes a dead child's pipe after wait() and before a
# restart. A pump can be between readline calls at that exact moment;
# closed-stream errors are normal completion, not uncaught thread
# failures. Preserve genuinely unexpected I/O diagnostics.
if not stream.closed:
_log(f"{name} output pump stopped: {type(exc).__name__}: {exc}")
def _spawn(spec: _DaemonSpec) -> subprocess.Popen[bytes]:
+137
View File
@@ -0,0 +1,137 @@
"""Shared resource boundaries for gateway stdlib HTTP services."""
from __future__ import annotations
import http.server
import io
import socket
import threading
import time
from dataclasses import dataclass
from typing import Any, Protocol
class Readable(Protocol):
def read(self, size: int = -1, /) -> bytes: ...
class Writable(Protocol):
def write(self, data: bytes, /) -> object: ...
@dataclass(frozen=True)
class BodyReadError(Exception):
status: int
message: str
def read_declared_body(
stream: Readable,
connection: socket.socket,
raw_length: str | None,
*,
maximum: int,
timeout_seconds: float,
require_length: bool,
) -> bytes:
"""Validate and read exactly one declared body under a read deadline."""
output = io.BytesIO()
copy_declared_body(
stream, output, connection, raw_length, maximum=maximum,
timeout_seconds=timeout_seconds, require_length=require_length,
)
return output.getvalue()
def copy_declared_body(
stream: Readable,
output: Writable,
connection: socket.socket,
raw_length: str | None,
*,
maximum: int,
timeout_seconds: float,
require_length: bool,
) -> int:
"""Copy one declared body to a sink without retaining it in memory."""
if raw_length is None:
if require_length:
raise BodyReadError(411, "Content-Length required")
raw_length = "0"
try:
length = int(raw_length)
except ValueError as exc:
raise BodyReadError(400, "invalid Content-Length") from exc
if length < 0:
raise BodyReadError(400, "invalid Content-Length")
if length > maximum:
raise BodyReadError(413, "request body too large")
previous_timeout = connection.gettimeout()
deadline = time.monotonic() + timeout_seconds
remaining = length
try:
while remaining:
timeout = deadline - time.monotonic()
if timeout <= 0:
raise BodyReadError(408, "request body read timed out")
connection.settimeout(timeout)
chunk = stream.read(min(remaining, 64 * 1024))
if not chunk:
raise BodyReadError(400, "incomplete request body")
output.write(chunk)
remaining -= len(chunk)
except TimeoutError as exc:
raise BodyReadError(408, "request body read timed out") from exc
finally:
connection.settimeout(previous_timeout)
return length
class BoundedThreadingHTTPServer(http.server.ThreadingHTTPServer):
"""ThreadingHTTPServer with a hard cap on in-flight request threads."""
daemon_threads = True
def __init__( # pylint: disable=consider-using-with
self, *args, max_workers: int = 32, **kwargs, # type: ignore[no-untyped-def]
):
if max_workers < 1:
raise ValueError("max_workers must be positive")
self._request_slots = threading.BoundedSemaphore(max_workers)
super().__init__(*args, **kwargs)
def process_request(
self, request: Any, client_address: Any,
) -> None:
if not self._request_slots.acquire( # pylint: disable=consider-using-with
blocking=False,
):
try:
request.sendall(
b"HTTP/1.1 503 Service Unavailable\r\n"
b"Content-Length: 0\r\nConnection: close\r\n\r\n"
)
finally:
self.shutdown_request(request)
return
try:
super().process_request(request, client_address)
except BaseException:
self._request_slots.release()
raise
def process_request_thread(
self, request: Any, client_address: Any,
) -> None:
try:
super().process_request_thread(request, client_address)
finally:
self._request_slots.release()
__all__ = [
"BodyReadError",
"BoundedThreadingHTTPServer",
"copy_declared_body",
"read_declared_body",
]
+62 -105
View File
@@ -16,7 +16,7 @@ import typing
from mitmproxy import http # type: ignore[import-not-found] # pylint: disable=import-error
from bot_bottle.constants import IDENTITY_HEADER
from bot_bottle.gateway.egress.dlp_detectors import redact_tokens, strip_crlf
from bot_bottle.gateway.egress.dlp_detectors import redact_tokens
from bot_bottle.gateway.egress.dlp_config import (
DEFAULT_OUTBOUND_ON_MATCH,
ON_MATCH_BLOCK,
@@ -25,19 +25,19 @@ from bot_bottle.gateway.egress.dlp_config import (
from bot_bottle.gateway.egress.context import resolve_client_context
from bot_bottle.gateway.egress.dlp import (
build_inbound_scan_text,
build_outbound_scan_text,
build_token_allow_payload,
outbound_scan_headers,
scan_inbound,
scan_outbound,
)
from bot_bottle.gateway.egress.outbound_pipeline import redact_request, scan_request
from bot_bottle.gateway.egress.matching import (
decide,
decide_git_fetch,
is_git_fetch_request,
is_git_push_request,
match_route,
)
from bot_bottle.gateway.egress.request_pipeline import (
evaluate_route_policy,
git_block_reason,
)
from bot_bottle.gateway.egress.schema import route_to_yaml_dict
from bot_bottle.gateway.egress.types import (
LOG_BLOCKS,
@@ -389,19 +389,9 @@ class EgressAddon:
self._passthrough_conns.discard(conn_id)
async def request(self, flow: http.HTTPFlow) -> None:
config, slug, env = self._request_context(flow)
request_path, _, query = flow.request.path.partition("?")
# Reuse the context stashed by http_connect for HTTPS flows (one
# orchestrator round-trip per connection). Plain-HTTP flows have no
# prior CONNECT stash, so resolve now and stash for response/websocket.
meta = getattr(flow, "metadata", None)
if isinstance(meta, dict) and _FLOW_CTX_KEY in meta:
config, slug, env = meta[_FLOW_CTX_KEY]
self._request_token(flow) # strip identity headers; token already resolved
else:
config, slug, env = self._resolve_flow(flow)
self._stash_flow_ctx(flow, config, slug, env)
# Introspection ("_egress.local/allowlist") reports the calling bottle's
# own resolved routes — served after resolution so it reflects this
# bottle's policy, not a stale global.
@@ -422,56 +412,66 @@ class EgressAddon:
# the path/query the git checks below rely on.
request_path, _, query = flow.request.path.partition("?")
if is_git_push_request(request_path, query):
self._block(
flow,
"egress: git push over HTTPS is not supported; "
"use the bottle.git SSH path (gitleaks-scanned by "
"git-gate's pre-receive hook).",
ctx=self._req_ctx(flow),
)
if not self._allow_git_request(flow, config, request_path, query):
return
if is_git_fetch_request(request_path, query):
git_decision = decide_git_fetch(
config.routes, flow.request.pretty_host,
)
if git_decision.action == "block":
self._block(
flow,
git_decision.reason,
ctx=self._req_ctx(flow),
)
return
self._apply_route_policy(flow, config, route, request_path, env)
def _request_context(
self, flow: http.HTTPFlow,
) -> tuple[Config, str, "typing.Mapping[str, str]"]:
"""Resolve one bottle context, reusing the HTTPS CONNECT snapshot."""
meta = getattr(flow, "metadata", None)
if isinstance(meta, dict) and _FLOW_CTX_KEY in meta:
config, slug, env = meta[_FLOW_CTX_KEY]
self._request_token(flow)
return config, slug, env
config, slug, env = self._resolve_flow(flow)
self._stash_flow_ctx(flow, config, slug, env)
return config, slug, env
def _allow_git_request(
self, flow: http.HTTPFlow, config: Config,
request_path: str, query: str,
) -> bool:
"""Apply the HTTPS Git push/fetch boundary before general routing."""
reason = git_block_reason(
config.routes, flow.request.pretty_host, request_path, query,
)
if not reason:
return True
self._block(flow, reason, ctx=self._req_ctx(flow))
return False
def _apply_route_policy(
self, flow: http.HTTPFlow, config: Config, route: Route | None,
request_path: str, env: "typing.Mapping[str, str]",
) -> None:
"""Strip agent auth, evaluate the route, then inject gateway auth."""
# Strip agent-set Authorization after DLP scan so smuggled tokens
# are caught above; the route may inject gateway-owned auth below.
# Routes with preserve_auth=True pass the header through as-is so the
# agent's own credentials (e.g. registry bearer tokens) reach the upstream.
if route is None or not route.preserve_auth:
result = evaluate_route_policy(
config,
route,
host=flow.request.pretty_host,
request_path=request_path,
method=flow.request.method,
headers=dict(flow.request.headers),
env=env,
)
if result.strip_authorization:
flow.request.headers.pop("authorization", None)
# Build headers mapping for match evaluation
req_headers = {k.lower(): v for k, v in flow.request.headers.items()}
decision = decide(
config.routes,
flow.request.pretty_host,
request_path,
env,
request_method=flow.request.method,
request_headers=req_headers,
deny_reason=config.deny_reason,
)
if decision.action == "block":
self._block(flow, decision.reason, ctx=self._req_ctx(flow))
if result.block_reason:
self._block(flow, result.block_reason, ctx=self._req_ctx(flow))
return
if decision.inject_authorization is not None:
flow.request.headers["authorization"] = decision.inject_authorization
if result.inject_authorization is not None:
flow.request.headers["authorization"] = result.inject_authorization
if config.log >= LOG_FULL:
if result.log_request:
self._log_request(flow, env)
def _block_dlp(self, flow: http.HTTPFlow, result: ScanResult) -> None:
@@ -495,20 +495,12 @@ class EgressAddon:
Loops so the supervise policy can re-scan after each approval a
second, un-approved token in the same request is still caught."""
while True:
request_path, _, query = flow.request.path.partition("?")
body = flow.request.get_text(strict=False) or ""
headers = outbound_scan_headers(route, dict(flow.request.headers))
scan_text = build_outbound_scan_text(
flow.request.pretty_host, request_path, query, headers, body,
)
# CRLF is scanned only over the request line + headers, never the
# body (see scan_outbound) — a body is not an injection vector.
crlf_text = build_outbound_scan_text(
flow.request.pretty_host, request_path, query, headers, "",
)
result = scan_outbound(
route, scan_text, env,
safe_tokens=self._safe_tokens_for(slug), crlf_text=crlf_text,
request_path, _, _ = flow.request.path.partition("?")
result = scan_request(
flow.request,
route,
env,
safe_tokens=self._safe_tokens_for(slug),
)
if result is None or result.severity != "block":
return True
@@ -518,7 +510,7 @@ class EgressAddon:
# redact scrubs every detection (tokens and structural CRLF) and
# forwards; it fails closed only if a match survives the scrub.
if policy == ON_MATCH_REDACT:
if self._redact_outbound(flow, route, env):
if redact_request(flow.request, route, env):
if self._flow_log(flow) >= LOG_BLOCKS:
sys.stderr.write(json.dumps({
"event": "egress_redacted",
@@ -551,41 +543,6 @@ class EgressAddon:
return False # _supervise_token_block wrote the 403 response
# loop: the approved value is now in safe_tokens; re-scan.
def _redact_outbound(
self, flow: http.HTTPFlow, route: Route, env: "typing.Mapping[str, str]",
) -> bool:
"""Scrub detected tokens (and CRLF injection sequences) from the mutable
request surfaces (body, headers, path/query) and re-scan. `env` is the
per-bottle env overlay. Returns True if the request is now clean; False
if a block-severity match remains on a surface redaction cannot rewrite
(the hostname) so the caller fails closed."""
body = flow.request.get_text(strict=False)
if body:
redacted_body = redact_tokens(body, env=env)
if redacted_body != body:
flow.request.text = redacted_body
for name, value in list(flow.request.headers.items()):
if name.lower() == "host":
continue # routing-critical; never a legitimate token
redacted = strip_crlf(redact_tokens(value, env=env))
if redacted != value:
flow.request.headers[name] = redacted
redacted_path = strip_crlf(redact_tokens(flow.request.path, env=env))
if redacted_path != flow.request.path:
flow.request.path = redacted_path
request_path, _, query = flow.request.path.partition("?")
new_body = flow.request.get_text(strict=False) or ""
headers = outbound_scan_headers(route, dict(flow.request.headers))
scan_text = build_outbound_scan_text(
flow.request.pretty_host, request_path, query, headers, new_body,
)
crlf_text = build_outbound_scan_text(
flow.request.pretty_host, request_path, query, headers, "",
)
result = scan_outbound(route, scan_text, env, crlf_text=crlf_text)
return result is None or result.severity != "block"
async def _supervise_token_block(
self,
flow: http.HTTPFlow,
@@ -0,0 +1,83 @@
"""Outbound DLP request scanning and redaction for the egress pipeline."""
from __future__ import annotations
from typing import ItemsView, Mapping, Protocol
from .dlp import (
build_outbound_scan_text,
outbound_scan_headers,
scan_outbound,
)
from .dlp_detectors import redact_tokens, strip_crlf
from .types import Route, ScanResult
class MutableHeaders(Protocol):
def items(self) -> ItemsView[str, str]: ...
def __getitem__(self, name: str, /) -> str: ...
def __setitem__(self, name: str, value: str, /) -> None: ...
class MutableRequest(Protocol):
pretty_host: str
path: str
headers: MutableHeaders
text: str
def get_text(self, strict: bool = False) -> str | None: ...
def scan_request(
request: MutableRequest,
route: Route,
env: Mapping[str, str],
*,
safe_tokens: set[str] | None = None,
) -> ScanResult | None:
"""Scan all mutable outbound request surfaces in their canonical order."""
request_path, _, query = request.path.partition("?")
headers = outbound_scan_headers(route, dict(request.headers.items()))
body = request.get_text(strict=False) or ""
scan_text = build_outbound_scan_text(
request.pretty_host, request_path, query, headers, body,
)
# Bodies cannot alter HTTP framing, so CRLF detection is deliberately
# restricted to the request line and headers.
crlf_text = build_outbound_scan_text(
request.pretty_host, request_path, query, headers, "",
)
return scan_outbound(
route,
scan_text,
env,
safe_tokens=safe_tokens,
crlf_text=crlf_text,
)
def redact_request(
request: MutableRequest,
route: Route,
env: Mapping[str, str],
) -> bool:
"""Redact mutable request surfaces and return whether the result is clean."""
body = request.get_text(strict=False)
if body:
redacted_body = redact_tokens(body, env=env)
if redacted_body != body:
request.text = redacted_body
for name, value in list(request.headers.items()):
if name.lower() == "host":
continue
redacted = strip_crlf(redact_tokens(value, env=env))
if redacted != value:
request.headers[name] = redacted
redacted_path = strip_crlf(redact_tokens(request.path, env=env))
if redacted_path != request.path:
request.path = redacted_path
result = scan_request(request, route, env)
return result is None or result.severity != "block"
__all__ = ["MutableRequest", "redact_request", "scan_request"]
@@ -0,0 +1,92 @@
"""Framework-neutral request policy stages for the egress adapter.
The mitmproxy addon owns flow mutation and response construction. This module
owns the ordered Git and route-policy decisions so those rules remain directly
testable without a live proxy flow.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Mapping, Sequence
from .matching import (
decide,
decide_git_fetch,
is_git_fetch_request,
is_git_push_request,
)
from .types import LOG_FULL, Config, Route
GIT_PUSH_BLOCK_REASON = (
"egress: git push over HTTPS is not supported; "
"use the bottle.git SSH path (gitleaks-scanned by "
"git-gate's pre-receive hook)."
)
@dataclass(frozen=True)
class RoutePolicyResult:
"""The flow mutations and outcome produced by general route policy."""
block_reason: str = ""
strip_authorization: bool = False
inject_authorization: str | None = None
log_request: bool = False
def git_block_reason(
routes: Sequence[Route],
host: str,
request_path: str,
query: str,
) -> str:
"""Return the HTTPS Git policy denial, or ``""`` when allowed."""
if is_git_push_request(request_path, query):
return GIT_PUSH_BLOCK_REASON
if not is_git_fetch_request(request_path, query):
return ""
decision = decide_git_fetch(routes, host)
return decision.reason if decision.action == "block" else ""
def evaluate_route_policy(
config: Config,
route: Route | None,
*,
host: str,
request_path: str,
method: str,
headers: Mapping[str, str],
env: Mapping[str, str],
) -> RoutePolicyResult:
"""Evaluate authorization stripping, matching, injection, and logging."""
strip_authorization = route is None or not route.preserve_auth
effective_headers = {
name.lower(): value
for name, value in headers.items()
if not (strip_authorization and name.lower() == "authorization")
}
decision = decide(
config.routes,
host,
request_path,
env,
request_method=method,
request_headers=effective_headers,
deny_reason=config.deny_reason,
)
return RoutePolicyResult(
block_reason=decision.reason if decision.action == "block" else "",
strip_authorization=strip_authorization,
inject_authorization=decision.inject_authorization,
log_request=config.log >= LOG_FULL,
)
__all__ = [
"GIT_PUSH_BLOCK_REASON",
"RoutePolicyResult",
"evaluate_route_policy",
"git_block_reason",
]
+48 -22
View File
@@ -21,12 +21,19 @@ from __future__ import annotations
import os
import subprocess
import sys
import tempfile
import threading
import typing
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from http.server import BaseHTTPRequestHandler
from pathlib import Path
from urllib.parse import urlsplit
from bot_bottle.constants import GIT_GATE_TIMEOUT_SECS, IDENTITY_HEADER
from bot_bottle.gateway.bounded_http import (
BodyReadError,
BoundedThreadingHTTPServer,
copy_declared_body,
)
from bot_bottle.gateway.policy_resolver import PolicyResolveError, PolicyResolver
@@ -77,6 +84,10 @@ def resolve_sandbox_root(
# Bound memory use while still allowing ordinary git push packfiles.
MAX_BODY_BYTES = 100 * 1024 * 1024
REQUEST_BODY_TIMEOUT_SECONDS = 30.0
MAX_REQUEST_WORKERS = 16
MAX_BODY_WORKERS = 2
_BODY_WORK_SLOTS = threading.BoundedSemaphore(MAX_BODY_WORKERS)
class GitHttpHandler(BaseHTTPRequestHandler):
@@ -184,27 +195,40 @@ class GitHttpHandler(BaseHTTPRequestHandler):
value = self.headers.get(header)
if value:
env[variable] = value
raw_length = self.headers.get("content-length", "0") or "0"
if not _BODY_WORK_SLOTS.acquire(blocking=False):
self.send_error(503, "git request capacity exhausted")
return
try:
length = int(raw_length)
except ValueError:
self.send_error(400, "Bad Content-Length")
return
if length < 0:
self.send_error(400, "Negative Content-Length")
return
if length > MAX_BODY_BYTES:
self.send_error(413, "Request body too large")
return
body = self.rfile.read(length) if length else b""
proc = subprocess.run(
["git", "http-backend"],
input=body,
env=env,
capture_output=True,
check=False,
timeout=GIT_GATE_TIMEOUT_SECS,
)
with tempfile.TemporaryFile() as body:
try:
copy_declared_body(
self.rfile,
body,
self.connection,
self.headers.get("content-length"),
maximum=MAX_BODY_BYTES,
timeout_seconds=REQUEST_BODY_TIMEOUT_SECONDS,
require_length=False,
)
except BodyReadError as exc:
self.send_error(exc.status, exc.message)
return
body.seek(0)
try:
proc = subprocess.run(
["git", "http-backend"],
stdin=body,
env=env,
capture_output=True,
check=False,
timeout=GIT_GATE_TIMEOUT_SECS,
)
except (OSError, subprocess.SubprocessError) as exc:
self.log_message("git http-backend unavailable: %s", exc)
self.send_error(503, "git backend unavailable")
return
finally:
_BODY_WORK_SLOTS.release()
self._write_cgi_response(proc.stdout)
def _repo_dir(self, sandbox_root: Path, path: str) -> Path | None:
@@ -273,7 +297,9 @@ def main() -> int:
"(no single-tenant flat-root fallback)\n"
)
return 1
server = ThreadingHTTPServer(("0.0.0.0", port), GitHttpHandler)
server = BoundedThreadingHTTPServer(
("0.0.0.0", port), GitHttpHandler, max_workers=MAX_REQUEST_WORKERS,
)
# Resolve each request's sandbox namespace by source IP against the
# orchestrator control plane.
server.policy_resolver = PolicyResolver(orch_url) # type: ignore[attr-defined]
+1 -1
View File
@@ -385,7 +385,7 @@ PY
;;
esac
echo "git-gate: queued # gitleaks:allow supervisor approval $proposal_id" >&2
echo "git-gate: approve with './cli.py supervise' to continue this push" >&2
echo "git-gate: approve with 'bot-bottle supervise' to continue this push" >&2
waited=0
while [ "$waited" -lt "$timeout" ]; do
status=$(PYTHONPATH="/app${PYTHONPATH:+:$PYTHONPATH}" python3 - "$proposal_id" <<'PY'
@@ -0,0 +1,92 @@
"""Framework-neutral MCP method and tool dispatch."""
from __future__ import annotations
import json
from dataclasses import dataclass
from typing import Callable, Protocol
from bot_bottle.gateway.egress.schema import load_config, route_to_yaml_dict
from bot_bottle.gateway.policy_resolver import PolicyResolveError, PolicyResolver
from bot_bottle.supervisor import types as _sv
class Request(Protocol):
@property
def method(self) -> str: ...
@property
def params(self) -> dict[str, object]: ...
class MethodNotFoundError(Exception):
"""Raised when a JSON-RPC method has no MCP handler."""
class RouteResolutionError(Exception):
"""The caller's live route table could not be resolved authoritatively."""
Handler = Callable[[dict[str, object]], object]
@dataclass(frozen=True)
class Handlers:
initialize: Handler
tools_list: Handler
list_routes: Handler
check_proposal: Handler
propose: Handler
def dispatch(request: Request, handlers: Handlers) -> object:
"""Route one parsed request without depending on the HTTP server."""
if request.method == "initialize":
return handlers.initialize(request.params)
if request.method == "notifications/initialized":
return None
if request.method == "tools/list":
return handlers.tools_list(request.params)
if request.method != "tools/call":
raise MethodNotFoundError(request.method)
tool = request.params.get("name")
if tool == _sv.TOOL_LIST_EGRESS_ROUTES:
return handlers.list_routes(request.params)
if tool == _sv.TOOL_CHECK_PROPOSAL:
return handlers.check_proposal(request.params)
return handlers.propose(request.params)
def resolved_routes_payload(
resolver: PolicyResolver,
source_ip: str,
identity_token: str,
) -> dict[str, object]:
"""Render an authoritatively resolved route table for the calling bottle."""
try:
policy, bottle_id, _tokens = resolver.resolve_policy_and_bottle_id(
source_ip, identity_token,
)
except PolicyResolveError as exc:
raise RouteResolutionError("orchestrator unavailable") from exc
if not bottle_id:
raise RouteResolutionError("request source is not attributed to a bottle")
try:
config = load_config(policy or "")
except ValueError as exc:
raise RouteResolutionError("resolved policy is invalid") from exc
body = json.dumps(
{"routes": [route_to_yaml_dict(route) for route in config.routes]},
indent=2,
)
return {"content": [{"type": "text", "text": body}], "isError": False}
__all__ = [
"Handlers",
"MethodNotFoundError",
"RouteResolutionError",
"dispatch",
"resolved_routes_payload",
]
+69 -63
View File
@@ -51,17 +51,27 @@ from __future__ import annotations
import http.server
import json
import os
import socketserver
import sys
import time
import typing
from dataclasses import dataclass
from bot_bottle.constants import IDENTITY_HEADER
from bot_bottle.gateway.egress.context import resolve_client_context
from bot_bottle.gateway.egress.schema import load_config, route_to_yaml_dict
from bot_bottle.gateway.bounded_http import (
BodyReadError,
BoundedThreadingHTTPServer,
read_declared_body,
)
from bot_bottle.gateway.egress.schema import load_config
from bot_bottle.gateway.egress.types import LOG_OFF
from bot_bottle.gateway.policy_resolver import PolicyResolveError, PolicyResolver
from bot_bottle.gateway.supervisor.mcp_dispatch import (
Handlers as DispatchHandlers,
MethodNotFoundError,
RouteResolutionError,
dispatch,
resolved_routes_payload,
)
from bot_bottle.supervisor import types as _sv
@@ -565,6 +575,8 @@ def format_unknown_proposal_text(proposal_id: str) -> str:
# Max request body the server accepts. 1 MB is well above any realistic
# routes.yaml proposal.
MAX_BODY_BYTES = 1 * 1024 * 1024
REQUEST_BODY_TIMEOUT_SECONDS = 10.0
MAX_REQUEST_WORKERS = 32
class MCPHandler(http.server.BaseHTTPRequestHandler):
@@ -587,19 +599,18 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
self._write_text(405, "use POST for MCP requests\n")
def do_POST(self) -> None:
length_header = self.headers.get("Content-Length")
if length_header is None:
self._write_text(411, "Content-Length required\n")
return
try:
length = int(length_header)
except ValueError:
self._write_text(400, "invalid Content-Length\n")
body = read_declared_body(
self.rfile,
self.connection,
self.headers.get("Content-Length"),
maximum=MAX_BODY_BYTES,
timeout_seconds=REQUEST_BODY_TIMEOUT_SECONDS,
require_length=True,
)
except BodyReadError as exc:
self._write_text(exc.status, exc.message + "\n")
return
if length < 0 or length > MAX_BODY_BYTES:
self._write_text(413, "request body too large\n")
return
body = self.rfile.read(length)
try:
req = parse_jsonrpc(body)
@@ -611,6 +622,11 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
try:
result = self._dispatch(req, config)
except MethodNotFoundError as e:
self._write_jsonrpc(
jsonrpc_error(req.id, ERR_METHOD_NOT_FOUND, f"method not found: {e}"),
)
return
except _RpcClientError as e:
self._write_jsonrpc(jsonrpc_error(req.id, e.code, e.message))
return
@@ -633,41 +649,42 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
self._write_jsonrpc(jsonrpc_result(req.id, result))
def _dispatch(self, req: JsonRpcRequest, config: ServerConfig) -> object:
method = req.method
if method == "initialize":
return handle_initialize(req.params)
if method == "notifications/initialized":
return None # ack-only
if method == "tools/list":
return handle_tools_list(req.params)
if method == "tools/call":
# `list-egress-routes` is read-only introspection. The shared gateway
# has no static route table (routes are resolved per request by
# source IP), so answer it from the calling bottle's resolved policy.
# Otherwise the agent sees an empty allowlist and composes an egress
# proposal that *replaces* the live routes instead of extending them
# — silently dropping base routes like api.anthropic.com on approval.
if req.params.get("name") == _sv.TOOL_LIST_EGRESS_ROUTES:
return self._resolved_routes_payload()
resolver = self._resolver_or_fail()
source_ip = self.client_address[0]
token = self._identity_token()
# `check-proposal` is a non-blocking read of the calling bottle's
# own queue — attributed by (source_ip, identity_token) like a
# proposal, but it never queues or blocks.
if req.params.get("name") == _sv.TOOL_CHECK_PROPOSAL:
return handle_check_proposal(
req.params, resolver=resolver,
source_ip=source_ip, identity_token=token,
)
# The control plane attributes the proposal to the source-IP + token
# resolved bottle, so the one shared queue holds each bottle's
# proposal under its own id — no slug is asserted by this daemon.
return handle_tools_call(
req.params, config, resolver=resolver,
source_ip=source_ip, identity_token=token,
def check(params: dict[str, object]) -> object:
return handle_check_proposal(
params,
resolver=self._resolver_or_fail(),
source_ip=self.client_address[0],
identity_token=self._identity_token(),
)
raise _RpcClientError(ERR_METHOD_NOT_FOUND, f"method not found: {method}")
def propose(params: dict[str, object]) -> object:
return handle_tools_call(
params,
config,
resolver=self._resolver_or_fail(),
source_ip=self.client_address[0],
identity_token=self._identity_token(),
)
def list_routes(_params: dict[str, object]) -> object:
try:
return resolved_routes_payload(
self._resolver_or_fail(),
self.client_address[0],
self._identity_token(),
)
except RouteResolutionError as exc:
raise _RpcInternalError(
f"could not resolve live egress routes: {exc}"
) from exc
return dispatch(req, DispatchHandlers(
initialize=handle_initialize,
tools_list=handle_tools_list,
list_routes=list_routes,
check_proposal=check,
propose=propose,
))
def _identity_token(self) -> str:
"""The agent's per-bottle identity token from the request header (the
@@ -686,20 +703,6 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
raise _RpcInternalError("supervise server has no policy resolver")
return resolver
def _resolved_routes_payload(self) -> dict[str, object]:
"""The calling bottle's live egress routes as the `list-egress-routes`
JSON payload, resolved by (source_ip, identity token). Fail-closed: an
unattributed source or an unreachable orchestrator yields an empty route
list (never another bottle's), courtesy of `resolve_client_context`."""
resolver = self._resolver_or_fail()
conf, _slug, _tokens = resolve_client_context(
resolver, self.client_address[0], self._identity_token(),
)
body = json.dumps(
{"routes": [route_to_yaml_dict(r) for r in conf.routes]}, indent=2,
)
return {"content": [{"type": "text", "text": body}], "isError": False}
def _write_jsonrpc(self, body: bytes) -> None:
self.send_response(200)
self.send_header("Content-Type", "application/json")
@@ -719,7 +722,7 @@ class MCPHandler(http.server.BaseHTTPRequestHandler):
self.wfile.write(encoded)
class MCPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
class MCPServer(BoundedThreadingHTTPServer):
allow_reuse_address = True
daemon_threads = True
config: ServerConfig = ServerConfig()
@@ -728,6 +731,9 @@ class MCPServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
# closed per request (see `_resolver_or_fail`).
policy_resolver: "PolicyResolver | None" = None
def __init__(self, *args, **kwargs): # type: ignore[no-untyped-def]
super().__init__(*args, max_workers=MAX_REQUEST_WORKERS, **kwargs)
# --- Entry point -----------------------------------------------------------
+48
View File
@@ -0,0 +1,48 @@
"""How to tell a user to re-run this CLI.
`bot-bottle ` is the right thing to print for anything the user runs as
themselves it is on their PATH, since that is how they got here.
Under `sudo` it is not. sudo replaces PATH with sudoers' `secure_path`
(`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` on Debian and
Ubuntu, similar elsewhere), which deliberately excludes user-writable
directories. Both supported install paths put the entry point in one of those:
pipx uses `~/.local/bin`, and `install.sh`'s venv fallback symlinks there too.
So `sudo bot-bottle ` fails with "command not found" for exactly the users who
followed the documented install, while working for anyone who happened to
install system-wide which is why it survives review so easily.
Naming the absolute path sidesteps secure_path entirely.
"""
from __future__ import annotations
import os
import shutil
import sys
def self_path() -> str:
"""Absolute path to this CLI's entry point.
Falls back to the bare name when the entry point cannot be resolved (an
unusual invocation such as `python -m`), because a slightly wrong hint is
better than a traceback while reporting an unrelated problem.
"""
argv0 = sys.argv[0] or "bot-bottle"
resolved = shutil.which(argv0) or argv0
if not os.path.isabs(resolved):
if os.path.exists(resolved):
resolved = os.path.abspath(resolved)
else:
return "bot-bottle"
return resolved
def sudo_command(*args: str) -> str:
"""A copy-pasteable `sudo …` invocation of this CLI.
>>> sudo_command("backend", "setup", "--backend=firecracker")
'sudo /home/u/.local/bin/bot-bottle backend setup --backend=firecracker'
"""
return " ".join(["sudo", self_path(), *args])
+3 -3
View File
@@ -44,7 +44,7 @@ if TYPE_CHECKING:
from ..gateway import Gateway, GatewayError
from .lifecycle import Orchestrator
from .service import OrchestratorCore
from .server import OrchestratorServer, dispatch, make_server
from .server import OrchestratorServer, create_app, make_server
# Facade name -> submodule that defines it. Lazy so importing a leaf (or the
@@ -67,8 +67,8 @@ _LAZY: dict[str, str] = {
"GatewayError": "..gateway",
"Orchestrator": ".lifecycle",
"OrchestratorCore": ".service",
"create_app": ".server",
"OrchestratorServer": ".server",
"dispatch": ".server",
"make_server": ".server",
}
@@ -100,7 +100,7 @@ __all__ = [
"GatewayError",
"Orchestrator",
"OrchestratorCore",
"create_app",
"OrchestratorServer",
"dispatch",
"make_server",
]
+13 -9
View File
@@ -1,6 +1,7 @@
"""Run the orchestrator control plane as a plain process (PRD 0070 dev-harness).
python -m bot_bottle.orchestrator [--host H] [--port P] [--db PATH]
BOT_BOTTLE_ORCHESTRATOR_TOKEN=<signing-key> \
python -m bot_bottle.orchestrator [--host H] [--port P] [--db PATH]
The PRD sequences the orchestrator as a plain-process dev-harness first, so
the consolidation core (registry + attribution + HTTP control plane + live
@@ -16,12 +17,13 @@ import secrets
from pathlib import Path
from .. import log
from .store.store_manager import StoreManager
from ..trust_domain import CONTROL_PLANE
from .broker import LaunchBroker, StubBroker
from .server import make_server
from .docker_broker import DockerBroker
from .store.registry_store import RegistryStore, default_db_path
from .server import make_server
from .service import OrchestratorCore
from .store.store_manager import StoreManager
from .store.registry_store import RegistryStore, default_db_path
def main(argv: list[str] | None = None) -> int:
@@ -38,6 +40,11 @@ def main(argv: list[str] | None = None) -> int:
help="launch broker: 'stub' records requests; 'docker' runs containers",
)
args = parser.parse_args(argv)
if not CONTROL_PLANE.key_from_env():
log.die(
f"{CONTROL_PLANE.key_env} is required; refusing to start the "
"orchestrator without caller authentication"
)
registry = RegistryStore(args.db)
registry.migrate()
@@ -55,17 +62,14 @@ def main(argv: list[str] | None = None) -> int:
orchestrator = OrchestratorCore(registry, broker, secret)
server = make_server(orchestrator, host=args.host, port=args.port)
bound_host, bound_port = server.server_address[0], server.server_address[1]
log.info(
"orchestrator control plane listening",
context={"host": bound_host, "port": bound_port, "db": str(registry.db_path)},
context={"host": args.host, "port": args.port, "db": str(registry.db_path)},
)
try:
server.serve_forever()
server.run()
except KeyboardInterrupt:
log.info("orchestrator shutting down")
finally:
server.server_close()
return 0
+353
View File
@@ -0,0 +1,353 @@
"""FastAPI control-plane routes for the orchestrator."""
# pyright: reportUnusedFunction=false
from __future__ import annotations
import asyncio
import math
import sys
from fastapi import FastAPI, HTTPException
from fastapi.exceptions import RequestValidationError
from fastapi.responses import JSONResponse
from pydantic import BaseModel, ConfigDict, StrictStr
from starlette.types import ASGIApp, Message, Receive, Scope, Send
from ..orchestrator_auth import ROLE_CLI, ROLES
from ..supervisor.types import TOOLS
from ..trust_domain import CONTROL_PLANE
from .http_contract import (
MAX_BODY_BYTES,
ORCHESTRATOR_AUTH_HEADER,
REQUEST_BODY_TIMEOUT_SECONDS,
)
from .service import OrchestratorCore
_GATEWAY_ROUTES = frozenset({
("POST", "/resolve"),
("POST", "/supervise/propose"),
("POST", "/supervise/poll"),
})
class _StrictModel(BaseModel):
model_config = ConfigDict(extra="ignore", strict=True)
class LaunchBody(_StrictModel):
source_ip: StrictStr
image_ref: StrictStr = ""
metadata: StrictStr = ""
policy: StrictStr = ""
tokens: dict[StrictStr, StrictStr] = {}
env_var_secret: StrictStr = ""
class PolicyBody(_StrictModel):
policy: StrictStr
class ReprovisionBody(_StrictModel):
env_var_secret: StrictStr
class ReconcileBody(_StrictModel):
live_source_ips: list[StrictStr]
grace_seconds: float | None = None
class IdentityBody(_StrictModel):
source_ip: StrictStr
identity_token: StrictStr = ""
class AttributeBody(_StrictModel):
source_ip: StrictStr
identity_token: StrictStr
class RespondBody(_StrictModel):
proposal_id: StrictStr
bottle_slug: StrictStr
decision: StrictStr
notes: StrictStr = ""
final_file: StrictStr | None = None
class ProposeBody(IdentityBody):
tool: StrictStr
proposed_file: StrictStr
justification: StrictStr
class PollBody(IdentityBody):
proposal_id: StrictStr
class ControlPlaneBoundary:
"""Reject unauthenticated and oversized requests before reading a body."""
def __init__(self, app: ASGIApp, signing_key: str) -> None:
self.app = app
self.signing_key = signing_key
async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
if scope["type"] != "http":
await self.app(scope, receive, send)
return
method = scope["method"]
route = scope["path"].rstrip("/") or "/"
if not (method == "GET" and route == "/health"):
headers = dict(scope["headers"])
presented = headers.get(
ORCHESTRATOR_AUTH_HEADER.encode(), b"",
).decode(errors="ignore")
role = CONTROL_PLANE.verify(presented, self.signing_key)
if role is None:
await self._reject(
scope, send, 401, "control-plane authentication required",
)
return
allowed = ROLES if (method, route) in _GATEWAY_ROUTES else {ROLE_CLI}
if role not in allowed:
await self._reject(scope, send, 403, "insufficient role for this route")
return
scope.setdefault("state", {})["role"] = role
raw_length = dict(scope["headers"]).get(b"content-length")
if raw_length is not None:
try:
length = int(raw_length)
except ValueError:
await self._reject(scope, send, 400, "invalid Content-Length")
return
if length < 0:
await self._reject(scope, send, 400, "invalid Content-Length")
return
if length > MAX_BODY_BYTES:
await self._reject(scope, send, 413, "request body too large")
return
try:
body = await self._read_body(receive)
except _BodyTooLarge:
await self._reject(scope, send, 413, "request body too large")
return
except TimeoutError:
await self._reject(scope, send, 408, "request body read timed out")
return
try:
await self.app(scope, self._replay_body(body), send)
except Exception as exc: # noqa: BLE001 - redact control-plane failures
sys.stderr.write(
f"orchestrator: {method} {route} failed "
f"[error_type={type(exc).__name__}]\n"
)
sys.stderr.flush()
await self._reject(scope, send, 500, "internal error")
@staticmethod
async def _reject(
scope: Scope, send: Send, status: int, error: str,
) -> None:
response = JSONResponse({"error": error}, status_code=status)
await response(scope, ControlPlaneBoundary._empty_receive, send)
@staticmethod
async def _empty_receive() -> Message:
return {"type": "http.disconnect"}
@staticmethod
async def _read_body(receive: Receive) -> bytes:
body = bytearray()
async with asyncio.timeout(REQUEST_BODY_TIMEOUT_SECONDS):
while True:
message = await receive()
if message["type"] != "http.request":
break
body.extend(message.get("body", b""))
if len(body) > MAX_BODY_BYTES:
raise _BodyTooLarge
if not message.get("more_body", False):
break
return bytes(body)
@staticmethod
def _replay_body(body: bytes) -> Receive:
sent = False
async def replay() -> Message:
nonlocal sent
if sent:
return {"type": "http.disconnect"}
sent = True
return {"type": "http.request", "body": body, "more_body": False}
return replay
class _BodyTooLarge(Exception):
"""The streamed request exceeded the control-plane body limit."""
def _required(value: str, name: str) -> str:
if not value:
raise HTTPException(400, f"{name} (string) is required")
return value
def create_app(orch: OrchestratorCore, *, signing_key: str) -> FastAPI:
"""Build the authenticated orchestrator ASGI application."""
key = signing_key.strip()
if not key:
raise ValueError(
"orchestrator control-plane signing key is required; "
"refusing to start without caller authentication"
)
app = FastAPI(
title="bot-bottle orchestrator",
docs_url=None,
redoc_url=None,
openapi_url=None,
)
app.add_middleware(ControlPlaneBoundary, signing_key=key)
@app.exception_handler(RequestValidationError)
async def invalid_request(
_request: object, exc: RequestValidationError,
) -> JSONResponse:
errors = exc.errors()
location = errors[0].get("loc", ()) if errors else ()
field = str(location[1]) if len(location) > 1 else ""
suffix = f": {field}" if field else ""
return JSONResponse(
{"error": f"invalid request body{suffix}"},
status_code=400,
)
@app.get("/health")
def health() -> dict[str, str]:
return {"status": "ok"}
@app.get("/gateway")
def gateway() -> dict[str, object]:
return orch.gateway_status()
@app.get("/bottles")
def bottles() -> dict[str, object]:
return {"bottles": [record.redacted() for record in orch.registry.all()]}
@app.post("/bottles", status_code=201)
def launch(body: LaunchBody) -> dict[str, str]:
rec = orch.launch_bottle(
_required(body.source_ip, "source_ip"),
image_ref=body.image_ref,
metadata=body.metadata,
policy=body.policy,
tokens=dict(body.tokens),
env_var_secret=body.env_var_secret,
)
return {"bottle_id": rec.bottle_id, "identity_token": rec.identity_token}
@app.put("/bottles/{bottle_id}/policy")
def set_policy(bottle_id: str, body: PolicyBody) -> dict[str, object]:
if orch.set_policy(bottle_id, body.policy):
return {"updated": True}
raise HTTPException(404, "no such bottle")
@app.post("/bottles/{bottle_id}/reprovision_gateway")
def reprovision(bottle_id: str, body: ReprovisionBody) -> dict[str, object]:
secret = _required(body.env_var_secret, "env_var_secret")
if orch.reprovision_from_secret(bottle_id, secret):
return {"reprovisioned": True}
raise HTTPException(404, "no stored secrets for this bottle")
@app.delete("/bottles/{bottle_id}")
def teardown(bottle_id: str) -> dict[str, object]:
if orch.teardown_bottle(bottle_id):
return {"torn_down": True}
raise HTTPException(404, "no such bottle")
@app.post("/reconcile")
def reconcile(body: ReconcileBody) -> dict[str, object]:
if any(not ip for ip in body.live_source_ips):
raise HTTPException(400, "live_source_ips must contain non-empty strings")
kwargs: dict[str, float] = {}
if body.grace_seconds is not None:
if not math.isfinite(body.grace_seconds) or body.grace_seconds < 0:
raise HTTPException(
400, "grace_seconds must be a non-negative finite number",
)
kwargs["grace_seconds"] = body.grace_seconds
return {"reaped": orch.reconcile(body.live_source_ips, **kwargs)}
@app.post("/attribute")
def attribute(body: AttributeBody) -> dict[str, str]:
rec = orch.attribute(body.source_ip, body.identity_token)
if rec is None:
raise HTTPException(403, "unattributed")
return {"bottle_id": rec.bottle_id}
@app.get("/supervise/proposals")
def proposals() -> dict[str, object]:
return {"proposals": orch.supervise_pending()}
@app.post("/supervise/respond")
def respond(body: RespondBody) -> dict[str, object]:
ok, error = orch.supervise_respond(
_required(body.proposal_id, "proposal_id"),
bottle_slug=_required(body.bottle_slug, "bottle_slug"),
decision=_required(body.decision, "decision"),
notes=body.notes,
final_file=body.final_file,
)
if not ok:
raise HTTPException(409, error)
return {"responded": True}
@app.post("/supervise/propose", status_code=201)
def propose(body: ProposeBody) -> dict[str, str]:
source_ip = _required(body.source_ip, "source_ip")
if body.tool not in TOOLS:
raise HTTPException(400, f"tool (string) must be one of {TOOLS}")
rec = orch.resolve(source_ip, body.identity_token)
if rec is None:
raise HTTPException(403, "unattributed")
proposal_id = orch.supervise_queue_proposal(
rec.bottle_id,
tool=body.tool,
proposed_file=_required(body.proposed_file, "proposed_file"),
justification=_required(body.justification, "justification"),
)
return {"proposal_id": proposal_id}
@app.post("/supervise/poll")
def poll(body: PollBody) -> dict[str, object]:
rec = orch.resolve(
_required(body.source_ip, "source_ip"), body.identity_token,
)
if rec is None:
raise HTTPException(403, "unattributed")
return orch.supervise_poll_response(
rec.bottle_id, _required(body.proposal_id, "proposal_id"),
)
@app.post("/resolve")
def resolve(body: IdentityBody) -> dict[str, object]:
rec = orch.resolve(
_required(body.source_ip, "source_ip"), body.identity_token,
)
if rec is None:
raise HTTPException(403, "unattributed")
return {
"bottle_id": rec.bottle_id,
"policy": rec.policy,
"tokens": orch.tokens_for(rec.bottle_id),
}
return app
__all__ = [
"ControlPlaneBoundary",
"MAX_BODY_BYTES",
"ORCHESTRATOR_AUTH_HEADER",
"create_app",
]
+1 -1
View File
@@ -21,7 +21,7 @@ from dataclasses import dataclass
from ..log import debug
from ..orchestrator_auth import ROLE_CLI
from ..trust_domain import CONTROL_PLANE
from .server import ORCHESTRATOR_AUTH_HEADER
from .http_contract import ORCHESTRATOR_AUTH_HEADER
DEFAULT_TIMEOUT_SECONDS = 5.0
+11
View File
@@ -0,0 +1,11 @@
"""Dependency-free constants shared by orchestrator HTTP clients and server."""
ORCHESTRATOR_AUTH_HEADER = "x-bot-bottle-orchestrator-auth"
MAX_BODY_BYTES = 1 * 1024 * 1024
REQUEST_BODY_TIMEOUT_SECONDS = 10.0
__all__ = [
"MAX_BODY_BYTES",
"ORCHESTRATOR_AUTH_HEADER",
"REQUEST_BODY_TIMEOUT_SECONDS",
]
+55 -439
View File
@@ -1,464 +1,80 @@
"""Orchestrator HTTP control plane (PRD 0070).
The backend-agnostic control-plane RPC (CLI / console -> orchestrator) over
**HTTP** the universal transport chosen in 0070 (works on every host; no
vsock / unix-socket portability caveats):
GET /health -> 200 {"status": "ok"}
GET /gateway -> 200 {"configured", ["name","running"]}
GET /bottles -> 200 {"bottles": [ <redacted record>, ...]}
POST /bottles -> 201 {"bottle_id","identity_token"} (launch)
body: {"source_ip", ["image_ref"],
["metadata"], ["policy"],
["tokens"], ["env_var_secret"]}
PUT /bottles/<bottle_id>/policy -> 200 {"updated": true} | 404 (live reload)
body: {"policy"}
POST /bottles/<bottle_id>/reprovision_gateway
-> 200 {"reprovisioned": true} | 404
body: {"env_var_secret"}
DELETE /bottles/<bottle_id> -> 200 {"torn_down": true} | 404 (teardown)
POST /reconcile -> 200 {"reaped": [bottle_id, ...]}
body: {"live_source_ips": [...],
["grace_seconds"]}
POST /attribute -> 200 {"bottle_id"} | 403
POST /resolve -> 200 {"bottle_id","policy"} | 403
body: {"source_ip","identity_token"}
GET /supervise/proposals -> 200 {"proposals": [ <proposal>, ...]}
POST /supervise/respond -> 200 {"responded": true} | 409 (operator)
body: {"proposal_id","bottle_slug",
"decision", ["notes"],["final_file"]}
POST /supervise/propose -> 201 {"proposal_id"} | 403 (agent)
body: {"source_ip","identity_token",
"tool","proposed_file","justification"}
POST /supervise/poll -> 200 {"status", ["notes"],["final_file"]} | 403
body: {"source_ip","identity_token",
"proposal_id"}
The `/supervise/propose` + `/supervise/poll` pair is the **agent** half of the
supervise flow: the data plane (supervise / egress / git-gate) queues a proposal
and polls for its response over RPC instead of opening `bot-bottle.db` directly.
`poll` is idempotent it never archives, so a dropped connection can't lose an
operator decision (the row is reaped when the bottle is torn down / reconciled).
Both attribute the caller by `(source_ip, identity_token)` exactly like
`/resolve`, so a bottle can only ever queue or read its own proposals.
`POST /bottles` / `DELETE` drive the full launch lifecycle: they mint (or
tear down) the bottle in the registry AND broker the backend-native launch
via the orchestrator. Register/deregister without a launch are internal to
`OrchestratorCore`, not exposed here.
Routing/handling is the pure function `dispatch()` so it is unit-testable
without a socket; `Handler` / `OrchestratorServer` / `make_server` are a
thin stdlib adapter around it. Listing redacts identity tokens they are
returned only once, to the caller that launches the bottle.
"""
"""Uvicorn transport for the FastAPI orchestrator control plane."""
from __future__ import annotations
import http.server
import json
import math
import os
import socketserver
import sys
import typing
from urllib.parse import urlsplit
import socket
import threading
import uvicorn
from ..orchestrator_auth import ROLE_CLI, ROLES
from ..trust_domain import CONTROL_PLANE
from ..supervisor.types import TOOLS
from .api import create_app
from .http_contract import MAX_BODY_BYTES, ORCHESTRATOR_AUTH_HEADER
from .service import OrchestratorCore
# JSON body payload type (parsed request / rendered response).
Json = dict[str, object]
# The request header carrying the caller's role-scoped control-plane token (a
# signed JWT naming the caller's role — see orchestrator_auth). The role gates which
# routes the caller may reach: the data plane holds a `gateway` token good only
# for the agent-facing lookups; the host CLI holds a `cli` token for the
# operator/mutating routes. An agent that can merely *reach* the port holds no
# token at all, and a compromised gateway holds only `gateway` — neither can
# drive the operator routes (approve proposals, rewrite policy, read tokens).
ORCHESTRATOR_AUTH_HEADER = "x-bot-bottle-orchestrator-auth"
# The routes the data plane (role `gateway`) is allowed to reach — exactly the
# per-request lookups PolicyResolver makes. Every other authenticated route is
# operator-only. `cli` is a superset role: it may reach any route.
_GATEWAY_ROUTES: frozenset[tuple[str, str]] = frozenset({
("POST", "/resolve"),
("POST", "/supervise/propose"),
("POST", "/supervise/poll"),
})
MAX_REQUESTS = 32
KEEP_ALIVE_TIMEOUT_SECONDS = 10
def _allowed_roles(method: str, route: str) -> frozenset[str]:
"""The roles permitted on `(method, route)`: `gateway` or `cli` on the
data-plane routes, `cli`-only everywhere else."""
if (method, route) in _GATEWAY_ROUTES:
return ROLES
return frozenset({ROLE_CLI})
class OrchestratorServer:
"""Small lifecycle wrapper around Uvicorn with an eagerly bound socket."""
def __init__(self, config: uvicorn.Config) -> None:
self._server = uvicorn.Server(config)
self._stopped = threading.Event()
self._socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self._socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self._socket.bind((config.host, config.port))
self._socket.listen(config.backlog)
self.server_address = self._socket.getsockname()
def _parse_json_object(body: bytes) -> Json:
"""Parse a JSON object body. Raises ValueError for non-objects / bad JSON."""
if not body:
return {}
obj = json.loads(body) # raises json.JSONDecodeError (a ValueError)
if not isinstance(obj, dict):
raise ValueError("request body must be a JSON object")
return obj
def dispatch( # pylint: disable=too-many-return-statements,too-many-branches
orch: OrchestratorCore, method: str, path: str, body: bytes, *, role: str | None = ROLE_CLI,
) -> tuple[int, Json]:
"""Route one control-plane request to a (status, payload) pair. Pure —
no I/O beyond the orchestrator so it is fully testable without a socket.
`role` is the caller's verified control-plane role (`gateway` or `cli`), or
None for an unauthenticated request; an open-mode server (no signing key
configured see `OrchestratorServer`) passes `cli`. Every route except
`GET /health` requires a role: a missing role is 401, and a role that
doesn't cover the route is 403 — so a `gateway` data-plane token can reach
`/resolve` + `/supervise/{propose,poll}` but not the operator routes
(rewrite policy, read injected tokens, approve its own supervise proposals).
The source-IP + identity-token checks inside `/resolve` and `/attribute`
authenticate the *bottle* a request is about, not the *caller*, so this role
gate is what protects the caller-privileged routes. Defaults `cli` so unit
tests of the routing logic don't have to thread it through."""
route = urlsplit(path).path.rstrip("/") or "/"
if method == "GET" and route == "/health":
return 200, {"status": "ok"}
# Role gate — every route below is a trusted-caller operation. Deny before
# touching the registry / broker / supervise store.
if role is None:
return 401, {"error": "control-plane authentication required"}
if role not in _allowed_roles(method, route):
return 403, {"error": "insufficient role for this route"}
if method == "GET" and route == "/gateway":
return 200, orch.gateway_status()
if method == "GET" and route == "/bottles":
return 200, {"bottles": [r.redacted() for r in orch.registry.all()]}
if method == "POST" and route == "/bottles":
def run(self) -> None:
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
source_ip = data.get("source_ip")
if not isinstance(source_ip, str) or not source_ip:
return 400, {"error": "source_ip (string) is required"}
image_ref = data.get("image_ref")
metadata = data.get("metadata")
policy = data.get("policy")
raw_tokens = data.get("tokens")
tokens = {
k: v for k, v in raw_tokens.items() if isinstance(k, str) and isinstance(v, str)
} if isinstance(raw_tokens, dict) else {}
env_var_secret = data.get("env_var_secret", "")
rec = orch.launch_bottle(
source_ip,
image_ref=image_ref if isinstance(image_ref, str) else "",
metadata=metadata if isinstance(metadata, str) else "",
policy=policy if isinstance(policy, str) else "",
tokens=tokens,
env_var_secret=env_var_secret if isinstance(env_var_secret, str) else "",
)
return 201, {"bottle_id": rec.bottle_id, "identity_token": rec.identity_token}
self._server.run(sockets=[self._socket])
finally:
self._stopped.set()
if method == "PUT" and route.startswith("/bottles/") and route.endswith("/policy"):
bottle_id = route[len("/bottles/"):-len("/policy")]
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
policy = data.get("policy")
if not isinstance(policy, str):
return 400, {"error": "policy (string) is required"}
if orch.set_policy(bottle_id, policy):
return 200, {"updated": True}
return 404, {"error": "no such bottle"}
def serve_forever(self) -> None:
self.run()
if (
method == "POST"
and route.startswith("/bottles/")
and route.endswith("/reprovision_gateway")
):
bottle_id = route[len("/bottles/") : -len("/reprovision_gateway")]
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
env_var_secret = data.get("env_var_secret")
if not isinstance(env_var_secret, str) or not env_var_secret:
return 400, {"error": "env_var_secret (string) is required"}
if orch.reprovision_from_secret(bottle_id, env_var_secret):
return 200, {"reprovisioned": True}
return 404, {"error": "no stored secrets for this bottle"}
def shutdown(self) -> None:
self._server.should_exit = True
self._stopped.wait(timeout=5)
if method == "DELETE" and route.startswith("/bottles/"):
bottle_id = route[len("/bottles/"):]
if orch.teardown_bottle(bottle_id):
return 200, {"torn_down": True}
return 404, {"error": "no such bottle"}
if method == "POST" and route == "/reconcile":
# Host-driven self-heal: the caller enumerates its live bottles (only
# the host can see the backend) and the orchestrator drops rows for
# every other active bottle. Trusted-caller only — an agent that could
# reach this would be able to unregister its neighbours.
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
raw_ips = data.get("live_source_ips")
if not isinstance(raw_ips, list):
return 400, {"error": "live_source_ips (list of strings) is required"}
if any(not isinstance(ip, str) or not ip for ip in raw_ips):
return 400, {"error": "live_source_ips must contain non-empty strings"}
live = raw_ips
grace = data.get("grace_seconds")
kwargs: dict[str, float] = {}
if grace is not None:
if isinstance(grace, bool) or not isinstance(grace, (int, float)):
return 400, {"error": "grace_seconds must be a non-negative finite number"}
parsed_grace = float(grace)
if not math.isfinite(parsed_grace) or parsed_grace < 0:
return 400, {"error": "grace_seconds must be a non-negative finite number"}
kwargs["grace_seconds"] = parsed_grace
return 200, {"reaped": orch.reconcile(live, **kwargs)}
if method == "POST" and route == "/attribute":
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
source_ip = data.get("source_ip")
token = data.get("identity_token")
if not isinstance(source_ip, str) or not isinstance(token, str):
return 400, {"error": "source_ip and identity_token (strings) required"}
rec = orch.attribute(source_ip, token)
if rec is None:
return 403, {"error": "unattributed"}
return 200, {"bottle_id": rec.bottle_id}
if method == "GET" and route == "/supervise/proposals":
# Operator TUI: pending supervise proposals across all bottles.
return 200, {"proposals": orch.supervise_pending()}
if method == "POST" and route == "/supervise/respond":
# Operator decision: apply (approve/modify rewrites egress policy),
# write the queued response, audit — all server-side on the one DB.
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
proposal_id = data.get("proposal_id")
bottle_slug = data.get("bottle_slug")
decision = data.get("decision")
if not (isinstance(proposal_id, str) and proposal_id):
return 400, {"error": "proposal_id (string) is required"}
if not (isinstance(bottle_slug, str) and bottle_slug):
return 400, {"error": "bottle_slug (string) is required"}
if not (isinstance(decision, str) and decision):
return 400, {"error": "decision (string) is required"}
notes = data.get("notes")
final_file = data.get("final_file")
ok, err = orch.supervise_respond(
proposal_id,
bottle_slug=bottle_slug,
decision=decision,
notes=notes if isinstance(notes, str) else "",
final_file=final_file if isinstance(final_file, str) else None,
)
if ok:
return 200, {"responded": True}
return 409, {"error": err}
if method == "POST" and route == "/supervise/propose":
# Agent half: queue a proposal, attributed to the caller resolved from
# (source_ip, identity_token) — never a caller-supplied slug — so the
# data plane can't forge attribution. Fail-closed 403 when unattributed.
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
source_ip = data.get("source_ip")
token = data.get("identity_token")
tool = data.get("tool")
proposed_file = data.get("proposed_file")
justification = data.get("justification")
if not isinstance(source_ip, str) or not source_ip:
return 400, {"error": "source_ip (string) is required"}
if not isinstance(tool, str) or tool not in TOOLS:
return 400, {"error": f"tool (string) must be one of {TOOLS}"}
if not isinstance(proposed_file, str) or not proposed_file:
return 400, {"error": "proposed_file (string) is required"}
if not isinstance(justification, str) or not justification:
return 400, {"error": "justification (string) is required"}
rec = orch.resolve(source_ip, token if isinstance(token, str) else "")
if rec is None:
return 403, {"error": "unattributed"}
proposal_id = orch.supervise_queue_proposal(
rec.bottle_id, tool=tool, proposed_file=proposed_file,
justification=justification,
)
return 201, {"proposal_id": proposal_id}
if method == "POST" and route == "/supervise/poll":
# Agent half: non-blocking read of the caller's own proposal decision.
# Attributed like /propose, and scoped to the resolved bottle id, so a
# guessed proposal_id can never read another bottle's response.
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
source_ip = data.get("source_ip")
token = data.get("identity_token")
proposal_id = data.get("proposal_id")
if not isinstance(source_ip, str) or not source_ip:
return 400, {"error": "source_ip (string) is required"}
if not isinstance(proposal_id, str) or not proposal_id:
return 400, {"error": "proposal_id (string) is required"}
rec = orch.resolve(source_ip, token if isinstance(token, str) else "")
if rec is None:
return 403, {"error": "unattributed"}
return 200, orch.supervise_poll_response(rec.bottle_id, proposal_id)
if method == "POST" and route == "/resolve":
# The per-request lookup the multi-tenant gateway makes: returns the
# bottle's policy. Requires a matching (source_ip, identity_token)
# pair — a missing/empty/mismatched token fail-closes (403), no
# source-IP-only fallback.
try:
data = _parse_json_object(body)
except ValueError as e:
return 400, {"error": f"invalid JSON: {e}"}
source_ip = data.get("source_ip")
token = data.get("identity_token")
if not isinstance(source_ip, str) or not source_ip:
return 400, {"error": "source_ip (string) is required"}
rec = orch.resolve(source_ip, token if isinstance(token, str) else "")
if rec is None:
return 403, {"error": "unattributed"}
# tokens are the in-memory per-bottle egress auth values the gateway
# injects; served here, never persisted.
return 200, {
"bottle_id": rec.bottle_id,
"policy": rec.policy,
"tokens": orch.tokens_for(rec.bottle_id),
}
return 404, {"error": "not found"}
class Handler(http.server.BaseHTTPRequestHandler):
"""Thin stdlib adapter: read the body, call `dispatch`, write JSON."""
# Quiet by default (the orchestrator has its own logging); opt back into
# stdlib access logging with BOT_BOTTLE_ORCHESTRATOR_DEBUG.
def log_message(self, format: str, *args: typing.Any) -> None: # noqa: A002
if os.environ.get("BOT_BOTTLE_ORCHESTRATOR_DEBUG"):
super().log_message(format, *args)
def _serve(self, method: str) -> None:
"""Read the request body, dispatch it, and write the JSON reply. A
dispatch failure (e.g. a broker error) returns a 500 rather than
crashing the connection, so one bad request can't take the control
plane down for the caller."""
server = self.server
assert isinstance(server, OrchestratorServer)
length = int(self.headers.get("Content-Length") or 0)
body = self.rfile.read(length) if length > 0 else b""
role = server.role_for(self.headers.get(ORCHESTRATOR_AUTH_HEADER, ""))
try:
status, payload = dispatch(
server.orchestrator, method, self.path, body, role=role)
except Exception as e: # noqa: BLE001 — the control plane must stay up
# Do not echo exception messages to the caller or logs: broker and
# persistence exceptions can contain request data. The operation,
# route, and exception type are enough to correlate a traceback.
sys.stderr.write(
f"orchestrator: {method} {self.path} failed "
f"[error_type={type(e).__name__}]\n"
)
sys.stderr.flush()
status, payload = 500, {"error": "internal error"}
data = json.dumps(payload).encode()
self.send_response(status)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def do_GET(self) -> None:
self._serve("GET")
def do_POST(self) -> None:
self._serve("POST")
def do_PUT(self) -> None:
self._serve("PUT")
def do_DELETE(self) -> None:
self._serve("DELETE")
class OrchestratorServer(socketserver.ThreadingMixIn, http.server.HTTPServer):
"""Threading HTTP server that carries the orchestrator for its handlers.
Holds the per-host control-plane *signing key* (from
`$BOT_BOTTLE_ORCHESTRATOR_TOKEN`, injected by the launcher into the
orchestrator process only) and verifies each request's role-scoped token
against it. When a key is set, every route but `/health` requires a valid
token whose role covers the route; when it is unset the server runs **open**
(full `cli` access) and says so loudly at startup a fail-visible fallback
for tests and any backend that hasn't wired the key yet (e.g. Firecracker,
whose nft boundary already blocks agents from the control-plane port)."""
daemon_threads = True
allow_reuse_address = True
def __init__(self, address: tuple[str, int], orchestrator: OrchestratorCore) -> None:
self.orchestrator = orchestrator
# The control-plane trust domain's signing key, as injected into THIS
# (the owning) process by the launcher (#476). Unset → open mode below.
self._signing_key = CONTROL_PLANE.key_from_env()
if not self._signing_key:
sys.stderr.write(
"orchestrator: WARNING — no control-plane signing key "
f"(${CONTROL_PLANE.key_env}); running WITHOUT caller "
"authentication. Any client that can reach this port can drive "
"it. Backends that put the control plane on an agent-reachable "
"network MUST set this.\n"
)
sys.stderr.flush()
super().__init__(address, Handler)
def role_for(self, presented: str) -> str | None:
"""The role the request is authorized as, or None if unauthenticated.
Open mode (no signing key) grants full `cli` access the fail-visible
fallback. Otherwise verify the presented signed token; a missing/invalid
token yields None ( 401), a valid one yields its `gateway`/`cli`
role ( per-route 401/403 in `dispatch`)."""
if not self._signing_key:
return ROLE_CLI
return CONTROL_PLANE.verify(presented, self._signing_key)
def server_close(self) -> None:
self._socket.close()
def make_server(
orchestrator: OrchestratorCore, host: str = "127.0.0.1", port: int = 0
orchestrator: OrchestratorCore,
host: str = "127.0.0.1",
port: int = 0,
*,
signing_key: str | None = None,
) -> OrchestratorServer:
"""Build (but do not start) a control-plane server. `port=0` binds an
ephemeral port read `server.server_address` for the actual one."""
return OrchestratorServer((host, port), orchestrator)
"""Build a bounded Uvicorn server around the orchestrator application."""
key = CONTROL_PLANE.key_from_env() if signing_key is None else signing_key
app = create_app(orchestrator, signing_key=key)
config = uvicorn.Config(
app,
host=host,
port=port,
access_log=bool(os.environ.get("BOT_BOTTLE_ORCHESTRATOR_DEBUG")),
log_level="info",
limit_concurrency=MAX_REQUESTS,
timeout_keep_alive=KEEP_ALIVE_TIMEOUT_SECONDS,
server_header=False,
)
return OrchestratorServer(config)
__all__ = [
"dispatch", "Handler", "OrchestratorServer", "make_server", "Json",
"KEEP_ALIVE_TIMEOUT_SECONDS",
"MAX_BODY_BYTES",
"MAX_REQUESTS",
"ORCHESTRATOR_AUTH_HEADER",
"OrchestratorServer",
"create_app",
"make_server",
]
+4 -2
View File
@@ -371,10 +371,12 @@ class OrchestratorCore:
if not encrypted:
return False
try:
self._tokens[bottle_id] = {k: decrypt_value(env_var_secret, v)
for k, v in encrypted.items()}
decrypted = {
k: decrypt_value(env_var_secret, v) for k, v in encrypted.items()
}
except ValueError:
return False
self._tokens[bottle_id] = decrypted
return True
# --- consolidated gateway ----------------------------------------------
@@ -129,6 +129,10 @@ _MIGRATIONS = TableMigrations(
# v5 — index for fast per-bottle lookups and bulk DELETE on teardown.
"CREATE INDEX IF NOT EXISTS idx_bottled_agent_secrets_id "
"ON bottled_agent_secrets (bottled_agent_id, type)",
# v6 — unauthenticated legacy ciphertext must never be selected by
# attacker-controlled blob contents. Existing local agents are
# intentionally reprovisioned instead of retaining downgrade support.
"DELETE FROM bottled_agent_secrets",
],
)
+49 -14
View File
@@ -12,9 +12,15 @@ reattachment path reads ENV_VAR_SECRET from the running agent container via
``POST /bottles/<id>/reprovision_gateway``; the orchestrator decrypts the
stored rows and re-populates ``_tokens``.
Encryption scheme: HMAC-SHA256 used as a PRF in CTR mode (stdlib-only,
no external deps). Each value is encrypted independently. The output blob is
``nonce (16 bytes) || ciphertext`` encoded as URL-safe base64 (no padding).
Encryption scheme: encrypt-then-MAC using independent HMAC-SHA256-derived
encryption and authentication subkeys (stdlib-only, no external deps). Each
value is encrypted independently. New output blobs are:
``version || nonce (16 bytes) || ciphertext || tag (32 bytes)``
encoded as URL-safe base64 (no padding). Unversioned legacy ciphertext is
rejected; the registry migration clears those rows rather than allowing blob
contents to select an unauthenticated decoder.
keystream_block_i = HMAC-SHA256(key, nonce || i.to_bytes(4, "big"))
ciphertext_i = plaintext_i XOR keystream_block_i[:len(plaintext_i)]
@@ -30,6 +36,8 @@ import secrets
_KEY_BYTES = 32 # 256-bit key from ENV_VAR_SECRET
_NONCE_BYTES = 16 # 128-bit random nonce per encrypt call
_BLOCK = 32 # HMAC-SHA256 output width == one keystream block
_TAG_BYTES = 32
_VERSION = b"BBSE1"
# Env-var name the agent container receives at startup.
ENV_VAR_SECRET_NAME = "ENV_VAR_SECRET"
@@ -41,7 +49,13 @@ def new_env_var_secret() -> str:
def _b64dec(s: str) -> bytes:
return base64.urlsafe_b64decode(s + "=" * (-len(s) % 4))
return base64.b64decode(
s + "=" * (-len(s) % 4), altchars=b"-_", validate=True,
)
def _subkey(key: bytes, purpose: bytes) -> bytes:
return hmac.new(key, b"bot-bottle-secret-store:" + purpose, hashlib.sha256).digest()
def _keystream(key: bytes, nonce: bytes, block_index: int) -> bytes:
@@ -53,37 +67,53 @@ def _keystream(key: bytes, nonce: bytes, block_index: int) -> bytes:
def encrypt_value(secret_b64: str, plaintext: str) -> str:
"""Encrypt a single string value with *secret_b64* (the ENV_VAR_SECRET).
Returns a URL-safe base64 blob ``nonce || ciphertext`` suitable for
Returns a URL-safe base64 authenticated blob suitable for
the ``bottled_agent_secrets.value`` column."""
key = _b64dec(secret_b64)
encryption_key = _subkey(key, b"encryption")
authentication_key = _subkey(key, b"authentication")
pt = plaintext.encode()
nonce = secrets.token_bytes(_NONCE_BYTES)
ct = bytearray()
for i in range(0, len(pt), _BLOCK):
chunk = pt[i : i + _BLOCK]
ks = _keystream(key, nonce, i)[: len(chunk)]
ks = _keystream(encryption_key, nonce, i // _BLOCK)[: len(chunk)]
ct.extend(p ^ k for p, k in zip(chunk, ks))
return base64.urlsafe_b64encode(nonce + bytes(ct)).rstrip(b"=").decode()
authenticated = _VERSION + nonce + bytes(ct)
tag = hmac.new(authentication_key, authenticated, hashlib.sha256).digest()
return base64.urlsafe_b64encode(authenticated + tag).rstrip(b"=").decode()
def decrypt_value(secret_b64: str, blob_b64: str) -> str:
"""Decrypt a blob produced by :func:`encrypt_value`.
Returns the original plaintext string. Raises ``ValueError`` for malformed
input or a key mismatch (wrong key produces garbage, not an error, unless
the plaintext is non-UTF-8 treat all such failures as wrong key)."""
input, authentication failure, or a key mismatch."""
key = _b64dec(secret_b64)
try:
blob = _b64dec(blob_b64)
except Exception as exc:
except (ValueError, TypeError) as exc:
raise ValueError(f"invalid ciphertext blob: {exc}") from exc
if len(blob) < _NONCE_BYTES:
if not blob.startswith(_VERSION):
raise ValueError("unsupported ciphertext format")
minimum = len(_VERSION) + _NONCE_BYTES + _TAG_BYTES
if len(blob) < minimum:
raise ValueError("ciphertext blob too short")
nonce, ciphertext = blob[:_NONCE_BYTES], blob[_NONCE_BYTES:]
authenticated, supplied_tag = blob[:-_TAG_BYTES], blob[-_TAG_BYTES:]
authentication_key = _subkey(key, b"authentication")
expected_tag = hmac.new(
authentication_key, authenticated, hashlib.sha256,
).digest()
if not hmac.compare_digest(supplied_tag, expected_tag):
raise ValueError("ciphertext authentication failed")
nonce_start = len(_VERSION)
nonce = blob[nonce_start : nonce_start + _NONCE_BYTES]
ciphertext = blob[nonce_start + _NONCE_BYTES : -_TAG_BYTES]
encryption_key = _subkey(key, b"encryption")
pt = bytearray()
for i in range(0, len(ciphertext), _BLOCK):
chunk = ciphertext[i : i + _BLOCK]
ks = _keystream(key, nonce, i)[: len(chunk)]
ks = _keystream(encryption_key, nonce, i // _BLOCK)[: len(chunk)]
pt.extend(c ^ k for c, k in zip(chunk, ks))
try:
return bytes(pt).decode()
@@ -91,4 +121,9 @@ def decrypt_value(secret_b64: str, blob_b64: str) -> str:
raise ValueError(f"decryption produced non-UTF-8 output (wrong key?): {exc}") from exc
__all__ = ["ENV_VAR_SECRET_NAME", "new_env_var_secret", "encrypt_value", "decrypt_value"]
__all__ = [
"ENV_VAR_SECRET_NAME",
"new_env_var_secret",
"encrypt_value",
"decrypt_value",
]
+1
View File
@@ -44,6 +44,7 @@ BUNDLED_RESOURCES: tuple[str, ...] = (
"Dockerfile.orchestrator",
"Dockerfile.orchestrator.fc",
"requirements.gateway.lock",
"requirements.orchestrator.lock",
"nix/firecracker-netpool.nix",
"scripts/firecracker-netpool.sh",
)
+57 -9
View File
@@ -2,7 +2,9 @@
from __future__ import annotations
import os
import sqlite3
import stat
from contextlib import contextmanager
from pathlib import Path
@@ -19,9 +21,62 @@ class DbStore:
def __init__(self, db_path: Path, migrations: TableMigrations) -> None:
self.db_path = db_path
self._migrations = migrations
self.db_path.parent.mkdir(parents=True, exist_ok=True)
self._secure_parent()
if self.db_path.exists():
self._chmod()
def _secure_parent(self) -> None:
"""Create and verify the private parent directory."""
parent = self.db_path.parent
parent.mkdir(mode=0o700, parents=True, exist_ok=True)
if parent.is_symlink():
raise PermissionError(f"database directory must not be a symlink: {parent}")
parent.chmod(0o700)
parent_stat = parent.lstat()
if not stat.S_ISDIR(parent_stat.st_mode):
raise PermissionError(f"database parent is not a directory: {parent}")
if stat.S_IMODE(parent_stat.st_mode) != 0o700:
raise PermissionError(f"database directory is not mode 0700: {parent}")
def _secure_db_file(self) -> None:
"""Create the database without a permissive filesystem window.
SQLite otherwise creates a missing database using the process umask.
This store contains control-plane identity tokens, so both creation and
repair are fail-closed rather than best-effort.
"""
flags = os.O_RDWR | os.O_CREAT | os.O_NOFOLLOW
fd = os.open(
self.db_path,
flags,
stat.S_IRUSR | stat.S_IWUSR,
)
try:
self._secure_open_file(fd)
finally:
os.close(fd)
def _chmod(self) -> None:
"""Enforce and verify the private database mode after every write."""
fd = os.open(self.db_path, os.O_RDWR | os.O_NOFOLLOW)
try:
self._secure_open_file(fd)
finally:
os.close(fd)
def _secure_open_file(self, fd: int) -> None:
"""Pin, validate, and secure an opened database filesystem object."""
file_stat = os.fstat(fd)
if not stat.S_ISREG(file_stat.st_mode):
raise PermissionError(
f"database must be a regular file: {self.db_path}"
)
os.fchmod(fd, stat.S_IRUSR | stat.S_IWUSR)
if stat.S_IMODE(os.fstat(fd).st_mode) != 0o600:
raise PermissionError(f"database is not mode 0600: {self.db_path}")
def _connect(self) -> sqlite3.Connection:
self._secure_db_file()
conn = sqlite3.connect(self.db_path)
conn.row_factory = sqlite3.Row
return conn
@@ -51,16 +106,9 @@ class DbStore:
return version == len(self._migrations.migrations)
def migrate(self) -> None:
"""Apply any pending migrations and set permissions on the DB file."""
"""Apply any pending migrations to the already-secured DB file."""
with self._connection() as conn:
self._migrations.apply(conn)
self._chmod()
def _chmod(self) -> None:
try:
self.db_path.chmod(0o600)
except OSError:
pass
__all__ = ["DbStore", "DbVersionError"]
+3 -4
View File
@@ -40,7 +40,7 @@ from .paths import (
class ProvisioningError(RuntimeError):
"""A control-plane auth invariant would be violated (e.g. starting the
orchestrator without its signing key which would run OPEN)."""
orchestrator without its signing key)."""
@dataclass(frozen=True)
@@ -67,9 +67,8 @@ class TrustDomain:
def key_from_env(self, environ: Mapping[str, str] | None = None) -> str:
"""The signing key as the owning process sees it — read from `key_env`
(default `os.environ`). "" when unset; the caller decides whether that is
fatal (`ControlPlaneProvisioning`) or the open-mode fallback
(`OrchestratorServer`)."""
(default `os.environ`). ``""`` when unset; owning services reject that
value rather than start without authentication."""
env = os.environ if environ is None else environ
return env.get(self.key_env, "").strip()
+6 -2
View File
@@ -3,6 +3,10 @@
- **Status:** Accepted
- **Date:** 2026-06-25
- **Deciders:** didericis
- **Revised:** 2026-07-27 — thresholds relaxed (critical minimum 90→85%,
diff-coverage gate 90→80%) to cut low-value test churn on changed lines.
The risk-weighting structure and the "global is informational" rule are
unchanged.
## Context
@@ -34,7 +38,7 @@ a regression (Goodhart's law).
Coverage is **risk-weighted**, measured over the **combined unit +
integration** suites, with three rules:
1. **Critical modules must remain ≥ 90%.** The curated security/logic core
1. **Critical modules must remain ≥ 85%.** The curated security/logic core
covers the host and gateway egress policy, manifest trust boundary,
git-gate enforcement, supervise protocol/server, YAML parser, and bottle
state. The concrete module list lives in `scripts/critical-modules.txt`;
@@ -55,7 +59,7 @@ integration** suites, with three rules:
The forward-looking guard is a **diff-coverage gate**
(`scripts/diff_coverage.py`): new/changed executable lines on a branch
must be ≥ 90% covered. This catches regressions where they are
must be ≥ 80% covered. This catches regressions where they are
introduced without forcing a back-fill crusade through legacy glue. The
gate skips lines in omitted files (there is no coverage data for them),
so the omit list cannot launder *new* logic into the dark: anything that
+2 -2
View File
@@ -1,4 +1,4 @@
# VHS tape — drives `./cli.py start demo` interactively and asks
# VHS tape — drives `bot-bottle start demo` interactively and asks
# claude (the AI) to run four probes via natural-language prompts.
# Setup (manifest + dummy SSH key + image pre-warm) and teardown
# happen outside the tape; record via `bash scripts/demo-record.sh`,
@@ -29,7 +29,7 @@ Show
# defaults), one git upstream (unreachable on purpose so gitleaks runs
# before the gate would forward), and a FAKE_TOKEN env var shaped like
# a GitHub PAT.
Type "./cli.py start demo"
Type "bot-bottle start demo"
Enter
Sleep 8s
+1 -1
View File
@@ -132,7 +132,7 @@ Each test runs against a temporary `$HOME` and a temporary `$CWD`:
can revisit, but the v1 of this PRD is one file = one bottle.
- **Hot-reload.** Changes to manifest files take effect at next
`./cli.py start`; we do not watch the directory.
`bot-bottle start`; we do not watch the directory.
## Scope
+2 -2
View File
@@ -290,7 +290,7 @@ After this PRD:
### Cleanup CLI
`./cli.py cleanup` switches from "list every container with prefix
`bot-bottle cleanup` switches from "list every container with prefix
`bot-bottle-` and every network with prefix `bot-bottle-net-`
or `bot-bottle-egress-`" to:
@@ -369,7 +369,7 @@ Sized for one PR each, in order.
`docker compose up -d` + attach + teardown. Per-sidecar `start()`/
`stop()` lifecycle methods deleted in the same chunk. Compose-
log dump on teardown added.
4. **Cleanup CLI on compose.** Switch `./cli.py cleanup` to
4. **Cleanup CLI on compose.** Switch `bot-bottle cleanup` to
`docker compose ls`-based discovery; keep prefix-scan as
fallback for one release.
5. **Dashboard.** Decide on the discovery question (open question
+4 -4
View File
@@ -37,7 +37,7 @@ Two rough edges in the current dashboard:
shows only pending proposals. If no agent has called a tool,
the screen reads "no pending proposals" — even when five
bottles are quietly working. The operator has to `docker
compose ls` (or `./cli.py cleanup -n` to see the y/N preview)
compose ls` (or `bot-bottle cleanup -n` to see the y/N preview)
to find out what's actually live.
2. **`e` / `p` re-discover-and-disambiguate every invocation.**
@@ -82,12 +82,12 @@ the "operator wants to make an unprompted change" case.
global across bottles. Filtering ("show me only this agent's
proposals") might be a follow-up but isn't this PRD.
- **Agent lifecycle from the dashboard.** Starting / stopping
agents stays in `./cli.py start` / `./cli.py cleanup`. The
agents stays in `bot-bottle start` / `bot-bottle cleanup`. The
dashboard reads state; it doesn't change it.
- **Preserved-but-not-running bottles.** The active-agents list
is strictly "what's running now" (cross-referenced from
`docker compose ls`). Preserved state dirs without a live
project don't appear — `./cli.py resume <identity>` is the
project don't appear — `bot-bottle resume <identity>` is the
path for those.
- **A separate per-agent detail view.** The agent rows are
one-line summaries. Pressing Enter on a proposal still drops
@@ -125,7 +125,7 @@ the "operator wants to make an unprompted change" case.
- Changes to proposal handling (`a` / `m` / `r` / Enter all
unchanged).
- Changes to the queue-dir / supervise sidecar protocol.
- New CLI surface beyond what's in `./cli.py dashboard`.
- New CLI surface beyond what's in `bot-bottle dashboard`.
- Touching the manifest, compose renderer, launch lifecycle.
## Proposed design
@@ -14,8 +14,8 @@
Today the dashboard is read-only: it surfaces pending proposals
and active agents (PRD 0019) but can't *start* an agent or
*re-enter* one. The operator's path is split — they launch
agents from one terminal (`./cli.py start <name>`), and watch
them from another (`./cli.py dashboard`).
agents from one terminal (`bot-bottle start <name>`), and watch
them from another (`bot-bottle dashboard`).
This PRD collapses that split. The dashboard becomes the
operator's single surface: pressing a key opens an agent picker,
@@ -31,7 +31,7 @@ claude session AND the dashboard process. Exit claude → back to
dashboard, bottle still running. Start another agent → two
bottles up at once. Quit the dashboard → bottles continue
running. Teardown is **always explicit**: the operator presses
`x` on an agent, or runs `./cli.py cleanup` later.
`x` on an agent, or runs `bot-bottle cleanup` later.
## Problem
@@ -45,7 +45,7 @@ Two real frictions today:
open and the dashboard's "active agents" pane is hopelessly
behind reality because they just spawned three in a row.
2. **`./cli.py start` ties the bottle to a single claude
2. **`bot-bottle start` ties the bottle to a single claude
session.** The start command's `ExitStack` brings the bottle
up, runs claude, and tears down on Ctrl-D — fine for a one-
shot session, wrong for "let me bounce in and out of this
@@ -60,7 +60,7 @@ captures full-merged logs per bottle (PRD 0018). It already
## Goals / Success Criteria
1. From inside `./cli.py dashboard`, pressing `n` (new) opens
1. From inside `bot-bottle dashboard`, pressing `n` (new) opens
an agent picker listing every agent defined in the manifest.
Selecting one runs `prepare → preflight → launch`.
2. The preflight Y/N summary renders cleanly — either as a
@@ -83,7 +83,7 @@ captures full-merged logs per bottle (PRD 0018). It already
state cleanup) without quitting the dashboard.
7. Quitting the dashboard (`q`) leaves every running bottle
running. Bottle teardown is always explicit (per-bottle `x`
or `./cli.py cleanup`). The next `./cli.py dashboard`
or `bot-bottle cleanup`). The next `bot-bottle dashboard`
invocation re-discovers them via `list_active_slugs()` and
surfaces re-attach for any it can reconstruct context for
(see "Cross-dashboard re-attach" below).
@@ -94,13 +94,13 @@ captures full-merged logs per bottle (PRD 0018). It already
embedded-emulator option from the research doc is out of
scope. The handoff (option 1) is the v1; option 2 is a
separate PRD if and when handoff is observably insufficient.
- **Adopting bottles started by an out-of-dashboard `./cli.py
- **Adopting bottles started by an out-of-dashboard `bot-bottle
start` invocation.** Those have their own ExitStack-owner and
the dashboard treats them as read-only-watch (already does
today). Re-attach only applies to bottles the *current
dashboard process* started.
- **Resurrecting an out-of-process bottle into a new dashboard
with full re-attach.** A bottle started by `./cli.py start`
with full re-attach.** A bottle started by `bot-bottle start`
in another terminal — or by a previous dashboard run, now
exited — appears in the agents pane (already does, PRD 0019)
and can be re-attached via `docker exec -it claude` because
@@ -109,12 +109,12 @@ captures full-merged logs per bottle (PRD 0018). It already
context object to drive teardown — e.g., the
ExitStack-tracked CA + state cleanup `_settle_state` performs
today. Cross-dashboard re-attach uses the existing
`./cli.py cleanup` for teardown, not an `x` keypress (see
`bot-bottle cleanup` for teardown, not an `x` keypress (see
open questions).
- **Multi-window UI.** Single curses window, two existing
panes (proposals + agents); the agent picker is a modal, not
a third pane.
- **Removing `./cli.py start`.** Stays as the script-friendly /
- **Removing `bot-bottle start`.** Stays as the script-friendly /
legacy entry point. The dashboard is the new default.
## Scope
@@ -140,7 +140,7 @@ captures full-merged logs per bottle (PRD 0018). It already
### Out of scope
- Changes to `./cli.py start` itself. It keeps its current
- Changes to `bot-bottle start` itself. It keeps its current
shape; the dashboard reuses its internal pieces (backend.
prepare / backend.launch) without reaching through the CLI
layer.
@@ -157,7 +157,7 @@ captures full-merged logs per bottle (PRD 0018). It already
Today's flow:
```
./cli.py start agent
bot-bottle start agent
└─ with backend.launch(plan) as bottle: ← bottle alive while inside `with`
bottle.exec_agent([...], tty=True) ← blocks until claude exits
# context exits → compose down → state cleanup
@@ -166,7 +166,7 @@ Today's flow:
The proposed dashboard-driven flow:
```
./cli.py dashboard
bot-bottle dashboard
└─ bottles: dict[str, tuple[ContextManager, DockerBottle]] = {}
# operator presses `n`, picks agent
@@ -205,7 +205,7 @@ Two shifts:
evaluation, state-dir reap) doesn't fire on a quit-while-
running bottle. It DOES fire when the operator explicitly
stops via `x`, because that calls `cm.__exit__`. For
bottles a previous dashboard quit on, `./cli.py cleanup`
bottles a previous dashboard quit on, `bot-bottle cleanup`
is the path — its compose-down + state-reap logic
already covers the case.
@@ -213,7 +213,7 @@ Two shifts:
When the dashboard discovers a bottle in `discover_active_agents`
that it didn't itself start (a previous-dashboard or external
`./cli.py start` bottle), Enter still attaches via `docker exec
`bot-bottle start` bottle), Enter still attaches via `docker exec
-it … claude` — the agent container is running `sleep infinity`
exactly the same way regardless of who started it. The only
thing the current dashboard lacks for those bottles is the
@@ -221,8 +221,8 @@ launch-context object needed to drive a clean teardown via
`x`.
For v1 we surface this honestly: pressing `x` on a non-owned
agent shows a status hint pointing at `./cli.py cleanup` (or
`./cli.py cleanup` targeted at the slug if we add that flag
agent shows a status hint pointing at `bot-bottle cleanup` (or
`bot-bottle cleanup` targeted at the slug if we add that flag
later). The agent stays alive; the operator handles teardown
out-of-band. Enter (re-attach) works for both owned and
non-owned bottles.
@@ -288,7 +288,7 @@ agents pane.
`x` on a non-owned agent (discovered via `list_active_slugs`
but not in `bottles` dict): no-op with status hint pointing
at `./cli.py cleanup` (the existing path that tears down
at `bot-bottle cleanup` (the existing path that tears down
ANY bot-bottle compose project plus reaps state dirs).
### Dashboard quit
@@ -300,7 +300,7 @@ the `docker compose` project keeps running. The next dashboard
invocation discovers the bottles via `list_active_slugs` and
surfaces re-attach.
This is a real departure from today's `./cli.py start`
This is a real departure from today's `bot-bottle start`
semantics (which couples bottle lifetime to the process via
ExitStack). It's intentional: the dashboard is a watching +
acting surface, not a lifetime owner.
@@ -322,7 +322,7 @@ Sized for one PR each.
dashboard's ExitStack; handoff invokes `attach_agent`.
3. **Re-attach via Enter on owned agents-pane row.** Looks up
the slug in the dashboard's `bottles` map; if present →
handoff; else → status-line hint pointing at `./cli.py
handoff; else → status-line hint pointing at `bot-bottle
resume`.
4. **Explicit per-bottle stop (`x` keybinding).** Pop the
bottle's `close` callback off the stack, call it, refresh.
@@ -369,7 +369,7 @@ Sized for one PR each.
bottles dict goes out of scope without invoking `__exit__`,
so the `docker compose` projects keep running. Bottle
teardown is always explicit: per-bottle `x` (for
dashboard-owned), or `./cli.py cleanup` (for everything).
dashboard-owned), or `bot-bottle cleanup` (for everything).
## Open questions
+3 -3
View File
@@ -46,7 +46,7 @@ window, two panes, no terminal handoff.
## Goals / Success Criteria
1. When the operator runs `./cli.py dashboard` from inside a
1. When the operator runs `bot-bottle dashboard` from inside a
tmux session (`$TMUX` set), the dashboard establishes a
two-pane layout: dashboard in the left pane, an initially-
empty right pane reserved for claude sessions.
@@ -313,7 +313,7 @@ Sized small.
3. **Dashboard launched OUTSIDE tmux but tmux is installed.**
Should the dashboard auto-exec itself inside a fresh tmux
session to get the split-pane experience? Convenient but
surprising (`./cli.py dashboard` shouldn't silently
surprising (`bot-bottle dashboard` shouldn't silently
change what session you're in). v1 leaves this off —
operators who want split-pane mode start tmux themselves
and then run the dashboard.
@@ -332,7 +332,7 @@ Sized small.
PRD-0019 focus indicator?
6. **Concurrent dashboards in different tmux windows.**
Multiple `./cli.py dashboard` invocations in different
Multiple `bot-bottle dashboard` invocations in different
tmux windows would each create their own right pane —
probably fine, each has its own state, but worth
verifying that `tmux list-panes` is scoped to the right
+1 -1
View File
@@ -14,7 +14,7 @@ Today bot-bottle is hard-wired around Claude Code assumptions. When Claude runs
## Goals / Success Criteria
- A Codex agent can be started from the dashboard and via `./cli.py start` alongside a Claude agent.
- A Codex agent can be started from the dashboard and via `bot-bottle start` alongside a Claude agent.
- The manifest can express the agent provider/template and, where needed, a custom agent Dockerfile.
- Claude-specific default egress/auth behavior is no longer implicit; provider-specific auth is expressed through explicit bottle egress routes and roles.
- The launcher preserves required infrastructure behavior for sidecars, egress, pipelock, supervisor MCP, CA handling, git, and shell basics.
@@ -30,7 +30,7 @@ across every bottle spin-up. This has several consequences:
to grant that access.
- **Manual rotation burden.** Operators must manage key files on disk, keeping
them secure, rotating them on a schedule, and distributing them across hosts
that run `./cli.py start`.
that run `bot-bottle start`.
## Goals / Success Criteria
@@ -6,7 +6,7 @@
## Summary
The `./cli.py dashboard` command has grown from its PRD 0013 roots
The `bot-bottle dashboard` command has grown from its PRD 0013 roots
(triage supervise proposals) into a parallel-agent control surface
(PRDs 0019/0020/0021): an active-agents pane, agent picker + start,
re-attach, per-bottle stop, tmux split-pane handoff, operator-
@@ -21,7 +21,7 @@ proposals, approve / modify / reject each one, write audit entries,
deliver the response that unblocks the agent's tool call. Everything
that's about *starting / re-entering / stopping* bottles, or about
*operator-initiated* config edits, comes out. The command is renamed
`./cli.py supervise` so the name matches what it does after the cut.
`bot-bottle supervise` so the name matches what it does after the cut.
Future agent-management UX is explicitly punted: if and when a
control surface for parallel agents resurfaces, the working
@@ -41,9 +41,9 @@ Three concrete pains, all downstream of the dashboard's growth:
ExitStack-free bottle ownership are intricate enough that
shipping the next polish increment costs more than it returns.
2. **No clear ownership of "starts and stops bottles".** Today
that responsibility is split: `./cli.py start` owns one-shot
that responsibility is split: `bot-bottle start` owns one-shot
sessions; the dashboard owns multi-session bottles it started
itself; `./cli.py cleanup` owns everything else. The dashboard
itself; `bot-bottle cleanup` owns everything else. The dashboard
tracking its own `bottles: dict[str, (cm, bottle, identity)]`
that doesn't survive a quit is a confusing third lane.
3. **Wrong target shape for a "manage many agents" UI.** The
@@ -97,12 +97,12 @@ problem is everything that got bolted onto that core after.
dashboard. After this PRD they don't exist anywhere — operators
who need ad-hoc edits use the same path the agents do (call the
supervise tool from inside the bottle) or hand-edit the host-
side files and restart the sidecar. Adding a `./cli.py routes
side files and restart the sidecar. Adding a `bot-bottle routes
edit <slug>` verb is a follow-up if the loss bites.
- **Removing `./cli.py start` or changing its semantics.** Start
- **Removing `bot-bottle start` or changing its semantics.** Start
remains the one-shot launch path. PRD 0020's bottle-outlives-
process model is removed; the only path to a long-running
bottle is `./cli.py start` (foreground) plus `cli.py cleanup`
bottle is `bot-bottle start` (foreground) plus `cli.py cleanup`
for teardown.
- **Removing the supervise-sidecar protocol or any of the three
block-remediation engines.** PRDs 00130016 stay Active. The
@@ -122,8 +122,8 @@ problem is everything that got bolted onto that core after.
### In scope
- **Rename the subcommand.** `./cli.py dashboard` becomes
`./cli.py supervise`. The module moves from `bot_bottle/cli/
- **Rename the subcommand.** `bot-bottle dashboard` becomes
`bot-bottle supervise`. The module moves from `bot_bottle/cli/
dashboard.py` to `bot_bottle/cli/supervise.py`. The dispatcher
in `bot_bottle/cli/__init__.py` and the help text both update.
- **Strip the curses loop to proposal-only.** The remaining
@@ -167,7 +167,7 @@ problem is everything that got bolted onto that core after.
- Any new feature in the supervise TUI. The cut is purely
subtractive (except for the rename).
- Behavior changes in `./cli.py start`, `cli.py cleanup`,
- Behavior changes in `bot-bottle start`, `cli.py cleanup`,
`cli.py resume`, `cli.py list`, `cli.py info`, `cli.py edit`,
`cli.py init` — unchanged.
- Changes to the supervise sidecar (`supervise_server.py`,
@@ -181,7 +181,7 @@ problem is everything that got bolted onto that core after.
### Final shape of the TUI
After this PRD the `./cli.py supervise` curses surface is:
After this PRD the `bot-bottle supervise` curses surface is:
```
bot-bottle supervise (3 pending)
@@ -307,8 +307,8 @@ The PR closes issue #174.
1. **`e` / `p` operator-initiated edits — gone for good or
moved to a separate CLI verb?** The PRD removes them with no
replacement. The simplest replacement is `./cli.py routes
edit <slug>` and `./cli.py pipelock edit <slug>`, sharing
replacement. The simplest replacement is `bot-bottle routes
edit <slug>` and `bot-bottle pipelock edit <slug>`, sharing
the existing `apply_routes_change` / `apply_allowlist_change`
engines. If the loss is felt within the first parallel
run after this lands, that follow-up is a small PR. Leaving
+9 -9
View File
@@ -7,12 +7,12 @@
## Summary
When `./cli.py start` is run without an agent name, or without a backend
When `bot-bottle start` is run without an agent name, or without a backend
explicitly specified, the user currently gets an argparse error (missing
positional) or falls through to the `docker` default silently. This PRD
adds a terminal UI that appears in those gaps: a filter-select screen
built with `curses` that lets the operator pick the agent and/or backend
interactively rather than memorising names or consulting `./cli.py list`.
interactively rather than memorising names or consulting `bot-bottle list`.
## Problem
@@ -29,15 +29,15 @@ visible.
## Goals / Success Criteria
1. `./cli.py start` (no arguments) shows an interactive agent selector;
1. `bot-bottle start` (no arguments) shows an interactive agent selector;
the selected name is used exactly as if it had been passed on the
command line.
2. `./cli.py start <name>` (no `--backend`, no `BOT_BOTTLE_BACKEND`)
2. `bot-bottle start <name>` (no `--backend`, no `BOT_BOTTLE_BACKEND`)
shows an interactive backend selector; the selected backend is used
exactly as if `--backend=<selected>` had been passed.
3. `./cli.py start <name> --backend=<b>` (both explicit) shows neither
3. `bot-bottle start <name> --backend=<b>` (both explicit) shows neither
screen — no behavioural change from today.
4. `./cli.py start` (no arguments, no env backend) shows the agent
4. `bot-bottle start` (no arguments, no env backend) shows the agent
selector first, then the backend selector.
5. The filter-select widget is a standalone utility
(`bot_bottle/cli/tui.py`) shared by both selectors.
@@ -57,7 +57,7 @@ visible.
- No pagination beyond what fits in the terminal window (scroll via
cursor movement is sufficient for typical agent counts).
- No multi-select; exactly one item is chosen per invocation.
- No changes to `./cli.py resume`, `./cli.py list`, or any other
- No changes to `bot-bottle resume`, `bot-bottle list`, or any other
subcommand.
## Design
@@ -83,7 +83,7 @@ def filter_select(
The widget renders to the tty file descriptor opened via `curses.initscr`
(or `curses.newterm` on the tty fd so stdout remains clean for callers
that pipe `./cli.py`).
that pipe `bot-bottle`).
Layout (full-width, minimal):
@@ -140,7 +140,7 @@ agent picker can populate itself from the real manifest. The same
`filter_select` opens `/dev/tty` and feeds it as the input file to
`curses.wrapper`-equivalent code (using `curses.newterm` to avoid
clobbering the caller's stdout/stderr). This keeps the picker
composable — callers can pipe `./cli.py` output without the curses
composable — callers can pipe `bot-bottle` output without the curses
draw sequences contaminating the pipe.
## Implementation chunks
+3 -3
View File
@@ -30,12 +30,12 @@ snapshot before a planned host reboot or hardware migration.
## Goals / Success Criteria
- `./cli.py commit [<slug>]` takes a snapshot of the running agent and
- `bot-bottle commit [<slug>]` takes a snapshot of the running agent and
stores it as a local artifact.
- Without a slug argument the command shows the same interactive picker
as `start` (the list of active slugs).
- The committed artifact reference is stored in per-bottle state so
that the next `./cli.py resume <slug>` automatically uses the
that the next `bot-bottle resume <slug>` automatically uses the
snapshot instead of rebuilding from the Dockerfile.
- `mark_preserved` is called so the state dir survives the normal
session-end cleanup.
@@ -81,7 +81,7 @@ to the committed `.smolmachine` artifact.
### `commit` command
```
./cli.py commit [<slug>]
bot-bottle commit [<slug>]
```
1. Resolve slug (arg or interactive picker from `enumerate_active_agents`).
@@ -37,7 +37,7 @@ egress policy), the operator must duplicate the agent file and change the
selection order, as the effective bottle for the session.
5. Confirming with an empty selection falls back to the agent's `bottle:` field.
If neither is set, a ManifestError is raised pointing the operator at the fix.
6. The ordered bottle list is stored in launch metadata so `./cli.py resume`
6. The ordered bottle list is stored in launch metadata so `bot-bottle resume`
uses the same bottles.
7. The preflight summary (`y/N` screen) shows the effective bottle name(s).
8. The multi-select picker supports incremental filtering, Space/Enter to toggle
@@ -52,7 +52,7 @@ egress policy), the operator must duplicate the agent file and change the
- Reordering the selection list from within the picker (order = insertion order;
drag-and-drop is out of scope).
- Storing bottle selection history / MRU.
- Changes to `./cli.py edit`, `./cli.py list`, or `./cli.py info`.
- Changes to `bot-bottle edit`, `bot-bottle list`, or `bot-bottle info`.
- Removing the `bottle:` key from the agent schema (it stays, now optional).
## Design
+1 -1
View File
@@ -74,7 +74,7 @@ macOS-only for v1. Three concrete blockers:
## Goals / Success Criteria
- `BOT_BOTTLE_BACKEND=smolmachines ./cli.py start <agent>` launches,
- `BOT_BOTTLE_BACKEND=smolmachines bot-bottle start <agent>` launches,
runs, and tears down a bottle on a Linux host with `/dev/kvm`.
- The TSI allowlist is enforced on Linux: PRD 0022's
`tests/integration/test_sandbox_escape.py` passes against
+2 -2
View File
@@ -39,7 +39,7 @@ end-to-end runner that would catch the *next* macOS-only launch regression.)
PR #470 (#414) already made the integration suite backend-agnostic:
`skip_unless_selected_backend_available()` gates on the *selected* backend's
own `is_backend_ready()` rather than `docker_available()`, and each
integration job runs `./cli.py backend status --backend=<name>` as a preflight
integration job runs `bot-bottle backend status --backend=<name>` as a preflight
that fails loudly when the backend is missing. That is the machinery this job
plugs into; this PRD supplies the runner and the job.
@@ -98,7 +98,7 @@ Modeled on `integration-firecracker`:
- `concurrency: { group: integration-macos-infra, cancel-in-progress: false }`
to serialize runs against the singleton.
- **Preflight**`command -v container`, `container system status`, then
`./cli.py backend status --backend=macos-container`; any failure exits
`bot-bottle backend status --backend=macos-container`; any failure exits
non-zero so a misprovisioned runner fails loudly instead of silently
skipping.
- Run the integration suite under coverage with
+1 -1
View File
@@ -16,7 +16,7 @@ verifies host prerequisites after install.
## Problem
There is currently no install path for new users. The only way to run
bot-bottle is to clone the repo and invoke `./cli.py`. This blocks any
bot-bottle is to clone the repo and invoke `bot-bottle`. This blocks any
public demo: readers want `curl | sh` or `pipx install`, not a manual
clone-and-configure flow. There is also no single command that tells a
user whether their host is actually ready to run a bottle.
@@ -56,8 +56,7 @@ key.
- Rewriting the HMAC primitive: `orchestrator_auth.mint/verify` gain an optional
`roles=` arg (default unchanged) so a key can carry a different role set;
nothing else changes.
- Network topology, the plane split (#469), or the server's open-mode fallback
for tests.
- Network topology or the plane split (#469).
## Design
@@ -0,0 +1,197 @@
# PRD 0082: Authoritative failure boundaries
- **Status:** Draft
- **Author:** codex
- **Created:** 2026-07-27
- **Issue:** #444
## Summary
Make every security- or lifecycle-sensitive snapshot distinguish authoritative
empty state from unavailable state, and make every destructive or
resource-consuming boundary revalidate the assumptions it acts on. This
finishes the focused quality work begun under #444 without broad rewrites:
cleanup cannot act on stale identities, policy introspection cannot publish a
fabricated empty policy, gateway servers bound untrusted work, and daemon
shutdown does not emit uncaught background-thread failures. Shared
control-plane storage and gateway credential provisioning also enforce their
filesystem security contract before sensitive data is written.
## Problem
Several paths are individually fail-closed but compose into unsafe or
misleading behavior:
1. `cleanup` prepares a plan, waits indefinitely for operator confirmation,
then kills stored PIDs and removes stored paths without checking that those
identities still describe the same orphan. A PID may be reused or a run
directory may become active during the prompt.
2. The supervisor reuses egress's deny-all fallback for
`list-egress-routes`. Deny-all is correct for enforcement, but presenting it
as a successful empty route table can cause a later replace-all proposal to
discard live routes.
3. The supervisor and Git HTTP services accept bounded declared body sizes but
use blocking reads and unbounded request threads. An untrusted bottle can
exhaust the shared gateway with slow or parallel requests.
4. macOS cleanup enumerates containers and networks independently and treats a
failed query as an empty class, so a partial snapshot can still become a
destructive plan.
5. Gateway log-pump threads race stream closure during shutdown and emit
uncaught exceptions even when shutdown otherwise succeeds.
6. Firecracker discovers VMs through whitespace-split `pgrep -a` output.
A configured cache path containing spaces can hide a live VM from the
snapshot and make its run directory appear orphaned.
7. Docker cleanup asks compose for its project snapshot in best-effort mode.
A transient query failure can therefore become an empty stopped-project
set and authorize deletion of associated state directories.
8. Firecracker artifact downloads and registry publication have no network
deadline, so an unresponsive registry can hold setup or release work
indefinitely.
9. Authenticated secret blobs select the unauthenticated legacy decoder when
their in-band version prefix is changed, allowing storage tampering to
bypass tag verification.
10. Cleanup executes the entire post-confirmation snapshot rather than the
intersection with what the operator saw, and mutation failures are not
reflected in the command result.
11. Git smart-HTTP can retain sixteen 100 MiB request bodies concurrently,
cleanup mutations have no subprocess deadline, and Firecracker signalling
failures bypass shared mutation accounting.
12. SQLite creates the shared control-plane database before its mode is
restricted, then suppresses permission-repair failures. Gateway transports
also differ in whether copied deploy-key modes are preserved.
These are one design problem: state used to authorize deletion, replacement,
or resource allocation must be authoritative at the point of use.
## Goals / Success Criteria
- Cleanup never signals a PID or recursively deletes a path solely because it
appeared in a pre-confirmation snapshot.
- Firecracker cleanup proves immediately before action that a PID is still the
same Firecracker process and that a run directory is still orphaned.
- Firecracker process discovery reads NUL-delimited argv from `/proc`; paths
are never reconstructed from whitespace-delimited process listings.
- All backend cleanup discovery primitives raise a typed enumeration error on
operational failure. No backend may independently continue from a partial
snapshot.
- Shared cleanup control flow lives in the backend layer; concrete backends
override resource-specific discovery and validation primitives rather than
each implementing a bespoke failure policy.
- `list-egress-routes` returns an MCP error when attribution or policy
resolution is unavailable. A genuine, authoritatively resolved empty policy
remains a successful empty list.
- Supervisor and Git HTTP request bodies have total read deadlines, and each
service bounds concurrent request work. Limits apply to authenticated
callers because bottles themselves are untrusted.
- Gateway child-output pumping treats expected stream closure during shutdown
as completion while preserving diagnostics for unexpected failures.
- Artifact pull, existence-check, and publication requests use explicit
network deadlines.
- Persisted secrets accept only the authenticated format. The schema migration
intentionally clears legacy rows; local agents are reprovisioned rather
than retaining a ciphertext-controlled downgrade path.
- Cleanup executes only resources present in both the displayed and current
authoritative plans, attempts every approved mutation, and returns failure
when any mutation does not complete.
- Git request bodies spool to disk behind a separate heavy-work semaphore;
cleanup commands have configurable deadlines; Firecracker signalling
failures aggregate while identity-verification uncertainty still aborts.
- The shared database directory and file are private before SQLite writes any
control-plane state; an inability to enforce those modes aborts startup.
- Gateway credential directories and files receive explicit private modes
inside the gateway, independent of Docker, Apple Container, or SSH copy
semantics.
- Unit tests cover PID/path reuse, partial backend enumeration, transient
policy resolution failure, slow bodies, concurrency saturation, and stream
closure races.
## Non-goals
- Further decomposition solely to reduce module line counts.
- Replacing gateway stdlib HTTP services with a web framework.
- Changing egress matching, DLP decisions, proposal semantics, or backend
launch behavior beyond the synchronization required for safe cleanup.
- Making cleanup silently skip uncertain resources. Uncertainty is an
operator-visible failure.
## Design
### Shared backend control flow
Follow the backend architecture rule used by gateway attachment: shared
behavior lives above concrete backends; subclasses provide primitives, not
control flow.
Cleanup remains previewable, but confirmation authorizes a *new authoritative
evaluation*, not blind execution of the displayed object. The shared flow:
1. asks each available backend for a preview;
2. displays the union and asks for confirmation;
3. refreshes each non-empty backend plan;
4. validates destructive identities immediately before action;
5. aborts loudly if the refreshed plan or any identity cannot be proven safe.
Backend-specific primitives define how to identify a resource. Firecracker
uses process start identity plus canonical config/run paths; container
backends use authoritative CLI queries and stable resource names/labels.
Container engines expose destructive name-based commands without a portable
compare-and-delete operation. Cleanup therefore refreshes after confirmation
and requires every discovery query to succeed, minimizing but not claiming to
eliminate the final name-reuse race. A future engine-specific stable-ID
primitive may close that residual window without moving control flow back
into each backend.
### Enforcement state versus introspection state
Egress enforcement retains its deny-all fallback because uncertainty must not
grant network access. Supervisor introspection uses a strict resolver path:
unattributed callers and resolver failures become typed MCP errors, while a
successfully resolved policy containing zero routes returns `routes: []`.
### Gateway resource boundaries
Both stdlib servers set a per-connection body deadline before reading and use a
bounded request executor or semaphore. Saturated capacity fails quickly with a
service-unavailable response. Existing size caps remain independent:
supervisor proposals retain the 1 MiB cap and Git pack requests retain their
larger protocol-appropriate cap.
### Shutdown diagnostics
The gateway output pump catches only stream-closure exceptions expected after
the supervisor closes child pipes. Other I/O failures remain visible and are
reported through the supervisor's normal diagnostic channel.
### Shared filesystem security
The common SQLite store owns database creation for every backend. It creates
the parent directory and an empty database with private modes before opening
SQLite, repairs existing modes, verifies the resulting state, and propagates
every enforcement failure. Backend launchers do not duplicate this policy.
The backend-neutral gateway provisioner likewise applies directory and file
modes after transport copies complete. This avoids relying on copy behavior
that differs among Docker, Apple Container, and Firecracker's SSH transport.
## Implementation chunks
1. Existing fail-closed security and backend enumeration fixes.
2. FastAPI orchestrator transport and bounded control-plane bodies.
3. Egress request-policy and outbound-DLP pipeline extraction.
4. Supervisor MCP dispatch extraction.
5. Shared cleanup refresh/revalidation plus authoritative macOS discovery.
6. Strict supervisor introspection and bounded supervisor/Git HTTP work.
7. Gateway shutdown log-pump closure handling.
8. Lossless Firecracker process identities, authoritative Docker cleanup
queries, and bounded Firecracker artifact transfers.
9. Mandatory authenticated secret storage, shared cleanup-plan intersection
and mutation accounting, and contained Git backend process failures.
10. Disk-spooled and separately bounded Git bodies, cleanup command deadlines,
and classified Firecracker signalling failures.
11. Fail-closed shared database creation and backend-neutral gateway credential
permissions.
## Open questions
None.
+293 -8
View File
@@ -32,9 +32,17 @@ not a principled scope exclusion: both are major hosted sandbox platforms and
belong in this landscape even though they target platform builders rather than
bot-bottle's local single-operator workflow.
Updated 2026-07-27 after a scan of recent Show HN launches: **Black LLAB,
Eve, CloudRouter, Nucleus, yolo-cage, and Sandbox Agent SDK** added as a
dated entrant cohort. They sharpen the comparison on three axes the original
table underweighted: the browser/preview loop, parallel-agent operator UX, and
a provider-neutral automation/session API.
## Summary
The main table compares bot-bottle against fifteen isolation/sandbox tools.
The main table compares bot-bottle against fifteen canonical
isolation/sandbox tools; a later section evaluates six recent HN entrants
without widening an already unwieldy table.
Governance/pre-action authorization and credential-only layers are covered
separately because they don't provide VM or container isolation. None
duplicate bot-bottle's combination of local
@@ -542,6 +550,199 @@ them.
framework runtime is not compromised.
- **Maturity**: Specification + reference implementation, 2026.
## Recent HN entrants (added 2026-07-27)
These are grouped by launch date rather than promoted into the main table.
Several are young or sparsely documented, and putting them beside mature
runtime platforms with false precision would obscure the useful comparison.
The HN launch posts are the evidence snapshot; feature claims should be
rechecked against their repositories before relying on them for a security
decision.
### Black LLAB
- **Source**: https://github.com/isaacdear/black-llab ;
HN launch https://news.ycombinator.com/item?id=47402394
- **Isolation/locality**: Local Docker environment, with an isolated container
created for each agent task. Shared host kernel; no stronger boundary is
claimed.
- **Agent integration**: General local/cloud model workspace. Its headline is
dynamic routing of simple prompts to local models and complex prompts to
hosted models, with code execution and web scraping inside the task
container.
- **Network/credentials**: No default-deny egress, payload inspection, or
host-side credential injection documented in the launch.
- **Competitive read**: Superficial overlap ("a container per agent task"),
but not a direct security-policy competitor. Its useful challenge is the
integrated model-selection UX, which bot-bottle intentionally leaves to the
selected agent provider.
- **Maturity**: Early solo project; HN launch received 1 point.
### Eve
- **Source**: https://eve.new/ ;
HN launch https://news.ycombinator.com/item?id=47721255
- **Isolation/locality**: Managed, hosted Linux sandbox per user/session
(claimed 2 vCPU, 4 GB RAM, 10 GB disk), with filesystem, code execution,
headless Chromium, and service connectors.
- **Agent integration**: End-user OpenClaw-style agent product. An orchestrator
routes subtasks to specialist models and can run parallel subagents that
coordinate through a shared filesystem. Web UI and iMessage are primary
interaction surfaces.
- **Network/credentials**: Broad connectors are a product feature; the launch
does not document bot-bottle-style default-deny route policy, content DLP,
or credentials held outside the sandbox.
- **Competitive read**: Adjacent, not direct. Eve sells a managed colleague;
bot-bottle lets an operator run existing coding-agent CLIs under local
containment. Eve nevertheless demonstrates the appeal of background work,
live progress, browser capability, and mobile notification.
- **Maturity**: Commercial hosted product; HN launch received 71 points and
39 comments.
### CloudRouter
- **Source**: https://github.com/manaflow-ai/manaflow/tree/main/packages/cloudrouter ;
HN launch https://news.ycombinator.com/item?id=47006393
- **Isolation/locality**: Claude Code or Codex runs locally and provisions
remote cloud VMs/GPUs for execution. Project files are uploaded to the VM;
each machine exposes auth-protected VNC, VS Code, and Jupyter surfaces.
- **Agent integration**: A skill plus CLI lets the coding agent itself start,
command, inspect, and tear down machines. Browser automation is integrated,
including snapshots and screenshots. Parallel disposable compute is the
central workflow.
- **Network/credentials**: The launch emphasizes remote resource isolation and
authenticated UI endpoints, not default-deny guest egress, payload DLP, or
proxy-held application credentials.
- **Competitive read**: The closest recent workflow competitor. It directly
addresses parallel coding agents, environmental conflict, and closing the
browser/test loop, but trades local custody for elastic cloud compute.
Cloud VMs and GPUs could be a future bot-bottle backend; they do not replace
its manifest/policy layer.
- **Maturity**: Active open-source monorepo project; HN launch received
138 points and 36 comments.
### Nucleus
- **Source**: https://github.com/coproduct-opensource/nucleus ;
HN launch https://news.ycombinator.com/item?id=46855770
- **Isolation/locality**: Firecracker microVM with an enforcing MCP tool proxy.
- **Agent integration/config**: Compositional permission envelope for
read/write/run actions. The envelope is non-escalating and can tighten or
terminate, with scoped approval tokens for gated operations.
- **Network/credentials**: Default-deny egress, DNS allowlist, iptables drift
detection, time/budget caps, and hash-chained audit logging are claimed.
Remote append-only audit storage and attestation were roadmap items at
launch.
- **Competitive read**: Direct on security architecture, especially
non-escalating policy and tamper-evident audit. It is an early execution/tool
proxy rather than a provider-neutral, one-command coding-agent product. Its
tool-level action envelope is semantically finer than bot-bottle's network
boundary; bot-bottle is stronger on turnkey agent/provider integration,
credential custody, Git mediation, and long-running operator workflow.
- **Maturity**: Early OSS experiment; HN launch received 3 points.
### yolo-cage
- **Source**: https://github.com/borenstein/yolo-cage ;
HN launch https://news.ycombinator.com/item?id=46706796
- **Isolation/locality**: Local sandbox for running multiple coding agents in
YOLO mode. The launch discussion describes a VM boundary.
- **Agent integration**: Built around the native Claude Code experience and
motivated by running many agents in parallel without permission-prompt
fatigue.
- **Network/Git/credentials**: Strict egress filtering, configurable HTTP
middleware, and mediated `git`/`gh` dispatch are the main value. The launch
discussion explicitly identifies provider credential handling as unfinished
and difficult because Claude state spans multiple host paths.
- **Competitive read**: The closest new threat-model competitor. It shares
bot-bottle's premise that filesystem isolation alone is insufficient and
that Git plus authorized HTTP channels need mediation. bot-bottle currently
leads on cross-provider support, proxy-held Claude/Codex/forge credentials,
typed per-role manifests, content DLP, and supervision. yolo-cage's simpler
pitch and narrower Claude-first setup may be easier to explain.
- **Maturity**: Early local tool; HN launch received 60 points and 76 comments.
### Sandbox Agent SDK
- **Source**: https://github.com/rivet-dev/sandbox-agent ;
HN launch https://news.ycombinator.com/item?id=46795584
- **Isolation/locality**: Does not provide the isolation primitive. It runs
inside E2B, Daytona, Modal, Cloudflare Containers, Agent Computer, BoxLite,
Docker, or another sandbox provider. Embedded mode can also run locally
without a sandbox.
- **Agent integration**: Provider-neutral Rust server/SDK exposing a common
HTTP/SSE/OpenAPI interface across Claude Code, Codex, OpenCode, Cursor, Amp,
and Pi, plus a universal event/session schema for external storage and
replay. It also exposes filesystem, managed-process, terminal, MCP, skills,
custom-tool, and computer-use APIs. TypeScript is the primary SDK surface.
- **Network/credentials**: Delegated to the chosen sandbox provider.
- **Credential posture**: Its documented convenience command extracts real
OpenAI/Anthropic credentials from local agent configuration and passes them
as environment variables into the sandbox. That is materially weaker than
bot-bottle's host-side credential custody, but it is an integration choice,
not a structural limitation: a sandbox provider could put a credential
proxy underneath the same SDK.
- **Competitive read**: A serious architectural threat despite not supplying
isolation. Sandbox Agent is trying to standardize the boundary *above* the
sandbox: one client protocol, session model, and UI/control surface across
every coding agent and runtime. If that boundary becomes the ecosystem
standard, users and application builders may choose a sandbox provider plus
Sandbox Agent rather than a vertically integrated launcher. bot-bottle's
manifests would then be valuable chiefly as a local policy/backend
implementation unless they expose an equally usable control contract.
- **Maturity**: Apache 2.0, ~1.5k stars and 426 commits at the 2026-07-27
check; HN launch received 41 points.
#### Why the Sandbox Agent architecture is strategically different
The manifest and the universal control protocol solve different layers:
- A bot-bottle manifest is a **trusted launch-time policy composition**. It
selects the agent role, isolation backend, image, skills, egress routes,
credentials, Git mediation, and supervision policy. Crucially, identity and
secret references live on the host side of the trust boundary.
- Sandbox Agent is a **runtime control and observation protocol**. A remote
client creates sessions, sends messages, handles permissions, configures
skills/MCP, manipulates files/processes/desktops, and streams normalized
events. It deliberately delegates sandbox lifecycle, Git management,
storage, network policy, and credential security to other products.
That makes it complementary in a component diagram but competitive in product
architecture. The layer that becomes the stable integration point tends to own
the ecosystem. Three plausible threat paths matter:
1. **Standard control plane, interchangeable runtimes.** Applications integrate
once with Sandbox Agent and treat E2B, Daytona, BoxLite, Docker, or a future
local microVM as replaceable compute. A provider that bundles adequate
egress and credential custody makes bot-bottle's end-to-end launcher less
necessary.
2. **Policy grows upward.** Sandbox Agent already configures permissions,
skills, MCP, custom tools, filesystem/process access, and computer use. If
it adds a declarative, host-verifiable policy document, the overlap with
agent/bottle manifests becomes substantial even if enforcement remains
delegated.
3. **UI and session ownership.** Its universal transcript schema, Inspector,
React components, event replay, and remote terminal/computer APIs can become
the natural basis for desktop, web, and mobile agent managers. bot-bottle's
security layer could remain stronger while losing the operator surface and
distribution channel.
The counter-position is not to claim that manifests and an API are mutually
exclusive. The defensible split is:
- bot-bottle owns the trusted policy and enforcement plane outside the agent;
- a provider-neutral protocol owns agent process control and normalized
events; and
- the operator UI consumes both.
This suggests an explicit compatibility decision rather than parallel,
accidental protocol design: evaluate running Sandbox Agent inside a bottle and
exposing it only through the authenticated bot-bottle control plane. If its
schema is suitable, adopting it could turn a threat into an integration while
keeping manifests as the higher-trust policy source. If it is unsuitable,
bot-bottle should still publish a stable provider-neutral session/event API so
frontends do not depend on Claude/Codex/Pi-specific process behavior.
## Comparison table
*Isolation/sandbox tools only. AGT and OAP are governance layers — see their per-project notes above.*
@@ -616,6 +817,70 @@ would be a *backend* bot-bottle could call, not a competitor to its
manifest layer. endo-familiar is in a different paradigm entirely:
capability passing rather than kernel boundaries.
**Recent entrants change two parts of this read.** yolo-cage is closer to the
actual threat model than agent-safehouse or litterbox: it combines a VM-style
boundary with mediated Git and filtered HTTP specifically for parallel coding
agents. Sandbox Agent SDK is the more important strategic entrant even though
it supplies no isolation. It can become the standard agent-control layer above
all of these runtimes, including a future bot-bottle backend. CloudRouter is
the clearest workflow challenge because its browser/desktop/GPU loop makes
parallel agents visibly more capable, not merely safer.
## Gap evaluation after the 2026-07-27 entrant scan
### Material gaps
1. **A stable provider-neutral control and event protocol.** This is the
largest newly visible gap. bot-bottle normalizes launch/provisioning across
providers, but an external UI or orchestrator still lacks one documented
contract for creating a Claude/Codex/Pi session, sending input, handling
permission/supervision events, streaming normalized output, reconnecting,
and replaying history. Sandbox Agent SDK addresses exactly this layer and
is already portable across many sandbox providers.
2. **Browser/preview closure.** CloudRouter and Eve make a browser or desktop
part of the standard agent environment and expose screenshots/live viewing
to the operator. bot-bottle can run dev servers and supports nested
containers, but it does not present a first-class browser/computer-use
primitive or an auth-protected preview surface. For coding agents expected
to verify UI work, this is a real product gap.
3. **Unified parallel-session operator UX.** Named persistent bottles and
supervision provide the substrate, but the recent products make task
switching, live progress, notifications, terminal attach, diffs, and
session history the product. Security depth will not compensate for a
visibly rougher daily loop.
4. **Normalized transcript persistence and replay.** bot-bottle preserves
provider-specific state for resume; it does not expose a provider-neutral
event record suitable for audit, replay, analytics, or a web/mobile client.
This is both a UX gap and an audit gap.
### Important, but not necessarily bot-bottle features
- **Cloud VM/GPU provisioning.** Valuable for elastic workloads and could be a
backend, but it conflicts with the local-custody default and should not
displace core policy work.
- **Automatic model routing.** Black LLAB and Eve sell task-to-model routing.
bot-bottle's provider-template boundary can host that choice without making
it part of the trusted sandbox policy.
- **A thousand SaaS connectors.** This broadens capability and blast radius.
The bot-bottle-native answer should remain explicit, scoped forge/egress
associations rather than connector count as a goal.
- **SDK-driven sandbox lifecycle as the primary configuration model.** Useful
for platform builders, but not a replacement for reviewable, host-owned
manifests. A control API and a declarative policy source are compatible;
neither should silently become the other.
### Areas where bot-bottle remains ahead
- real provider and forge credentials remain outside the agent process rather
than being extracted into its environment;
- authorized HTTP payloads are scanned, not merely destination-filtered;
- Git writes traverse a distinct gate with secret scanning and host-held
upstream credentials;
- role policy is host-owned, composable, and separate from untrusted repo
content; and
- local Firecracker/Apple Container execution preserves operator custody
without requiring a hosted sandbox platform.
## Borrowable ideas
### Already shipped or otherwise addressed
@@ -642,6 +907,19 @@ capability passing rather than kernel boundaries.
### Still worth considering
- **Sandbox Agent compatibility or an equivalent stable protocol (highest
priority):** spike running its server inside a bottle behind bot-bottle's
authenticated control plane. Compare its session/event schema, permission
model, restore semantics, and provider coverage with current provider
adapters. Adopt compatibility if it preserves the host-owned trust boundary;
otherwise specify bot-bottle's own stable API before building another UI.
- **First-class browser/preview loop** (from CloudRouter and Eve): give a
bottle an optional browser/computer-use capability plus an operator-visible,
authenticated preview/screenshot surface. Treat its network access as part
of the bottle policy, not an implicit bypass.
- **Provider-neutral transcript/event persistence** (from Sandbox Agent SDK):
retain enough normalized structure for replay and audit while preserving the
provider-native state needed for exact resume.
- **Live network activity in the supervisor TUI** (from Docker sbx): show
allowed and blocked connections and let the operator propose policy changes
from the existing supervision surface.
@@ -652,10 +930,11 @@ capability passing rather than kernel boundaries.
closer review. This needs a carefully specified trust model before it can be
more than a heuristic.
Not worth borrowing: the SDK-first programmatic API style of boxlite /
microsandbox (cuts against the declarative-manifest stance), and the
hosted-SaaS dashboard model of tilde.run (cuts against the
"infrastructure I control" goal).
Not worth borrowing: SDK-first *policy configuration* as used by boxlite /
microsandbox (cuts against the reviewable declarative-manifest stance), and
the hosted-SaaS custody model of tilde.run (cuts against the "infrastructure I
control" goal). A provider-neutral runtime-control API is a separate concern
and is worth borrowing.
## Publishing and positioning verdict
@@ -679,9 +958,15 @@ bot-bottle remains unusual in combining:
The practical wedge is “as easy as native yolo, with declarative role policy
and self-hosted custody,” including scoped access to private LAN/Tailnet
services that cloud-first runtimes cannot provide without additional network
plumbing. The main competitive risks are a local wrapper such as claudebox or
Docker sbx growing a role-manifest layer, and GUI products such as SuperHQ
adding equivalent policy and audit depth.
plumbing. The main competitive risks are now:
- a local wrapper such as yolo-cage, claudebox, or Docker sbx growing a
role-manifest and credential-custody layer;
- Sandbox Agent SDK becoming the standard control/session boundary and making
the runtime beneath it interchangeable; and
- GUI products such as SuperHQ or CloudRouter adding equivalent policy and
audit depth before bot-bottle closes the browser/preview and
parallel-session UX gaps.
## Caveats
+6 -6
View File
@@ -6,7 +6,7 @@ Can bot-bottle grow a built-in supervisor — TUI inventory plus PR-feedback rou
## Context
bot-bottle today is a fleet *executor*: `./cli.py start <agent>` brings up one bottle (agent container + pipelock + optional git-gate + optional cred-proxy on a per-bottle internal network), and `cli.py` tears it down when the session ends. There is no inventory view, no idle-detection, no automated reaction to PR or CI events. In parallel use, a human is the supervisor — opening one terminal per bottle, switching between them, and watching upstream PR state by hand.
bot-bottle today is a fleet *executor*: `bot-bottle start <agent>` brings up one bottle (agent container + pipelock + optional git-gate + optional cred-proxy on a per-bottle internal network), and `cli.py` tears it down when the session ends. There is no inventory view, no idle-detection, no automated reaction to PR or CI events. In parallel use, a human is the supervisor — opening one terminal per bottle, switching between them, and watching upstream PR state by hand.
A separate survey of the broader ecosystem ([agent control dashboards research, mid-2026](https://gitea.dideric.is/didericis/consilium-research/src/branch/main/developer-workflow/agent-control-dashboards-2026-05-24.md)) sorts dashboards into five tiers (session managers, parallel runners, Kanban boards, mission-control SPAs, observability backends). The earlier first-pass conclusion was that a full SPA tier conflicts with bot-bottle's isolation model. This doc reconsiders the smaller question: a TUI supervisor in the existing Python CLI.
@@ -25,13 +25,13 @@ A supervisor doesn't have to be heavy. A TUI built into the existing Python CLI,
Three layers, each independently useful, in order of ambition:
### 1. `./cli.py status` — read-only inventory
### 1. `bot-bottle status` — read-only inventory
Reads `docker ps` filtered by a bottle label and tails each bottle's session log. Reports per bottle: name, agent, uptime, last-activity timestamp, token spend if available, associated PR/branch if recorded.
No new daemons. No new ports. No new credentials. ~100 lines.
### 2. `./cli.py watch` — TUI over the same data
### 2. `bot-bottle watch` — TUI over the same data
Same data as `status`, rendered with auto-refresh and keyboard shortcuts that shell out to the existing `cli.py attach / stop / start` commands.
@@ -39,7 +39,7 @@ Library choice: prefer the stdlib `curses` module to stay stdlib-first; fall bac
This is the Claude Squad / tmux-agent-status pattern, applied to bottles instead of tmux sessions. The whole category exists *because* a TUI is the lightweight shape that doesn't require what the SPA tier requires.
### 3. `./cli.py supervise` — PR feedback router
### 3. `bot-bottle supervise` — PR feedback router
The optional, more ambitious layer. The bottle manifest gains an optional field:
@@ -49,7 +49,7 @@ pr_watch:
branch: agent/task-42
```
`./cli.py supervise` polls the named upstream for new review comments and CI failures on `branch`. When one fires, it surfaces as a desktop notification or a flash in the TUI. The human decides what to do with the feedback — there is no autonomous loop that feeds the comment back into a bottle's next prompt (see "Where to be conservative" for why).
`bot-bottle supervise` polls the named upstream for new review comments and CI failures on `branch`. When one fires, it surfaces as a desktop notification or a flash in the TUI. The human decides what to do with the feedback — there is no autonomous loop that feeds the comment back into a bottle's next prompt (see "Where to be conservative" for why).
The polling token is a **host** token (the same `GH_PAT` / Gitea token the host already keeps in shell env), not a bottle credential. The supervisor never holds bottle secrets.
@@ -62,7 +62,7 @@ The load-bearing question is whether the supervisor introduces the privileged-ch
| Reaching into running bottles | Supervisor reads `docker ps` and host-side log files. The host already sees both — Docker is the trust boundary, the supervisor is on the host side of it. |
| Holding bottle credentials | The polling token is a host token. The supervisor never receives `bottle.cred_proxy.routes` entries; it has no path to them. |
| Bridging between bottles | The supervisor does not relay state from bottle A to bottle B. It relays *upstream PR state* to a bottle's next prompt — and only if the manifest opts in. |
| New attack surface | All "control" actions go through `./cli.py start <agent>`, which already enforces the manifest. The supervisor is an automated caller of the existing CLI, not a parallel control plane. |
| New attack surface | All "control" actions go through `bot-bottle start <agent>`, which already enforces the manifest. The supervisor is an automated caller of the existing CLI, not a parallel control plane. |
The boundary stays at the bottle wall. The supervisor looks outward at git/PR state and downward at Docker; it does not look *inward* through pipelock.
@@ -13,11 +13,11 @@ What's the cheapest path to that, and where does it bottom out?
## What "interact" means
Today the flow is bimodal. `./cli.py start <agent>` brings the
Today the flow is bimodal. `bot-bottle start <agent>` brings the
bottle up and immediately drops you into an interactive
`docker exec -it bot-bottle-<slug> claude ...` — claude-code
owns the whole terminal until you Ctrl-D out, at which point the
bottle tears down. The dashboard (`./cli.py dashboard`) is a
bottle tears down. The dashboard (`bot-bottle dashboard`) is a
*separate* invocation that watches across bottles but never
exposes the claude TUI itself.
@@ -107,7 +107,7 @@ What's not good:
This is the v1 the project's existing code-shape strongly
prefers. It clears the bar of "let me talk to claude-code
without quitting `./cli.py dashboard`."
without quitting `bot-bottle dashboard`."
## Option 2: Embedded emulator
+2 -2
View File
@@ -84,7 +84,7 @@ dangerous changes before they left*.
(`supervise_gitleaks_allow`, [`git_gate_render.py`](../../bot_bottle/git_gate_render.py)).
Extend the same flow to **high-risk file classes**: any commit touching
CI/build/deploy scripts, auth/crypto code, egress config, or
adding/changing dependencies → route to `./cli.py supervise`. This is
adding/changing dependencies → route to `bot-bottle supervise`. This is
attribution/policy, not detection, and it's the strongest thing here —
a human on exactly the temporal-escape surfaces.
4. **LLM semantic diff-review — the behavioral backstop.** The only
@@ -146,7 +146,7 @@ preserves the bottom-up distribution funnel.
**governed code-egress review**, not "we resell inference" (the
monetization notes explicitly warn against reselling compute).
- **The web-console supervise/review flow — the strongest anchor.** Turn
the CLI `./cli.py supervise` approval into a real review surface:
the CLI `bot-bottle supervise` approval into a real review surface:
rendered diff + finding context, approve/reject, **who-approved audit
trail, RBAC on approvers, mobile/phone-control** (ties to the
dashboard/vault north star). This is "central enforcement +
@@ -100,7 +100,7 @@ resolver globs each directory.
becomes file ops (mkdir, mv, rm) instead of editing one file.
Power users prefer that; new users may not.
- Discovery requires `ls`, not "grep one file." Tooling helps
(e.g. `./cli.py list`) but the manifest is no longer a single
(e.g. `bot-bottle list`) but the manifest is no longer a single
artifact to email or ship.
- Atomicity: swapping a bottle name across agents touches
multiple files. Git handles this fine; a one-shot text editor
@@ -364,7 +364,7 @@ wins; none of the body-prose or dependency story.
warns / ignores / breaks. If it warns, we'd want a different
field name (e.g. `bot-bottle-bottle`) or a namespaced block.
- **Migration story.** Is the project willing to ship a one-shot
`./cli.py migrate-manifest` command that does the JSON → MD
`bot-bottle migrate-manifest` command that does the JSON → MD
conversion? Or do users just rewrite by hand from the new docs?
- **Bottle file body content.** If most bottle .md files have an
empty body, is the MD-with-frontmatter format still warranted?
+1 -1
View File
@@ -34,7 +34,7 @@ on top of working onboarding.
A first-time user today goes through five steps: install Docker,
install `uv`, set `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN`, write
`bot-bottle.json`, run `./cli.py start`. One of those is
`bot-bottle.json`, run `bot-bottle start`. One of those is
"author a JSON manifest." Polished tools in this category let
users skip that step on day one. The fix is an `init` subcommand
that drops a working `bot-bottle.json` with a default `coder`
+1 -1
View File
@@ -131,7 +131,7 @@ The minimum-viable workflow, no bot-bottle code changes:
3. SSH in.
4. `git clone` bot-bottle on the VM, drop a manifest in place,
inject `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` via the provider's secrets path.
5. `./cli.py start <agent>` — the existing launcher handles the rest.
5. `bot-bottle start <agent>` — the existing launcher handles the rest.
6. On exit: destroy the VM. No host artifacts persist.
For the "VPN pivot" failure mode, see
@@ -0,0 +1,536 @@
# Sandbox Agent SDK and bot-bottle: protocol versus product
This note asks whether [Sandbox Agent SDK](https://github.com/rivet-dev/sandbox-agent)
and bot-bottle compete for the same architectural layer, whether bot-bottle
can productize the turnkey ecosystem/DX layer above it, and how far the
Docker/OCI analogy actually holds.
Research conducted 2026-07-27. Sandbox Agent SDK was at the `0.4.x` line,
Apache 2.0, and documented support for Claude Code, Codex, OpenCode, Cursor,
Amp, and Pi at the time of review.
## Summary
**The projects are complementary at the component boundary and competitive at
the product boundary.** Sandbox Agent SDK normalizes how software controls a
coding-agent process inside an arbitrary sandbox. bot-bottle decides what
sandbox to create, what trusted role and policy it receives, how credentials
and Git access cross the boundary, how traffic is constrained, and how an
operator launches and supervises the result.
The Docker analogy is useful with one correction:
- Sandbox Agent SDK is not equivalent to Linux container APIs or OCI itself.
It is closer to a **containerd shim plus a portable exec/session API for
coding agents**. It adapts incompatible agent processes to one HTTP/SSE
contract.
- A future independent agent-session specification would be the closer OCI
analogue.
- bot-bottle can credibly occupy the **Docker Engine / Compose / Desktop**
layer: packaging, policy composition, lifecycle, networking, credentials,
storage, operator UX, and a one-command experience above interchangeable
agent adapters and isolation runtimes.
That is a viable position, but “turnkey wrapper” undersells it. A thin wrapper
is replaceable. The valuable product is a **turnkey, policy-first coding-agent
runtime** whose manifest compiles trusted operator intent into multiple
enforcement planes. Sandbox Agent SDK may be one internal process-control
component of that product.
The recommended direction is:
1. Keep the bot-bottle manifest as the host-owned source of trusted policy.
2. Spike Sandbox Agent SDK as the in-bottle provider/session adapter.
3. Expose a stable, provider-neutral bot-bottle control API, compatible with
Sandbox Agent where practical.
4. Keep security decisions and authoritative audit outside the sandbox.
5. Build the ecosystem around policy packs, agent images, skills, backends,
operator UI, and trusted integrations—not around a proprietary transcript
protocol.
## What each project is today
### Sandbox Agent SDK
Sandbox Agent is a Rust server that runs alongside the coding agent. A client
connects over HTTP, streams events over SSE, and uses one API across agent
implementations. Its documented surface includes:
- creating and restoring agent sessions;
- sending messages and streaming normalized events;
- handling permissions;
- configuring MCP servers, skills, and custom tools;
- filesystem and managed-process APIs;
- interactive terminal access;
- computer-use/desktop operations;
- a universal session/transcript schema;
- an Inspector UI, React components, CLI, TypeScript SDK, and OpenAPI spec.
It can run in embedded mode or inside E2B, Daytona, Modal, Cloudflare
Containers, Agent Computer, BoxLite, Docker, and other environments. It
explicitly leaves these concerns to the caller or sandbox provider:
- sandbox creation and lifecycle;
- Git repository management;
- durable session storage;
- network policy;
- isolation strength; and
- secure credential delivery.
Its documented credential convenience path extracts real provider credentials
from local agent configuration and passes them into the sandbox environment.
That is convenient but is not an acceptable security boundary for bot-bottle.
Sources:
- [Sandbox Agent repository and architecture](https://github.com/rivet-dev/sandbox-agent)
- [Sandbox Agent documentation](https://sandboxagent.dev/docs)
- [HTTP API](https://sandboxagent.dev/docs/api-reference)
- [Universal session/transcript schema](https://sandboxagent.dev/docs/session-transcript-schema)
### bot-bottle
bot-bottle is a host-side launch, policy, and enforcement system for existing
coding-agent CLIs. Its current architecture includes:
- agent and bottle manifests with composition via `extends:`;
- a host-only trust boundary for roles, identity, and secret references;
- provider templates and plugins for Claude Code, Codex, Pi, and custom
providers;
- Firecracker on KVM Linux and Apple Container on macOS, with Docker fallback;
- image construction and provider-specific provisioning;
- default-deny inspected egress with path/method/header policy;
- payload DLP on authorized channels;
- real credentials held outside the agent and injected by the gateway;
- Git mediation, upstream credential custody, and gitleaks scanning;
- a per-host authenticated orchestrator and shared gateway;
- named bottle lifecycle, resume, supervision, and audit state; and
- a CLI/TUI intended to make full-permission agents operationally tolerable.
The provider layer currently normalizes launch-time concerns—command, image,
prompt delivery, files, skills, environment, verification, and provider-owned
egress routes. It does **not** yet expose a stable provider-neutral runtime
contract for sessions, messages, transcripts, terminals, or normalized events.
That is the gap Sandbox Agent directly illuminates.
Sources in this repository:
- [`README.md`](../../README.md)
- [`0070-per-host-orchestrator.md`](../prds/0070-per-host-orchestrator.md)
- [`0026-agent-provider-templates.md`](../prds/0026-agent-provider-templates.md)
- [`0053-user-provider-plugins.md`](../prds/0053-user-provider-plugins.md)
- [`agent_provider.py`](../../bot_bottle/agent_provider.py)
## The layer model
The cleanest architecture has four layers:
| Layer | Responsibility | Likely owner |
|---|---|---|
| Operator product | Install, select a role, launch, observe, intervene, resume, review changes | bot-bottle |
| Trusted policy and lifecycle | Compose manifest, choose backend/image, hold credentials, enforce egress/Git, persist authoritative audit | bot-bottle |
| Agent control protocol | Start provider process, create session, send input, stream normalized events, terminal/computer operations | Sandbox Agent or a compatible protocol |
| Isolation primitive | VM/container/process boundary, filesystem, CPU/memory, networking substrate | Firecracker, Apple Container, Docker, E2B, Daytona, BoxLite, etc. |
The important boundary is between trusted policy/lifecycle and agent control.
The agent-control daemon runs in the environment being treated as untrusted.
It can report what the agent says happened, but it cannot authoritatively prove
that policy was enforced. Egress decisions, credential custody, Git scanning,
bottle identity, and security audit must remain outside it.
### Proposed composition
```text
operator UI / CLI / API
|
v
bot-bottle orchestrator (trusted)
- resolves manifest
- owns bottle identity and lifecycle
- stores authoritative audit
- authenticates clients
|
+--------------------------+
| |
v v
isolation backend shared gateway (trusted)
Firecracker / Apple / Docker - egress policy + DLP
| - credential injection
| - Git mediation
v
bottle / guest (untrusted)
- Sandbox Agent server
- Claude Code / Codex / Pi subprocess
- workspace, skills, MCP configuration
```
The bot-bottle manifest would compile into both sides:
- **outside the bottle:** backend, network, egress, credentials, Git,
supervision, identity, and authoritative lifecycle;
- **inside the bottle:** selected provider, prompt, skills, MCP configuration,
startup arguments, and non-secret session metadata.
Sandbox Agent should never receive real secrets merely because its API offers
a credential extraction helper. Provider and forge requests should continue
to use bot-bottle's placeholder/proxy pattern.
## How accurate is the Docker/OCI analogy?
### The useful part
The container ecosystem separates low-level execution from a product that
ordinary developers operate. OCI defines interoperable image, runtime, and
distribution specifications. Docker Engine adds a daemon, API, CLI, object
model, images, networks, volumes, and lifecycle; Docker Desktop and related
products add installation, updates, UI, integrations, policy, and team
workflows.
The same separation can exist for coding agents:
| Container ecosystem | Agent-sandbox ecosystem |
|---|---|
| OCI/runtime contract | A future open agent session/event contract |
| `runc` / runtime adapter | Claude/Codex/Pi adapter |
| containerd shim and task/exec API | Sandbox Agent server and HTTP/SSE session API |
| containerd / CRI-style lifecycle | Sandbox-provider lifecycle APIs |
| Docker Engine / Compose | bot-bottle orchestrator + manifests + backends + gateway |
| Docker Desktop / Hub ecosystem | bot-bottle desktop/mobile UX, policy packs, agent images, skills, trusted integrations |
Sandbox Agent makes coding-agent processes portable in roughly the way a shim
makes runtimes consumable through a common lifecycle interface. bot-bottle can
make the entire safe-agent system usable without asking the operator to
assemble that plumbing.
Official container references:
- [Open Container Initiative](https://opencontainers.org/)
- [OCI Runtime Specification](https://github.com/opencontainers/runtime-spec)
- [Docker Engine architecture](https://docs.docker.com/engine/)
- [Docker alternative runtimes and containerd shims](https://docs.docker.com/engine/daemon/alternative-runtimes/)
### Where the analogy breaks
1. **Sandbox Agent is an implementation, not an independent standard.**
Its OpenAPI document is public, but the project currently owns the server,
adapters, schema, and evolution. OCI is an independently governed set of
specifications with multiple implementations.
2. **It sits above, not below, the isolation boundary.** Linux namespaces,
cgroups, VMs, and OCI runtimes create the boundary. Sandbox Agent controls a
process after some other system has created that boundary.
3. **It reaches into product territory.** Inspector, React components,
computer-use APIs, skills/MCP configuration, transcripts, and restoration
are not merely low-level primitives. Sandbox Agent can continue growing
upward into the same UI and orchestration space bot-bottle might occupy.
4. **Coding agents are semantically uneven.** Normalizing a container
lifecycle is easier than claiming full behavioral parity across Claude
Code, Codex, Cursor, Amp, OpenCode, and Pi. A universal schema can become a
lowest common denominator or accumulate provider-specific escape hatches.
5. **The security contract is not standardized.** An agent-session API says
little about whether credentials are visible, egress is controlled, Git is
mediated, or audit is trustworthy. Those are core bot-bottle concerns.
The positioning should therefore say “Docker-like product layer above an open
agent-control protocol,” not “Sandbox Agent is OCI” or “bot-bottle implements
OCI for agents.”
## Can bot-bottle be the turnkey product layer?
Yes, if it owns substantially more than launch syntax.
The turnkey promise is:
> Choose a trusted role, point it at a project, and run any supported coding
> agent with full permissions. bot-bottle builds the environment, isolates it,
> supplies only the capabilities it needs, keeps credentials outside, mediates
> external writes, and gives the operator one place to watch and intervene.
That product has several defensible jobs:
### 1. Packaging and reproducibility
- provider and toolchain images;
- pinned, verified build inputs;
- skills and MCP configuration;
- role/bottle composition;
- cached startup and portable environment definitions; and
- compatibility testing across agents and backends.
### 2. Trusted policy compilation
The manifest is valuable because one reviewable document compiles into:
- an isolation plan;
- gateway routes and DLP policy;
- credential slots;
- Git-gate repositories and identities;
- provider configuration;
- supervision behavior; and
- operator-facing preflight.
Sandbox Agent's runtime configuration does not replace this. The policy must
be resolved before an untrusted guest or agent-control daemon exists.
### 3. Security enforcement
- dedicated-kernel isolation where available;
- no direct guest route to the internet;
- credentials injected outside the agent;
- content inspection on allowed destinations;
- Git secrets scanning and upstream-key custody;
- fail-closed policy resolution; and
- authoritative host-side audit.
This is the strongest current differentiation from a generic
“Sandbox Agent + Docker/E2B” assembly.
### 4. Lifecycle and operations
- install and host preflight;
- image build/update;
- start, stop, resume, cleanup, and migration;
- concurrent named agents;
- state recovery after crashes;
- live supervision and policy remediation; and
- backend selection without changing the role definition.
### 5. Ecosystem and DX
A product layer can support:
- curated provider images;
- signed policy/bottle packs;
- reusable role templates;
- skills and MCP bundles;
- backend plugins;
- an authenticated desktop/web/mobile operator client;
- browser/preview integration;
- normalized transcripts and change review; and
- team policy distribution and compliance exports.
The analogy to Docker is strongest here: users adopt the coherent workflow and
ecosystem, not because the low-level process API is proprietary.
## Business and product positioning
“Turnkey wrapper” is understandable internally but weak externally. It implies
that the hard work lives underneath and that another wrapper can replace it.
Prefer one of:
- **The policy-first runtime for coding agents**
- **Run any coding agent with full permissions, without giving it your host or
credentials**
- **A turnkey local control plane for isolated coding agents**
- **Docker-like packaging and operations for coding agents, with the security
boundary outside the agent**
The open/product split could resemble the container ecosystem:
### Open foundation
- manifest schema and composition;
- local CLI and core orchestrator;
- provider adapters;
- Firecracker/Apple Container/Docker backends;
- gateway policy format and enforcement;
- Sandbox Agent compatibility;
- local audit and supervision; and
- conformance tests for providers/backends/policy.
### Productizable ecosystem/DX
- polished desktop and mobile clients;
- fleet/remote-host management;
- signed and curated role/image/policy registry;
- team policy distribution and administrative controls;
- durable searchable transcripts and audit exports;
- SSO, RBAC, retention, and tamper-evident audit;
- managed update/compatibility channels;
- remote browser/preview relay;
- enterprise support; and
- optional managed build/cache infrastructure.
OCI itself is not the thing Docker sells. Interoperability expands the market;
the product captures value through reliable packaging, workflow, distribution,
management, and trust. bot-bottle should follow that logic rather than trying
to make its session protocol the moat.
## Strategic threat from Sandbox Agent
Sandbox Agent is a real threat for three reasons:
1. **It can become the integration default.** A frontend or agent platform can
integrate one API and choose among many agents and sandbox vendors.
2. **It can own session data and UI.** The universal event schema, Inspector,
React components, restoration, terminal, and computer-use APIs give it a
natural path toward the operator surface.
3. **Sandbox providers can move upward.** If E2B, Daytona, BoxLite, or another
runtime combines Sandbox Agent with adequate network policy and credential
custody, it can offer much of the turnkey stack.
The threat is not that its manifest syntax is better. It currently has no
equivalent trusted policy composition. The threat is that **the ecosystem may
standardize around its API before bot-bottle has a stable external control
surface**. In that world bot-bottle is evaluated as one sandbox provider,
while the SDK and its consumers own the user relationship.
## Why bot-bottle can still win its layer
Sandbox Agent's scope exclusions align with bot-bottle's deepest work:
- it does not choose or operate the sandbox provider;
- it does not mediate Git;
- it does not own network policy;
- it does not securely deliver credentials;
- it does not durably store sessions; and
- it cannot make guest-generated telemetry authoritative.
Those are not incidental features. Together they define the trusted system
around an untrusted coding agent. bot-bottle also has a narrower and coherent
initial customer: a developer or small operator who wants existing agent CLIs
to run locally with broad permissions and bounded consequences.
The durable advantage is therefore:
> Sandbox Agent makes agents controllable. bot-bottle makes them safe and
> operable.
That sentence remains true only if bot-bottle closes its operator-DX gaps.
Security without a browser/preview loop, stable API, normalized session view,
and good parallel-task UX risks becoming an invisible backend feature.
## Integration options
### Option A — Embed Sandbox Agent inside each bottle
bot-bottle launches Sandbox Agent as the provider process supervisor and
connects it to the host orchestrator through a bottle-scoped authenticated
channel.
**Benefits**
- immediate provider-neutral session API;
- more supported agents;
- normalized streaming and transcripts;
- terminal, filesystem, process, and computer-use primitives;
- Inspector/React ecosystem; and
- less provider-specific reverse engineering in bot-bottle.
**Risks**
- `0.x` API/schema churn;
- extra binary and release-supply-chain dependency;
- lowest-common-denominator normalization;
- conflict with provider-native resume state;
- an in-guest daemon is attacker-controlled after guest compromise;
- duplicate orchestration responsibilities; and
- upstream can move into policy/lifecycle and compete more directly.
**Security rule**
Treat every event and state claim from Sandbox Agent as untrusted telemetry.
Never delegate egress authorization, credential release, bottle identity,
authoritative audit, or Git policy to it.
### Option B — Implement a Sandbox Agent-compatible endpoint
bot-bottle maps the external protocol onto its existing provider adapters and
process model without running the upstream server.
**Benefits**
- ecosystem compatibility with tighter component control;
- no in-guest daemon dependency; and
- room to preserve bot-bottle-native lifecycle semantics.
**Risks**
- large and continuing compatibility burden;
- “full feature coverage” is expensive across all providers;
- accidental protocol fork; and
- effort diverted from policy and UX differentiation.
### Option C — Define an independent bot-bottle session API
Build only the control surface bot-bottle needs.
**Benefits**
- clean fit with the trust model and persistent named bottles;
- no upstream dependency; and
- deliberate support for supervision and security events.
**Risks**
- recreates a fast-growing open-source project;
- no existing client ecosystem;
- slower browser/desktop/mobile work; and
- increases the chance that Sandbox Agent becomes the de facto standard first.
### Recommendation
Start with **Option A as a bounded compatibility spike**, not a product
commitment. Do not begin with a clean-room competing protocol.
The spike should answer:
1. Can Claude Code, Codex, and Pi retain exact native resume behavior?
2. Can Sandbox Agent run without receiving real provider credentials?
3. Can its server be reached through a bottle-scoped authenticated channel
without exposing the orchestrator or broadening guest egress?
4. Which permission events overlap or conflict with bot-bottle supervision?
5. Can normalized events be stored while clearly separating untrusted
transcript telemetry from authoritative gateway/Git audit?
6. Can manifest skills, MCP servers, prompt, and startup arguments compile
deterministically into its configuration?
7. Does its versioning policy permit a compatibility contract bot-bottle can
support?
8. What image-size, startup-time, and update burden does the binary add?
If the answers are favorable, adopt it behind a bot-bottle-owned interface and
pin/test the supported version. If not, implement the smallest compatible
subset needed by external clients before inventing a wholly separate API.
## Product roadmap implications
The competitor scan and this architecture comparison reorder the likely work:
1. **Provider-neutral control/session compatibility spike**
2. **Stable authenticated external bot-bottle API**
3. **Normalized transcript/event persistence**
4. **Parallel-session operator UI**
5. **Browser/preview/computer-use capability**
6. **Policy/image/skill distribution and signing**
7. **Remote host/fleet management**
This does not mean pausing security work. It means exposing the shipped
security work through a product surface that can compete with the SDK-plus-
sandbox ecosystem.
## Decision
Treat Sandbox Agent SDK as a potentially standard **agent process-control
layer**, not as a sandbox replacement and not as a minor complementary
library. Position bot-bottle one layer above it:
- manifests express trusted role and environment policy;
- bot-bottle compiles and enforces that policy across host, gateway, Git, and
isolation backends;
- Sandbox Agent or a compatible protocol controls the selected agent process;
and
- bot-bottle owns the turnkey operator experience.
The Docker analogy is strategically sound when stated as:
> Sandbox Agent can be the portable task/exec protocol; bot-bottle can be the
> opinionated engine, Compose-like policy layer, and Desktop-like operator
> product.
It is not sound when stated as:
> Sandbox Agent is OCI and bot-bottle is Docker.
There is no independent OCI-equivalent agent specification yet, and Sandbox
Agent already reaches into UI/session territory. Compatibility should be
pursued quickly, while the trusted manifest/enforcement plane and operator
experience remain the parts bot-bottle deliberately owns.
@@ -0,0 +1,183 @@
# Testing a clean bot-bottle install on Linux
How do you exercise `install.sh` the way a brand-new user would — on a
pristine Linux environment you can throw away afterward — *without*
polluting your daily-driver host, and across the several package-management
regimes Linux fragments into? This is the Linux counterpart to
[`testing-clean-install-on-macos.md`](testing-clean-install-on-macos.md);
the conclusion is different because Linux gives us a boundary macOS doesn't.
## Summary
On macOS the honest options were a throwaway user or a VM, and the throwaway
user won on pragmatics (nested virtualization is gated to M3+). On Linux the
calculus flips: a **disposable KVM virtual machine, booted from a distro
cloud image and deleted per run, is both the cleanest boundary and the one
that lets a single harness cover Ubuntu, Fedora, Arch, Alpine, and NixOS**.
The host already requires KVM for the Firecracker backend, so the VM is cheap
here.
The harness lives at [`scripts/linux-install-test.sh`](../../scripts/linux-install-test.sh).
Per run it caches one read-only base image, boots a throwaway copy-on-write
overlay (`qemu-img create -b base`), installs the distro's prerequisites,
pipes *this checkout's* `install.sh` into the guest exactly as `curl … | sh`
would, asserts the CLI installed, and deletes the overlay — the Linux
equivalent of `docker run --rm`, for a whole machine.
## Why a VM, not a container or a throwaway user
| Mechanism | Why it's the wrong boundary here |
|---|---|
| **Container** (`docker run --rm`) | Shares the host kernel and ships a deliberately minimal userland — no systemd, a stubbed-out package manager story, and (crucially) it doesn't reproduce the *externally-managed Python* (PEP 668) that real desktop/server installs put in front of the user. It tests "does install.sh run in a container," not "does it run on a real distro." |
| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
## Two variants: `test` (bare host) and `test-ready` (prepared host)
`install.sh` never installs a backend, and never installs its own toolchain
prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
runs `doctor`, which *reports* what's missing
([`install.sh`](../../install.sh) header,
[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
That leaves two distinct things worth testing, split into two subcommands that
mirror the macOS harness's `test` / `test-ready` convention (there the split is
the backend service; here it is the toolchain the installer needs):
- **`test`** — `install.sh` runs on the **bare cloud image**, prerequisites and
all left as the vendor ships them. This exercises `install.sh`'s own
prerequisite-guard logic — the entire first half of the script (python
version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
- **`test-ready`** — the harness installs python3 + git + pipx first (the
`prereqs` step), then runs `install.sh`. This is the *prepared-host happy
path*: does a clean install actually land and produce a working CLI?
**Pass criteria differ by variant:**
| Variant | PASS when |
|---|---|
| `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
| `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
Neither variant requires a green `doctor`: inside the VM there is no nested KVM
or Docker, so **the backend is correctly reported not-ready** — install.sh does
not install a backend and cannot regress one, and this harness does not
provision the Docker backend. This is where Linux necessarily diverges from the
macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
verdict instead classifies `doctor`'s output the way the macOS harness does — a
`Traceback` is an install defect (fail), a missing `python`/`config` line is a
fail, a not-ready backend is reported — so a genuine installer regression (a
broken shim, an import error, a botched PATH) stays visible.
`test-all` runs the full matrix — every distro × both variants — each cell in
its own throwaway VM, and prints a per-cell PASS/FAIL summary.
## The distro matrix is the point
Each distro exercises a different corner of the installer:
| Distro | Cloud image | What it stresses |
|---|---|---|
| **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. |
| **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. |
| **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. |
| **Alpine** | Alpine `nocloud_` (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
| **NixOS** | locally built with `nixos-generators` | No FHS `~/.local` on PATH by default; `nix profile install` prereqs; pipx laying a self-contained venv on a non-FHS host. |
### Validation run (2026-07-27) — full green
Full matrix on the delphi KVM host, QEMU 11.0.2, both variants × all five
distros passing:
| Distro | `test` (bare) | `test-ready` (prepared) |
|---|---|---|
| Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green |
| Fedora 44 | ✅ declines at git gate | ✅ installs |
| Arch (latest) | ✅ declines at git gate | ✅ installs |
| Alpine 3.21 | ✅ declines at git gate | ✅ installs |
| NixOS 24.11 | ✅ declines (no python3) | ✅ installs |
The bare `test` sees `install.sh` decline soundly — exit 1 at the
git-for-git-specs gate on the Debian/Fedora/Arch/Alpine images (they ship
python3 but not git), and at the python3 gate on NixOS (no python3 on PATH) —
and `test-ready` installs cleanly with `doctor` reporting a usable python and
config (backends all not-ready, as expected in a plain VM).
Getting to green surfaced and fixed a series of real defects:
- **Fedora 41 was EOL/404** → bumped to 44.
- The liveness probe used `bot-bottle --version`, which the CLI does not
implement (unknown args die non-zero), so every *successful* install was
misreported as failed → switched to `bot-bottle --help`.
- **Alpine** needed three fixes: the `generic_` image ignores a NoCloud seed
(switched to the `nocloud_` variant); OpenRC does not auto-start sshd after
cloud-init injects the key (start it via `runcmd`); and Alpine's non-PAM
sshd refuses pubkey auth for a cloud-init-*locked* account (give it a
throwaway password). It also has no `sudo` by default (install it via
cloud-init `packages:`).
- **NixOS** publishes no downloadable cloud qcow2 (its cloud images are
Hydra-built AMIs), so the harness builds one with `nixos-generators`
([`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)):
cloud-init for the key, flakes enabled, deliberately no python/git/pipx. The
`test-ready` prereq install pins `nixpkgs/nixos-24.11` because the guest's
default unstable registry builds pipx from source (and its test suite
currently fails to build).
- Two harness-hygiene bugs also fixed: `cmd_down` left `serial.log` behind
(orphaned run dirs), and the teardown trap was armed after `cmd_up`, leaking
a VM when `wait_for_ssh` timed out.
In `test-ready` the harness installs `python3 + git + pipx` first on each
distro (install.sh installs none of them), so all five drive the recommended
pipx path. `test` then removes that scaffolding and lets each distro's bare
image collide with install.sh's guards — on most cloud images python3 is
present (cloud-init needs it) but git and pipx are not, so install.sh is
expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
an actionable message. Both are legitimate, and the two variants together cover
the whole first half of the installer as well as the happy path.
## What a clean install touches (the footprint that decides "wipeable")
| Artifact | Location | In the guest's `$HOME`? | Survives VM teardown? |
|---|---|---|---|
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
| pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix profile` | ❌ | ❌ **overlay deleted** |
Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
footprint *survives*), **every row here dies with the overlay** — that is the
VM's whole advantage. The cached base image is read-only backing and is the
only thing that persists between runs, on purpose.
## Design notes baked into the harness
- **User-mode networking** (`-netdev user,hostfwd=tcp:127.0.0.1:PORT-:22`):
no root, no bridge, no host network state touched. Only SSH is forwarded.
- **cloud-init seed ISO** (`cloud-localds`) injects an ephemeral SSH keypair
and a passwordless-sudo login. The keypair is generated per run and deleted
on teardown; the guest can't be logged into after it's gone.
- **Copy-on-write overlay**: the cached base is never mutated, so a corrupt or
interrupted run can't poison the cache; downloads land at `*.partial` and
are renamed only after checksum verification.
- **Checksums**: verified against each vendor's published sums file at
download time (GNU `hash file`, bare-hash, and Fedora's BSD
`SHA256 (file) = hash` formats are all handled). Alpine ships `.sha512` only
(this verifier is sha256) so it is skipped; NixOS is built locally, not
downloaded, so there is nothing to verify.
- **NixOS is built, not downloaded**: `ensure_base_image` runs
`nixos-generate -f qcow` against
[`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)
once and caches the result; the per-run seed/overlay flow is otherwise
identical to the downloaded distros.
- **`test-all`** runs every distro × both variants (`test` and `test-ready`),
each cell in its own subshell on its own forwarded port, so one cell's
failure (or teardown trap) can't abort the matrix; it prints a per-cell
PASS/FAIL summary.
## Not wired into PR CI
Like the macOS harness, the runtime is host-specific (needs `/dev/kvm`,
`qemu`, and `cloud-localds`) and is not exercised by the Linux pull-request
runner. It is validated statically (`bash -n`, `shellcheck`) and run by hand
on a KVM-capable host. The cloud-image URLs in the `DISTRO` table are the one
place to bump when a distro cuts a newer build.
@@ -0,0 +1,305 @@
# Testing a clean bot-bottle install on macOS
How do you exercise `install.sh` (and, ideally, a first `bot-bottle start`)
the way a brand-new user would — on a pristine macOS environment you can
throw away afterward — *without* permanently polluting your daily-driver
Mac? The user's framing: is there a VM or boundary that avoids creating a
separate account, or is spinning up and tearing down a throwaway macOS
user on the CLI easy enough to just do that?
## Summary
There is no lightweight, in-place macOS sandbox that hands you a clean home
directory and wipeable system state without *either* a VM or a separate
user account. `sandbox-exec` (Seatbelt) is deprecated and confines a
process, not an environment; App Sandbox is for shipping apps, not for
provisioning a fresh dev host. So the real choice is exactly the two the
user named: **a disposable macOS VM** or **a throwaway user account**
and which one is right turns on a detail specific to *this* project.
bot-bottle's default macOS backend is Apple's `container`, which runs each
container in its own lightweight VM via `Virtualization.framework`
([`README.md:27`](../README.md), [`apple-container-backend.md`](apple-container-backend.md)).
That means a full end-to-end test — install *and* `bot-bottle start`
needs virtualization to work wherever bot-bottle runs. Inside a macOS guest
VM that requires **nested virtualization, which Apple gates to M3 or newer
chips on macOS 15+**. On M1/M2 you cannot run the Apple Container backend
(or Docker Desktop, same reason) inside a macOS VM at all.
The recommendation splits on what you're testing and what silicon you have:
- **Install-script correctness only** (does `curl | sh` → pipx → config dir
`doctor`'s Python/config checks pass?): a **disposable Tart VM** is the
cleanest boundary and works on any Apple Silicon Mac. `doctor` will report
the backend as not-ready inside the VM on M1/M2, which is fine — you're
testing the installer, not the runtime.
- **Full runtime** (actually launch a bottle) on **M3/M4**: a **disposable
Tart VM from a golden base image, cloned per run** is the gold standard —
a genuine kernel/state boundary that wipes to nothing.
- **Full runtime** on **M1/M2**, or when you'd rather not fight nested virt:
a **throwaway admin user via `sysadminctl`** is the pragmatic pick. It
tests the real backend because the backend runs on the host hypervisor —
but it is a *hygiene* boundary, not a security one, and it does **not**
clean the system-level footprint (see below).
Prefer the VM. Reach for the throwaway user only when nested virt is off the
table and you accept an imperfect wipe.
## Why "a boundary without a separate user" doesn't really exist on macOS
macOS has no namespace/overlay story like Linux `unshare` + tmpfs. The
options that sound like in-place sandboxes don't fit:
| Mechanism | Why it doesn't give you a clean, wipeable env |
|---|---|
| `sandbox-exec` / Seatbelt | Officially deprecated; confines *one process's* syscalls against a profile. It cannot present a fresh `$HOME` or a pristine `/usr/local`, and it won't let the Apple Container system service work. |
| App Sandbox | Entitlement-based confinement for signed `.app` bundles, not a provisioning tool for a CLI dev environment. |
| A second `$HOME` via `HOME=/tmp/foo` | Redirects only what honors `$HOME`. `install.sh` mostly does (it writes `~/.bot-bottle` and pipx/pip `--user` paths), but the Apple `container` install lands in `/usr/local` + a **system service**, and Homebrew lands in `/opt/homebrew` — all outside any `$HOME` you set. You'd get a false sense of "clean." |
| APFS snapshot rollback (`tmutil localsnapshot`) | You can't roll the live boot volume back to a local snapshot without booting to Recovery; it's not a per-run userspace undo. |
So the honest answer to "is there some boundary that avoids a separate
user?": yes — a **VM** — and it's the *stronger* boundary anyway. The only
lighter-weight option is the separate user, with the caveats below.
## What a clean install actually touches (the footprint that decides "wipeable")
Grounding the teardown story in what `install.sh` and the backend create:
| Artifact | Location | In `$HOME`? | Survives user deletion? |
|---|---|---|---|
| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,state,db}` ([`install.sh:80-83`](../install.sh), [`bot_bottle/paths.py:59`](../bot_bottle/paths.py)) | ✅ | ❌ removed with home |
| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` ([`install.sh:87-89`](../install.sh)) | ✅ | ❌ removed with home |
| private venv fallback (no pipx) | `~/.bot-bottle/venv` + symlink in `~/.local/bin` ([`install.sh`](../install.sh)) | ✅ | ❌ removed with home |
| PATH / token exports | shell profile (`~/.zprofile`, etc.); `BOT_BOTTLE_CLAUDE_OAUTH_TOKEN` ([`README.md:74`](../README.md)) | ✅ | ❌ removed with home |
| **Apple `container` install** | `/usr/local/...` + notarized `.pkg` receipts | ❌ | ✅ **stays** |
| Apple `container` **service state** | per-user: `~/Library/Application Support/com.apple.container/` (`container system start`) | ✅ | ❌ removed with home |
| **Homebrew** (if used for `container`/python) | `/opt/homebrew` | ❌ | ✅ **stays** |
| Rosetta 2 (needed for image builds) | system | ❌ | ✅ **stays** |
The bold rows are the crux: **deleting the throwaway user does not uninstall
the Apple Container runtime, Homebrew, or Rosetta.** A VM, by contrast, wipes
100% of the above by definition — that's its entire advantage for this task.
The service row is the exception, and a live harness run corrected it: the
Apple `container` service is **per-user**, not a host-wide launchd service.
`container system status` reports an `appRoot` under
`~/Library/Application Support/com.apple.container/`, and a freshly created
account sees `container system service: NOT running` even while the creating
admin's is running. That cuts both ways — the state is genuinely removed with
the home, so the reset is *more* complete than this table first claimed, but
it also means **no brand-new user can run a bottle until they run
`container system start` once**. `doctor` correctly fails until they do, which
is why `test` judges backend readiness separately from install correctness.
## Option A — Disposable Tart VM (recommended)
[Tart](https://tart.run) is a CLI-first macOS/Linux VM manager built on
`Virtualization.framework`, purpose-built for exactly this "does it work on
a clean macOS, without my settings/permissions/data" workflow. Keep one
pristine *golden* image, clone a throwaway per run, delete it after.
```sh
brew install cirruslabs/cli/tart
# One-time: build a golden base (either a prebuilt image or a vanilla IPSW).
tart clone ghcr.io/cirruslabs/macos-tahoe-base:latest golden # ~25 GB pull
# — or a truly vanilla install you click through once —
# tart create golden --from-ipsw latest --disk-size 60
# Per test run: clone → boot → test → destroy.
tart clone golden test-run
tart run test-run &
ssh admin@"$(tart ip test-run)"
# inside the guest:
# curl -fsSL https://gitea.dideric.is/didericis/bot-bottle/raw/branch/main/install.sh | sh
# bot-bottle doctor
tart stop test-run
tart delete test-run # back to pristine; golden is untouched
```
Cloning is cheap (sparse files), so the golden image is your reset button —
every `tart clone` is a fresh macOS. This is the closest thing to a Linux
`docker run --rm` for a whole Mac.
**The nested-virt caveat (read before relying on it for runtime tests).**
The Apple Container backend inside the guest needs
`Virtualization.framework` to work *inside* the VM. Apple enables nested
virtualization only on **M3 or newer**, on **macOS 15 (Sequoia) or later**;
M2 and earlier are excluded by Apple, confirmed by Apple DTS. Consequences:
- **M3/M4 host:** full runtime works in the guest. `bot-bottle doctor`
reports the backend ready and `start` can launch a bottle. Gold standard.
- **M1/M2 host:** the guest can install bot-bottle and pass the Python /
config-dir checks, but `doctor`'s backend check will fail and you cannot
launch a bottle in the VM. Still perfectly good for testing *the
installer*; not for the runtime.
- **M4-specific:** a known bug blocks pre-Ventura guests on M4; use a
current macOS guest (which you want anyway, since Apple `container`
targets macOS 26 Tahoe).
UTM is the GUI equivalent on the same framework (and was first to expose
nested virt) if you'd rather click; Tart wins for a scriptable
spin-up/tear-down loop.
## Option B — Throwaway user via `sysadminctl` (pragmatic fallback)
Creating and deleting a user from the CLI is genuinely a two-liner, and it
tests the **real** backend on any Apple Silicon Mac because the backend runs
on the host hypervisor — no nested virt needed.
```sh
# Create a self-contained admin user (admin needed for the container service).
sudo sysadminctl -addUser bbtest -fullName "bot-bottle test" \
-password 'throwaway' -admin
# Log into that account (fast-user-switch or the login window), then run the
# installer as bbtest exactly as a new user would. When done:
sudo sysadminctl -deleteUser bbtest -secure # -secure erases the home dir
```
Honest accounting of what this does and doesn't buy you:
- **Boundary strength:** it's a *hygiene / fresh-`$HOME`* boundary, **not a
security boundary.** Same kernel, same admin group; an admin test user can
touch system state. If the point is "clean environment," fine. If the point
is "contain something untrusted," this is the wrong tool — use a VM.
- **Wipe completeness:** `-secure` erases the home dir (so `~/.bot-bottle`,
the pipx venv, and profile exports go away), but as the footprint table
shows, the **Apple Container runtime, its launchd system service,
Homebrew, and Rosetta persist.** For a truly repeatable "did a *system with
nothing installed* work?" test, that residue defeats the purpose — the
second run isn't clean.
- **Operational gotchas:** don't pass real passwords on the command line (they
land in `ps` and history — this is a throwaway credential, so it's
tolerable here). Deletion must run as root from a normally-booted, admin-
logged-in session; the Terminal needs **Full Disk Access** or you'll hit
error `-14120` and a half-deleted account. Prefer letting the system place
the home dir (don't pass `-home`), or deletion can orphan it.
Use this when you're on M1/M2, you specifically want to exercise the live
backend, and you can tolerate the system-level runtime staying installed
between runs (or you uninstall Apple `container` / brew by hand to reset).
## Honorable mentions
- **External bootable macOS volume.** A fresh macOS on an external SSD (or a
separate APFS volume) is bare-metal disposable: no nested-virt limit, real
backend works, and you `diskutil` the volume away to reset. Cost is reboot
friction per run — good for an occasional thorough pass, poor for a tight
loop.
- **Rented / cloud Mac.** AWS EC2 Mac (dedicated Mac minis), Scaleway Apple
silicon, or MacStadium give a genuinely throwaway host you release when
done. Overkill for local iteration, but this is essentially what the
project's own advisory `integration-macos` CI job needs — a self-hosted
Apple Silicon runner with the `container` CLI, Python ≥ 3.11, and coverage
on the launchd service's PATH ([`README.md:78`](../README.md)). If you end
up standing up a cloud Mac for install testing, it doubles as that runner.
## Recommendation
Default to a **disposable Tart VM** — it's the only option that wipes the
*entire* footprint (including the Apple Container system service that a user
deletion leaves behind), it's a real boundary, and the spin-up/tear-down
loop is a two-command `tart clone` / `tart delete`. Confirm your chip first:
on **M3/M4** it tests install *and* runtime end-to-end; on **M1/M2** it still
cleanly tests `install.sh` + `doctor`'s Python/config path, and you fall back
to a **throwaway `sysadminctl` admin user** for live-backend testing —
accepting that it's a hygiene boundary and that you'll manually uninstall the
Apple Container runtime / Homebrew between runs to get back to truly clean.
There is no third, lighter-weight "in-place boundary without a user" that
actually delivers a clean, wipeable macOS — the VM *is* that answer, and it's
the better one.
## Harness
The throwaway-user loop is scripted in
[`scripts/macos-install-test.sh`](../../scripts/macos-install-test.sh):
`up` creates the account, `run` pipes *this checkout's* `install.sh` into it
headlessly (so a PR is verifiable before it lands) and lets the installer run
`doctor`, `down` deletes the account and its home (the full reset), and
`deep-reset` additionally uninstalls the host `container` runtime. It leans on
the footprint analysis above — the reset is just user deletion because
everything `install.sh` writes is user-home-local.
There are two one-shot cycles, because "does the installer work" and "can a new
user actually run a bottle" are different questions:
```sh
sudo ./scripts/macos-install-test.sh test # up → run → status → down
sudo ./scripts/macos-install-test.sh test-ready # ... + prereqs before status
```
`test` models a macOS system **without** the prerequisites set up for this user
— which is the default state of every new account, since the `container`
service is per-user. It asserts the install is sound and *reports* backend
readiness without failing on it, because install.sh provides no backend and so
cannot regress one.
`test-ready` models a system **with** them, then demands doctor go fully green,
backend included. Both variants pass as of this writing.
### The per-user prerequisite is two steps, not one
Running it revealed that "set up the backend for this account" is more than
starting a service:
1. **`container system start`** — the service is per-user. The run confirms it
directly: the throwaway account's `appRoot` is
`/Users/bbtest/Library/Application Support/com.apple.container/`, its own,
and starting it left the admin's service untouched.
2. **A guest kernel**, which also lives in that per-user app root. A fresh
account has none, so `container system start` prompts to download one —
and *only* prompts, since the flags default to asking. Headless callers
must pass `--enable-kernel-install` or the command dies on
`failed to read user input`.
Neither step is done by `bot-bottle backend setup --backend=macos-container`,
which only checks and then tells you to run `container system start` yourself.
So a new account's real path to a working backend is:
`container system start --enable-kernel-install`.
The harness must also enter the user's launchd domain via
`launchctl asuser <uid>` to do any of this. `container system start` registers
`com.apple.container.apiserver` as a per-user launchd agent and talks to it
over XPC; from plain `sudo -u` the caller is still in root's bootstrap
namespace, the lookup crosses domains, and the apiserver answers
`invalidState: "unauthorized request"` even though the agent started fine.
It refuses to start against an existing account (a reused home is not a clean
install), and it tears the account down from an `EXIT`/`INT` trap armed the
moment the account exists, so a failed or Ctrl-C'd run still leaves the machine
clean. Its verdict is deliberately stricter than the installer's own: note that
`install.sh` exits **0** when it finishes but `doctor` reports unmet
prerequisites, so "the installer succeeded" is not the assertion — `test` fails
if the install fails, if `bot-bottle` never reached the new user's `PATH`, or if
`doctor` is unhappy. `BB_TEST_KEEP=1` skips the teardown to poke at a failure.
### What a fresh account actually inherits
Expect the first honest run on a developer Mac to fail at the *Python* gate,
and expect that to be correct. A new account's `PATH` is just `/etc/paths`
(`/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin`),
which notably does **not** include `/opt/homebrew/bin`. Homebrew's `shellenv`
line lives in the *installing* user's `~/.zprofile` and is not inherited, so a
throwaway user resolves `python3` to `/usr/bin/python3` — the Command Line
Tools stub, still **3.9.6** on macOS 26 — and `install.sh` correctly dies on its
`3.11+` requirement. Your own shell resolving `python3` to a 3.14 Homebrew
build says nothing about what a new user sees; that gap is exactly what this
harness exists to expose.
## Sources
- [Apple Containers on macOS: technical comparison with Docker — The New Stack](https://thenewstack.io/apple-containers-on-macos-a-technical-comparison-with-docker/)
- [How to Set Up Apple Containerization on macOS 26 — Stéphane Paquet](https://spaquet.medium.com/how-to-set-up-apple-containerization-on-macos-26-f870cc8c26cd)
- [Install Apple Container CLI (macOS 15/26) — 4sysops](https://4sysops.com/archives/install-apple-container-cli-running-containers-natively-on-macos-15-sequoia-and-macos-26-tahoe/)
- [Nested virtualization on Apple Silicon (M3+, macOS 15) — UTM issue #6700](https://github.com/utmapp/UTM/issues/6700)
- [macOS 15 Sequoia nested virtualization for M3+ — Parallels Forums](https://forum.parallels.com/threads/macos-15-sequoia-nested-virtualization-for-m3-macs.364397/)
- [M2 nested virtualization restriction (Apple DTS) — Apple Developer Forums](https://developer.apple.com/forums/thread/756723)
- [M4 can't virtualize older macOS — Yahoo/Tech](https://tech.yahoo.com/computing/articles/m4-mac-computers-cant-virtualize-175122301.html)
- [Tart — macOS/Linux VMs on Apple Silicon (Cirrus Labs)](https://tart.run/quick-start/)
- [Tart GitHub](https://github.com/cirruslabs/tart)
- [macOS VMs in a single command — frr.dev](https://www.frr.dev/posts/tart-macos-vms-from-terminal/)
- [sysadminctl reference — SS64](https://ss64.com/mac/sysadminctl.html)
- [User management from the macOS command line — macnotes](https://macnotes.wordpress.com/2019/03/28/user-management-create-remove-change-password-secure-token-from-macos-command-line/)
+131 -51
View File
@@ -8,14 +8,20 @@
# pipx install bot-bottle # from a checkout or a published index
# uv tool install bot-bottle
#
# This script is a thin bootstrapper: it checks prerequisites, installs the
# package with pipx (falling back to pip --user), creates the config dir, and
# runs `bot-bottle doctor`. It is idempotent (safe to re-run) and never uses
# sudo. It does NOT install Docker or a VM backend for you — `doctor` reports
# what's missing after install.
# This script is a thin bootstrapper: it finds a Python 3.11+ interpreter,
# installs the package with pipx (falling back to a private venv), creates the
# config dir, and runs `bot-bottle doctor`. It is idempotent (safe to re-run)
# and never uses sudo. It does NOT install Docker or a VM backend for you —
# `doctor` reports what's missing after install.
#
# Env:
# BOT_BOTTLE_PYTHON interpreter to install with (skips the search)
# BOT_BOTTLE_INSTALL_SPEC pip/git spec to install instead of the default
# BOT_BOTTLE_VENV where the non-pipx install lives (~/.bot-bottle/venv)
set -eu
PACKAGE_SPEC="${BOT_BOTTLE_INSTALL_SPEC:-git+https://gitea.dideric.is/didericis/bot-bottle.git}"
VENV="${BOT_BOTTLE_VENV:-${HOME}/.bot-bottle/venv}"
MIN_PYTHON_MAJOR=3
MIN_PYTHON_MINOR=11
@@ -28,17 +34,97 @@ die() {
exit 1
}
# --- prerequisites -----------------------------------------------------------
# --- prerequisites: find an interpreter new enough ----------------------------
command -v python3 >/dev/null 2>&1 \
|| die "python3 ${MIN_PYTHON_MAJOR}.${MIN_PYTHON_MINOR}+ is required but was not found"
python3 - "$MIN_PYTHON_MAJOR" "$MIN_PYTHON_MINOR" <<'PY' || die "python3 ${MIN_PYTHON_MAJOR}.${MIN_PYTHON_MINOR} or newer is required"
# Is $1 an interpreter that exists and meets the floor?
python_ok() {
[ -n "${1:-}" ] || return 1
command -v "$1" >/dev/null 2>&1 || return 1
"$1" - "$MIN_PYTHON_MAJOR" "$MIN_PYTHON_MINOR" <<'PY' >/dev/null 2>&1
import sys
want = (int(sys.argv[1]), int(sys.argv[2]))
raise SystemExit(0 if sys.version_info[:2] >= want else 1)
PY
}
python_version() {
"$1" -c 'import sys; print("%d.%d.%d" % sys.version_info[:3])' 2>/dev/null
}
# `python3` on PATH is often NOT the newest interpreter installed, and on macOS
# it is usually the oldest: a fresh login shell's PATH is just /etc/paths, so
# python3 resolves to the Command Line Tools stub (3.9.x) while the usable
# 3.11+ build sits in /opt/homebrew/bin or a python.org framework directory,
# reachable only via a line in the *installing* user's shell profile. A new
# account inherits none of that. Look past PATH before giving up, so the common
# case installs instead of dead-ending on a version error.
find_python() {
for candidate in \
"${BOT_BOTTLE_PYTHON:-}" \
python3 \
python3.14 python3.13 python3.12 python3.11 \
/opt/homebrew/bin/python3 \
/usr/local/bin/python3 \
"${HOME}/.local/bin/python3" \
/Library/Frameworks/Python.framework/Versions/*/bin/python3
do
# An unmatched glob arrives here literally; python_ok rejects it.
if python_ok "$candidate"; then
command -v "$candidate"
return 0
fi
done
return 1
}
# An explicit choice that doesn't work is an error, not a reason to quietly
# search elsewhere and install somewhere the caller didn't ask for.
if [ -n "${BOT_BOTTLE_PYTHON:-}" ] && ! python_ok "${BOT_BOTTLE_PYTHON}"; then
if command -v "${BOT_BOTTLE_PYTHON}" >/dev/null 2>&1; then
die "BOT_BOTTLE_PYTHON=${BOT_BOTTLE_PYTHON} is $(python_version "${BOT_BOTTLE_PYTHON}"), "\
"below the ${MIN_PYTHON_MAJOR}.${MIN_PYTHON_MINOR} floor. Unset it to search for a newer one."
fi
die "BOT_BOTTLE_PYTHON=${BOT_BOTTLE_PYTHON} is not an executable interpreter."
fi
PYTHON="$(find_python || true)"
if [ -z "${PYTHON}" ]; then
path_python="$(command -v python3 2>/dev/null || true)"
if [ -n "${path_python}" ]; then
found="the python3 on your PATH is ${path_python} ($(python_version "${path_python}")), which is too old"
else
found="no python3 was found on your PATH"
fi
case "$(uname -s)" in
Darwin) fix=" brew install python@3.12
# or install from https://www.python.org/downloads/macos/
# macOS itself ships only /usr/bin/python3, which is too old" ;;
*) fix=" sudo apt install python3.12 # Debian/Ubuntu
sudo dnf install python3.12 # Fedora/RHEL" ;;
esac
die "bot-bottle needs python3 ${MIN_PYTHON_MAJOR}.${MIN_PYTHON_MINOR} or newer, and none was found.
${found}.
Also checked: python3.11-3.14, /opt/homebrew/bin, /usr/local/bin,
~/.local/bin, and python.org framework builds.
Install a newer Python, then re-run this installer:
${fix}
Already have one somewhere? Point at it directly:
BOT_BOTTLE_PYTHON=/path/to/python3 sh install.sh"
fi
# Be explicit when the interpreter isn't the obvious one, so nobody is left
# wondering which Python their install ended up on.
path_python="$(command -v python3 2>/dev/null || true)"
if [ "${PYTHON}" != "${path_python}" ]; then
say "using ${PYTHON} ($(python_version "${PYTHON}"))"
if [ -n "${path_python}" ]; then
say "note: 'python3' on your PATH is ${path_python} ($(python_version "${path_python}")), which is below the ${MIN_PYTHON_MAJOR}.${MIN_PYTHON_MINOR} floor"
fi
fi
# Installing a `git+` spec (the default) shells out to git under the hood,
# whether via pipx or pip. Fail early with a clear message rather than deep
@@ -51,30 +137,6 @@ case "${PACKAGE_SPEC}" in
;;
esac
# The pip fallback needs a usable pip. Externally-managed interpreters
# (PEP 668, common on Debian/Ubuntu/Homebrew) reject `pip install --user`;
# pipx sidesteps that, so recommend it when pip can't be used.
if ! command -v pipx >/dev/null 2>&1; then
python3 -m pip --version >/dev/null 2>&1 || die \
"neither pipx nor a usable 'python3 -m pip' was found. Install pipx "\
"(recommended): 'python3 -m pip install --user pipx' or your OS package manager."
if python3 - <<'PY'
import os
import sys
import sysconfig
# PEP 668: an EXTERNALLY-MANAGED marker in the stdlib dir means pip refuses
# to install into this interpreter without --break-system-packages.
marker = os.path.join(sysconfig.get_path("stdlib"), "EXTERNALLY-MANAGED")
raise SystemExit(0 if os.path.exists(marker) else 1)
PY
then
die "this Python is externally managed (PEP 668), so 'pip install --user' is "\
"blocked. Install pipx and re-run: 'python3 -m pip install --user --break-system-packages pipx', "\
"then 'pipx ensurepath'."
fi
fi
# --- config directories ------------------------------------------------------
mkdir -p \
@@ -84,32 +146,50 @@ mkdir -p \
# --- install -----------------------------------------------------------------
BIN_DIR="${HOME}/.local/bin"
if command -v pipx >/dev/null 2>&1; then
say "installing with pipx"
pipx install --force "${PACKAGE_SPEC}"
# --python pins the venv to the interpreter we vetted. Without it pipx uses
# whichever Python it was itself installed with, which is not necessarily
# the one that passed the version check above.
say "installing with pipx (python: ${PYTHON})"
pipx install --python "${PYTHON}" --force "${PACKAGE_SPEC}"
# Ask pipx where it puts entry points rather than assuming ~/.local/bin.
pipx_bin="$(pipx environment --value PIPX_BIN_DIR 2>/dev/null || true)"
[ -n "${pipx_bin}" ] && BIN_DIR="${pipx_bin}"
else
say "pipx not found; installing with 'python3 -m pip install --user'"
python3 -m pip install --user --upgrade "${PACKAGE_SPEC}"
# No `pip install --user` fallback: PEP 668 makes it unusable on nearly
# every interpreter a Mac offers (Homebrew and python.org are both
# externally managed), and on Debian/Ubuntu too. A private venv sidesteps
# that entirely — PEP 668 does not apply inside a venv — and `venv` is
# stdlib, so unlike pipx there is nothing to bootstrap first.
say "pipx not found; installing into a managed venv at ${VENV}"
"${PYTHON}" -m venv --clear "${VENV}" || die \
"could not create a virtualenv at ${VENV} using ${PYTHON}. On Debian/Ubuntu "\
"the venv module ships separately: 'sudo apt install python3-venv'."
"${VENV}/bin/python" -m pip install --upgrade "${PACKAGE_SPEC}"
# Expose the entry point outside the venv, the way pipx would.
mkdir -p "${BIN_DIR}"
ln -sf "${VENV}/bin/bot-bottle" "${BIN_DIR}/bot-bottle"
fi
# --- locate the entry point --------------------------------------------------
# The pip --user scripts directory is platform-specific: ~/.local/bin on Linux,
# but ~/Library/Python/<X.Y>/bin on a python.org macOS interpreter. Ask the
# interpreter for its own user-scheme scripts dir instead of hardcoding.
USER_SCRIPTS="$(python3 - <<'PY'
import sysconfig
print(sysconfig.get_path("scripts", sysconfig.get_preferred_scheme("user")))
PY
)"
if command -v bot-bottle >/dev/null 2>&1; then
BOT_BOTTLE_BIN="bot-bottle"
elif [ -n "${USER_SCRIPTS}" ] && [ -x "${USER_SCRIPTS}/bot-bottle" ]; then
BOT_BOTTLE_BIN="${USER_SCRIPTS}/bot-bottle"
say "note: add ${USER_SCRIPTS} to your PATH to run 'bot-bottle' directly"
elif [ -x "${BIN_DIR}/bot-bottle" ]; then
BOT_BOTTLE_BIN="${BIN_DIR}/bot-bottle"
# Name the file the user's own login shell actually reads. ~/.profile is
# the safe default for non-zsh: bash falls back to it, and suggesting
# ~/.bash_profile could shadow an existing ~/.profile.
case "${SHELL:-}" in
*/zsh) profile="~/.zprofile" ;;
*) profile="~/.profile" ;;
esac
say "note: add ${BIN_DIR} to your PATH to run 'bot-bottle' directly:"
say " echo 'export PATH=\"${BIN_DIR}:\$PATH\"' >> ${profile}"
else
die "bot-bottle was installed but is not on PATH; add ${USER_SCRIPTS:-your user scripts dir} to PATH and re-run"
die "bot-bottle was installed but no entry point turned up in ${BIN_DIR}"
fi
# --- verify ------------------------------------------------------------------
+3
View File
@@ -2,9 +2,12 @@
# The bot-bottle project itself has no runtime dependencies.
# These tools are used for code quality checks in CI/CD.
-r requirements.orchestrator.in
pylint>=3.0.0
pyright>=1.1.411
coverage>=7.0.0
# PEP 517 build front-end used by tests/unit/test_wheel_install.py to build and
# install a real wheel (proves the installed distribution is self-contained).
build>=1.0.0
# FastAPI's in-process TestClient transport.
httpx>=0.28.0
+3
View File
@@ -0,0 +1,3 @@
# Runtime dependencies baked only into the orchestrator images.
fastapi==0.140.0
uvicorn==0.51.0
+182
View File
@@ -0,0 +1,182 @@
#
# This file is autogenerated by pip-compile with Python 3.13
# by the following command:
#
# pip-compile --generate-hashes --output-file=requirements.orchestrator.lock requirements.orchestrator.in
#
annotated-doc==0.0.4 \
--hash=sha256:571ac1dc6991c450b25a9c2d84a3705e2ae7a53467b5d111c24fa8baabbed320 \
--hash=sha256:fbcda96e87e9c92ad167c2e53839e57503ecfda18804ea28102353485033faa4
# via fastapi
annotated-types==0.8.0 \
--hash=sha256:13b2beaad985e05e2d6407ee4c4f35590b11f8d693a258a561055cac8f64cab7 \
--hash=sha256:f072f4d804ea359e4eaf198b1af7a8b0943881a87f31bb764f8bf219bb9419e0
# via pydantic
anyio==4.14.2 \
--hash=sha256:9f505dda5ac9f0c8309b5e8bd445a8c2bf7246f3ce950121e45ea15bc41d1494 \
--hash=sha256:cfa139f3ed1a23ee8f88a145ddb5ac7605b8bbfd8592baacd7ce3d8bb4313c7f
# via starlette
click==8.4.2 \
--hash=sha256:9a6cea6e60b17ebe0a44c5cc636d94f09bd66142c1cd7d8b4cd731c4917a15f6 \
--hash=sha256:e6f9f66136c816745b9d65817da91d61d957fb16e02e4dcd0552553c5a197b76
# via uvicorn
fastapi==0.140.0 \
--hash=sha256:e951c0a0d9540bf5d9a2a9e078fd415da2ab7e312d435139e7d9e2e7fe9f0b23 \
--hash=sha256:f338951b82fd74ca8f843163aec43ea1a1ce84d515415a50fa98fa25572a5544
# via -r requirements.orchestrator.in
h11==0.16.0 \
--hash=sha256:4e35b956cf45792e4caa5885e69fba00bdbc6ffafbfa020300e549b208ee5ff1 \
--hash=sha256:63cf8bbe7522de3bf65932fda1d9c2772064ffb3dae62d55932da54b31cb6c86
# via uvicorn
idna==3.18 \
--hash=sha256:7f952cbe720b688055e3f87de14f5c3e5fdaa8bc3928985c4077ca689de849a2 \
--hash=sha256:ffb385a7e039654cef1ab9ef32c6fafe283c0c0467bba1d9029738ce4a14a848
# via anyio
pydantic==2.13.4 \
--hash=sha256:45a282cde31d808236fd7ea9d919b128653c8b38b393d1c4ab335c62924d9aba \
--hash=sha256:c40756b57adaa8b1efeeced5c196f3f3b7c435f90e84ea7f443901bec8099ef6
# via fastapi
pydantic-core==2.46.4 \
--hash=sha256:00c603d540afdd6b80eb39f078f33ebd46211f02f33e34a32d9f053bba711de0 \
--hash=sha256:0186750b482eefa11d7f435892b09c5c606193ef3375bcf94aa00ae6bfb66262 \
--hash=sha256:041bde0a48fd37cf71cab1c9d56d3e8625a3793fef1f7dd232b3ff37e978ecda \
--hash=sha256:0c563b08bca408dc7f65f700633d8442fffb2421fc47b8101377e9fd65051ff0 \
--hash=sha256:0cbe8b01f948de4286c74cdd6c667aceb38f5c1e26f0693b3983d9d74887c65e \
--hash=sha256:0ce40cd7b21210e99342afafbd4d0f76d784eb5b1d60f3bdc566be4983c6c73b \
--hash=sha256:0e96592440881c74a213e5ad528e2b24d3d4f940de2766bed9010ab1d9e51594 \
--hash=sha256:10e17cbb10a330363733efc4d7c4d0dd827ac0909b8f6a6542298fed1ea62f29 \
--hash=sha256:133878133d271ade3d41d1bfb2a45ec38dbdbda40bc065921c6b04e4630127e2 \
--hash=sha256:14d4edf427bdcf950a8a02d7cb44a08614388dd6e1bdcbf4f67504fa7887da9c \
--hash=sha256:14f4c5d6db102bd796a627bbb3a17b4cf4574b9ae861d8b7c9a9661c6dd3362d \
--hash=sha256:17299feefe090f2caa5b8e37222bb5f663e4935a8bfa6931d4102e5df1a9f398 \
--hash=sha256:184c081504d17f1c1066e430e117142b2c77d9448a97f7b65c6ac9fd9aee238d \
--hash=sha256:18e5ceec2ab67e6d5f1a9085e5a24c9c4e2ac4545730bfe668680bca05e555f3 \
--hash=sha256:19e51f073cd3df251856a8a4189fbdf1de4012c3ebacfb1884f94f1eb406079f \
--hash=sha256:1a7dd0b3ee80d90150e3495a3a13ac34dbcbfd4f012996a6a1d8900e91b5c0fb \
--hash=sha256:1d8ba486450b14f3b1d63bc521d410ec7565e52f887b9fb671791886436a42f7 \
--hash=sha256:2108ba5c1c1eca18030634489dc544844144ee36357f2f9f780b93e7ddbb44b5 \
--hash=sha256:228ee9bae8bef5b1e97ec58302f80357c37199e0d0a99174e138d28e6957b9d9 \
--hash=sha256:23ace664830ee0bfe014a0c7bc248b1f7f25ed7ad103852c317624a1083af462 \
--hash=sha256:2412e734dcb48da14d4e4006b82b46b74f2518b8a26ee7e58c6844a6cd6d03c4 \
--hash=sha256:29c61fc04a3d840155ff08e475a04809278972fe6aef51e2720554e96367e34b \
--hash=sha256:2f84c03c8607173d16b5a854ec68a2f9079ae03237a54fb506d13af47e1d018d \
--hash=sha256:3009f12e4e90b7f88b4f9adb1b0c4a3d58fe7820f3238c190047209d148026df \
--hash=sha256:3245406455a5d98187ec35530fd772b1d799b26667980872c8d4614991e2c4a2 \
--hash=sha256:3447661d99f75a3683a4cf5c87da72f2161964611864dbbeac7fbb118bb4bfc0 \
--hash=sha256:372429a130e469c9cd698925ce5fc50940b7a1336b0d82038e63d5bbc4edc519 \
--hash=sha256:395aebd9183f9d112f569aeb5b2214d1a10a33bec8456447f7fbdfa51d38d4cd \
--hash=sha256:3a233125ac121aa3ffba9a2b59edfc4a985a76092dc8279586ab4b71390875e7 \
--hash=sha256:3be77f45df024d789a672ae34f8b06fb346c4f9f46ea714956660ea4862e89ac \
--hash=sha256:3bf92c5d0e00fefaab325a4d27828fe6b6e2a21848686b5b60d2d9eeb09d76c6 \
--hash=sha256:3ecbc122d18468d06ca279dc26a8c2e2d5acb10943bb35e36ae92096dc3b5565 \
--hash=sha256:3fb702cd90b0446a3a1c5e470bfa0dd23c0233b676a9099ddcc964fa6ca13898 \
--hash=sha256:428e04521a40150c85216fc8b85e8d39fece235a9cf5e383761238c7fa9b96fb \
--hash=sha256:432c179df7874eeb73307aad2df0755e1ae0efa61ff0ea89b93e194411ae3928 \
--hash=sha256:4a05d69cba51d852c5c3e92758653245a50c0b646ced0cf05bd793ed592839d6 \
--hash=sha256:4c63ebc82684aa89d9a3bcbd13d515b3be44250dc68dd3bd81526c1cb31286c3 \
--hash=sha256:4fc73cb559bdb54b1134a706a2802a4cddd27a0633f5abb7e53056268751ac6a \
--hash=sha256:4fcbe087dbc2068af7eda3aa87634eba216dbda64d1ae73c8684b621d33f6596 \
--hash=sha256:56cb4851bcaf3d117eddcef4fe66afd750a50274b0da8e22be256d10e5611987 \
--hash=sha256:5855698a4856556d86e8e6cd8434bc3ac0314ee8e12089ae0e143f64c6256e4e \
--hash=sha256:5a4330cdbc57162e4b3aa303f588ba752257694c9c9be3e7ebb11b4aca659b5d \
--hash=sha256:5b712b53160b79a5850310b912a5ef8e57e56947c8ad690c227f5c9d7e561712 \
--hash=sha256:5d5902252db0d3cedf8d4a1bc68f70eeb430f7e4c7104c8c476753519b423008 \
--hash=sha256:617d7e2ca7dcb8c5cf6bcb8c59b8832c94b36196bbf1cbd1bfb56ed341905edd \
--hash=sha256:62f875393d7f270851f20523dd2e29f082bcc82292d66db2b64ea71f64b6e1c1 \
--hash=sha256:633147d34cf4550417f12e2b1a0383973bdf5cdfde212cb09e9a581cf10820be \
--hash=sha256:66ce7632c22d837c95301830e111ad0128a32b8207533b60896a96c4915192ea \
--hash=sha256:6b3ace8194b0e5204818c92802dcdca7fc6d88aabbb799d7c795540d9cd6d292 \
--hash=sha256:6f2eeda33a839975441c86a4119e1383c50b47faf0cbb5176985565c6bb02c33 \
--hash=sha256:7027560ee92211647d0d34e3f7cd6f50da56399d26a9c8ad0da286d3869a53f3 \
--hash=sha256:7283d57845ecf5a163403eb0702dfc220cc4fbdd18919cb5ccea4f95ee1cdab4 \
--hash=sha256:7a5f930472650a82629163023e630d160863fce524c616f4e5186e5de9d9a49b \
--hash=sha256:7bfb192b3f4b9e8a89b6277b6ce787564f62cfd272055f6e685726b111dc7826 \
--hash=sha256:811ff8e9c313ab425368bcbb36e5c4ebd7108c2bbf4e4089cfbb0b01eff63fac \
--hash=sha256:8233f2947cf85404441fd7e0085f53b10c93e0ee78611099b5c7237e36aacbf7 \
--hash=sha256:82cf5301172168103724d49a1444d3378cb20cdee30b116a1bd6031236298a5d \
--hash=sha256:8358a950c8909158e3df31538a7e4edc2d7265a7c54b47f0864d9e5bae9dcebf \
--hash=sha256:85bb3611ff1802f3ee7fdd7dbff26b56f343fb432d57a4728fdd49b6ef35e2f4 \
--hash=sha256:86e1a4418c6cd97d60c95c71164158eaf7324fae7b0923264016baa993eba6fc \
--hash=sha256:8b9bab013d1c7a79d3501ff86d0bc9c31bf587db4551677b96bec07df78c6b15 \
--hash=sha256:8c5dac79fa1614d1e06ca695109c6105923bd9c7d1d6c918d4e637b7e6b32fd3 \
--hash=sha256:8d0820e8192167f80d88d64038e609c31452eeca865b4e1d9950a27a4609b00b \
--hash=sha256:8daafc69c93ee8a0204506a3b6b30f586ef54028f52aeeeb5c4cfc5184fd5914 \
--hash=sha256:9037063db01f09b09e237c282b6792bd4da634b5402c4e7f0c61effed7701a04 \
--hash=sha256:905a0ed8ea6f2d61c1738835f99b699348d7857379083e5fc497fa0c967a407c \
--hash=sha256:90884113d8b48f760e9587002789ddd741e76ab9f89518cd1e43b1f1a52ec44b \
--hash=sha256:91a06d2e259ecfbd8c901d70c3c507900458498142b3026a296b7de4d1322cc9 \
--hash=sha256:926c9541b14b12b1681dca8a0b75feb510b06c6341b70a8e500c2fdcff837cce \
--hash=sha256:9401557acd873c3a7f3eb9383edef8ac4968f9510e340f4808d427e75667e7b4 \
--hash=sha256:9551187363ffc0de2a00b2e47c25aeaeb1020b69b668762966df15fc5659dd5a \
--hash=sha256:962ccbab7b642487b1d8b7df90ef677e03134cf1fd8880bf698649b22a69371f \
--hash=sha256:97e7cf2be5c77b7d1a9713a05605d49460d02c6078d38d8bef3cbe323c548424 \
--hash=sha256:9aa768456404a8bf48a4406685ac2bec8e72b62c69313734fa3b73cf33b3a894 \
--hash=sha256:9bc519fbf2b7578398853d815009ae5e4d4603d12f4e3f91da8c06852d3da3e9 \
--hash=sha256:9d56801be94b86a9da183e5f3766e6310752b99ff647e38b09a9500d88e46e76 \
--hash=sha256:9f444c499b3eefd3a92e348059471ea0c3a6e303d9c1cec09fa748fd9f895201 \
--hash=sha256:9fa8ae11da9e2b3126c6426f147e0fba88d96d65921799bb30c6abd1cb2c97fb \
--hash=sha256:a0f62d0a58f4e7da165457e995725421e0064f2255d8eccebc49f41bbc23b109 \
--hash=sha256:a396dcc17e5a0b164dbe026896245a4fa9ff402edca1dff0be3d53a517f74de4 \
--hash=sha256:aaa2a54443eff1950ba5ddc6b6ccda0d9c84a364276a62f969bdf2a390650848 \
--hash=sha256:ad785e92e6dc634c21555edc8bd6b64957ab844541bcb96a1366c202951ae526 \
--hash=sha256:af8244b2bef6aaad6d92cda81372de7f8c8d36c9f0c3ea36e827c60e7d9467a0 \
--hash=sha256:b078afbc25f3a1436c7a1d2cd3e322497ee99615ba97c563566fdf46aff1ee01 \
--hash=sha256:b2f69dec1725e79a012d920df1707de5caf7ed5e08f3be4435e25803efc47458 \
--hash=sha256:b8458003118a712e66286df6a707db01c52c0f52f7db8e4a38f0da1d3b94fc4e \
--hash=sha256:bb63e0198ca18aad131c089b9204c23079c3afa95487e561f4c522d519e55aba \
--hash=sha256:bfec22eab3c8cc2ceec0248aec886624116dc079afa027ecc8ad4a7e62010f8a \
--hash=sha256:c1747f85cee84c26985853c6f3d9bd3e75da5212912443fa111c113b9c246f39 \
--hash=sha256:c1b3f518abeca3aa13c712fd202306e145abf59a18b094a6bafb2d2bbf59192c \
--hash=sha256:c50f2528cf200c5eed56faf3f4e22fcd5f38c157a8b78576e6ba3168ec35f000 \
--hash=sha256:c68fcd102d71ea85c5b2dfac3f4f8476eff42a9e078fd5faefff6d145063536b \
--hash=sha256:c7a7bd4e39e8e4c12c39cd480356842b6a8a06e41b23a55a5e3e191718838ddf \
--hash=sha256:c94f0688e7b8d0a67abf40e57a7eaaecd17cc9586706a31b76c031f63df052b4 \
--hash=sha256:cbaf13819775b7f769bf4a1f066cb6df7a28d4480081a589828ef190226881cd \
--hash=sha256:cd2213145bcc2ba85884d0ac63d222fece9209678f77b9b4d76f054c561adb28 \
--hash=sha256:ce5c1d2a8b27468f433ca974829c44060b8097eedc39933e3c206a90ee49c4a9 \
--hash=sha256:d396ec2b979760aaf3218e76c24e65bd0aca24983298653b3a9d7a45f9e47b30 \
--hash=sha256:d51026d73fcfd93610abc7b27789c26b313920fcfb20e27462d74a7f8b06e983 \
--hash=sha256:d80ee3d731373b24cebbc10d689ca4ee1875caf0d5703a245db18efd4dd37fc1 \
--hash=sha256:d995260fdf4e1db774581b4900e0f832abe3c7c84996726bbc161b19c8f29e76 \
--hash=sha256:da4b951fe36dc7c3a1ccb4e3cd1747c3542b8c9ceede8fc86cae054e764485f5 \
--hash=sha256:daa27d92c36f24388fe3ad306b174781c747627f134452e4f128ea00ce1fe8c4 \
--hash=sha256:db06ffe51636ffe9ca531fe9023dd64bdd794be8754cb5df57c5498ae5b518a7 \
--hash=sha256:e0d65b8c354be7fb5f720c3caa8bc940bc2d20ce749c8e06135f07f8ed95dd7c \
--hash=sha256:e68b7a074f65a2fd746c52a7ce6142ab7006074ac269ace0c25cd8ba171f8066 \
--hash=sha256:e739fee756ba1010f8bcccb534252e85a35fe45ae92c295a06059ce58b74ccd3 \
--hash=sha256:e846ae7835bf0703ae43f534ab79a867146dadd59dc9ca5c8b53d5c8f7c9ef02 \
--hash=sha256:e9c26f834c65f5752f3f06cb08cb86a913ceb7274d0db6e267808a708b46bc89 \
--hash=sha256:ea793e075b70290d89d8142074262885d3f7da19634845135751bd6344f73b50 \
--hash=sha256:f027324c56cd5406ca49c124b0db10e56c69064fec039acc571c29020cc87c76 \
--hash=sha256:f13a646d65d09fbf1bc6b3a9635d30095c8e7e5cc419ff35ecc563c5fd04cd49 \
--hash=sha256:f47286a97f0bc9b8859519809077b91b2cefe4ae47fcbf5e466a009c1c5d742b \
--hash=sha256:f747929cf940cddb5b3668a390056ddd5ba2e5010615ea2dcf4f9c4f3ab8791d \
--hash=sha256:f99626688942fb746e545232e7726926f3be91b5975f8b55327665fafda991c7 \
--hash=sha256:f9fa868638bf362d3d138ea55829cefb3d5f4b0d7f142234382a15e2485dbec4 \
--hash=sha256:fbdb89b3e1c94a30cc5edfce477c6e6a5dc4d8f84665b455c27582f211a1c72c \
--hash=sha256:fc010ab034c8c7452522748bf937df58020d256ccae0874463d1f4d01758af8e \
--hash=sha256:fc3e9034a63de20e15e8ade85358bc6efc614008cab72898b4b4952bea0509ff \
--hash=sha256:fd8b3d9fd264be37976686c7f65cd52a83f5e84f4bfd2adf9c1d469676bbb6ae
# via pydantic
starlette==1.3.1 \
--hash=sha256:05d0213193f2fbaae60e2ecb593b4add4262ad4e46536b54abe36f11a71724e0 \
--hash=sha256:c7372aae11c3c3f26a42df7bd626cec2f47d03483d261d369516a615a53714c6
# via fastapi
typing-extensions==4.16.0 \
--hash=sha256:481caa481374e813c1b176ada14e97f1f67a4539ce9cfeb3f350d78d6370c2e8 \
--hash=sha256:dc983d19a509c94dba722ee6abd33940f7c05a89e243c47e907eb4db6f1a43e5
# via
# fastapi
# pydantic
# pydantic-core
# typing-inspection
typing-inspection==0.4.2 \
--hash=sha256:4ed1cacbdc298c220f1bd249ed5287caa16f34d44ef4e9c3d0cbad5b521545e7 \
--hash=sha256:ba561c48a67c5958007083d386c3295464928b01faa735ab8547c5692e87f464
# via
# fastapi
# pydantic
uvicorn==0.51.0 \
--hash=sha256:5d38af6cd620f2ae3849fb44fd4879e0890aa1febe8d47eb355fb45d93fe6a5b \
--hash=sha256:f6f4b69b657c312f516dd2d268ab9ae6f254b11e4bac504f37b2ab58b24dd0b0
# via -r requirements.orchestrator.in
+5 -5
View File
@@ -13,7 +13,7 @@
# are re-executed; no KVM or Docker dependency.
#
# Pass "critical" as the last argument in either mode to also report just the
# critical modules (ADR 0004 target: 90%).
# critical modules (ADR 0004 target: 85%).
set -euo pipefail
cd "$(dirname "$0")/.."
@@ -34,8 +34,8 @@ if [ "${1:-}" = "aggregate" ]; then
"$PY" -m coverage report -m
if [ "${2:-}" = "critical" ]; then
echo "== critical modules (ADR 0004 minimum: 90%) ==" >&2
"$PY" -m coverage report --include="$CRITICAL" --fail-under=90
echo "== critical modules (ADR 0004 minimum: 85%) ==" >&2
"$PY" -m coverage report --include="$CRITICAL" --fail-under=85
fi
exit 0
fi
@@ -55,6 +55,6 @@ echo "== combined report ==" >&2
"$PY" -m coverage report -m
if [ "${1:-}" = "critical" ]; then
echo "== critical modules (ADR 0004 minimum: 90%) ==" >&2
"$PY" -m coverage report --include="$CRITICAL" --fail-under=90
echo "== critical modules (ADR 0004 minimum: 85%) ==" >&2
"$PY" -m coverage report --include="$CRITICAL" --fail-under=85
fi
+1 -1
View File
@@ -1,4 +1,4 @@
# Critical security/logic core held to the >=90% coverage bar by
# Critical security/logic core held to the >=85% coverage bar by
# docs/decisions/0004-coverage-policy.md.
#
# SINGLE SOURCE OF TRUTH: scripts/coverage.sh (the `critical` report) and
+3 -3
View File
@@ -13,8 +13,8 @@ policy.
Usage:
scripts/coverage.sh # produce .coverage first
python3 scripts/diff_coverage.py # gate against origin/main, min 90%
python3 scripts/diff_coverage.py --base main --min 85
python3 scripts/diff_coverage.py # gate against origin/main, min 80%
python3 scripts/diff_coverage.py --base main --min 75
"""
from __future__ import annotations
@@ -74,7 +74,7 @@ def main() -> int:
ap = argparse.ArgumentParser()
ap.add_argument("--base", default="origin/main",
help="git ref to diff against (default: origin/main)")
ap.add_argument("--min", type=float, default=90.0,
ap.add_argument("--min", type=float, default=80.0,
help="minimum %% of changed executable lines covered")
args = ap.parse_args()
+1 -1
View File
@@ -9,7 +9,7 @@
#
# Why a pool + one-time setup: creating a TAP and assigning it an IP
# needs CAP_NET_ADMIN. Pre-creating user-owned, pre-addressed TAPs
# means `./cli.py start` never needs root. The nft table is static
# means `bot-bottle start` never needs root. The nft table is static
# (keyed on the `bbfc*` interface wildcard), so it covers every slot
# without per-launch changes.
#
+24
View File
@@ -0,0 +1,24 @@
# NixOS image for scripts/linux-install-test.sh.
#
# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
# AMIs), so the harness BUILDS this one with nixos-generators (-f qcow). It is
# deliberately minimal — no python3/git/pipx — so the bare `test` variant is
# genuinely under-provisioned and exercises install.sh's guards; `test-ready`
# provisions them with `nix profile install` (hence flakes below).
{ lib, ... }:
{
# Consume the same NoCloud seed the other distros use: cloud-init injects the
# per-run ephemeral SSH key for root. Leave networking to NixOS's default
# dhcpcd (QEMU user-mode NAT) — enabling cloud-init's networkd here conflicts
# with dhcpcd and can drop the guest's network.
services.cloud-init.enable = true;
services.cloud-init.network.enable = false;
services.openssh.enable = true;
services.openssh.settings.PermitRootLogin = lib.mkForce "prohibit-password";
# Flakes so the `test-ready` prereq step can `nix profile install nixpkgs#...`.
nix.settings.experimental-features = [ "nix-command" "flakes" ];
system.stateVersion = "24.11";
}
+753
View File
@@ -0,0 +1,753 @@
#!/usr/bin/env bash
# Clean-install test harness for the Linux path.
#
# Exercises install.sh the way a brand-new user would, inside a THROWAWAY
# QEMU/KVM virtual machine that is booted from a distro cloud image and
# deleted afterward. install.sh's entire footprint is user-home-local (the
# pipx venv under ~/.local, the ~/.bot-bottle config dir, and a printed PATH
# hint), so a fresh VM's fresh $HOME is the clean surface we want — and unlike
# a throwaway user account, tearing the VM down also wipes any OS-level
# prerequisites installed into it, so the reset is total. Full rationale in
# docs/research/testing-clean-install-on-linux.md.
#
# Why a VM and not a container or a throwaway user: a container shares the
# host kernel and cannot exercise a genuinely pristine OS (systemd, the distro
# package manager, PEP 668 externally-managed Python) the way a real guest
# does, and a throwaway user leaves every system package it installs behind.
# The host already needs KVM for the Firecracker backend, so a per-run,
# copy-on-write VM is cheap here: one cached base image, a throwaway overlay
# per run (`qemu-img create -b base`), deleted on teardown — the Linux
# equivalent of `docker run --rm`, but for a whole machine.
#
# Usage:
# ./scripts/linux-install-test.sh test # up -> run -> verdict -> down (bare host)
# ./scripts/linux-install-test.sh test-ready # ... with prerequisites installed first
# ./scripts/linux-install-test.sh test-all # every distro × both variants, with a summary
# ./scripts/linux-install-test.sh up # fetch base image, boot a fresh VM
# ./scripts/linux-install-test.sh prereqs # install python3 + git + pipx in the VM
# ./scripts/linux-install-test.sh run # pipe install.sh into the VM
# ./scripts/linux-install-test.sh status # VM reachable? is the install sound?
# ./scripts/linux-install-test.sh down # kill the VM, delete overlay + seed (the reset)
# ./scripts/linux-install-test.sh ssh # open an interactive shell in the running VM
#
# TWO TEST VARIANTS, because "does install.sh handle an unprepared host" and
# "does a prepared host get a clean install" are different questions (mirrors
# the macOS harness's test / test-ready split):
#
# test A Linux system WITHOUT the prerequisites set up — the default
# state of a stock cloud image (python3 is usually present for
# cloud-init, but git and pipx are not). This exercises
# install.sh's own prerequisite-guard logic. It is SOUND — a
# PASS — when install.sh EITHER installs cleanly (the image
# already had enough) OR declines with one of its own recognized,
# actionable errors (missing python3/git, no usable pip, PEP 668
# externally-managed). A crash or an unrecognized failure fails.
#
# test-ready A Linux system WITH the prerequisites satisfied — the harness
# installs python3 + git + pipx first (see the DISTRO table),
# then runs install.sh. A graceful decline is no longer good
# enough here: the install MUST land, the entry point must run,
# and doctor must report a usable python and config without
# crashing.
#
# Neither variant requires a green backend. install.sh does not install a
# backend and cannot regress one, and inside a plain VM there is no nested KVM
# for Firecracker; the harness does not provision the Docker backend either. So
# backend readiness is reported, not required (this is where Linux necessarily
# diverges from the macOS test-ready, which reaches the host backend).
# BB_TEST_REQUIRE_BACKEND=1 makes it fatal anyway, for a nested-virt host.
#
# Config via env:
# BB_TEST_DISTRO ubuntu | fedora | arch | alpine | nixos (default: ubuntu)
# BB_TEST_SSH_PORT host port forwarded to the guest's :22 (default: 2222)
# BB_TEST_CACHE_DIR where base images are cached (default: ~/.cache/bot-bottle-install-test)
# BB_TEST_RUN_DIR per-run scratch (overlay, seed, key, …) (default: a mktemp dir)
# BB_TEST_MEM_MB guest RAM (default: 2048)
# BB_TEST_CPUS guest vCPUs (default: 2)
# BB_TEST_DISK overlay virtual size (default: 12G)
# BB_TEST_BOOT_TIMEOUT seconds to wait for SSH after boot (default: 300)
# BB_TEST_KEEP 1 = the test cycles skip teardown, to poke at a failure
# BB_TEST_REQUIRE_BACKEND 1 = make a not-ready backend fatal (needs nested virt)
# BB_TEST_INSTALL_URL curl this install.sh in the guest instead of piping the local checkout
# BB_TEST_SKIP_VERIFY 1 = skip base-image checksum verification (not recommended)
# BOT_BOTTLE_INSTALL_SPEC passed through to install.sh (pip / git spec)
#
# Notes:
# * Needs /dev/kvm, qemu-system-x86_64, and cloud-localds (cloud-image-utils
# / cloud-utils); the nixos distro additionally needs nixos-generate. On the
# NixOS host: nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#nixos-generators
# * Networking is user-mode (`-netdev user,hostfwd`) so the harness needs no
# root, no bridge, and touches no host network state. Only SSH is forwarded.
# * The cloud-image URLs in the DISTRO table are the one place to bump when a
# distro cuts a new build; each is verified against the vendor's published
# checksum at download time (guarding against truncated/corrupt pulls).
set -euo pipefail
DISTRO="${BB_TEST_DISTRO:-ubuntu}"
SSH_PORT="${BB_TEST_SSH_PORT:-2222}"
CACHE_DIR="${BB_TEST_CACHE_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/bot-bottle-install-test}"
MEM_MB="${BB_TEST_MEM_MB:-2048}"
CPUS="${BB_TEST_CPUS:-2}"
DISK="${BB_TEST_DISK:-12G}"
BOOT_TIMEOUT="${BB_TEST_BOOT_TIMEOUT:-300}"
_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
_REPO_ROOT="$(cd "$_SCRIPT_DIR/.." && pwd)"
# Per-run scratch. Persisted across sub-commands (up/prereqs/run/status/down)
# via a marker file so `up` in one invocation and `down` in the next find the
# same VM; the test cycles set RUN_DIR themselves and never write the marker.
RUN_DIR="${BB_TEST_RUN_DIR:-}"
# Set by the test cycles, which chain the steps and suppress the per-step "next
# command" hints. _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL are the per-variant
# knobs the shared cycle and teardown read.
IN_TEST=0
_STEPS=4
_PASS_CLAIM=""
_REQUIRE_INSTALL=0
# --- distro table ----------------------------------------------------
# For each distro: cloud-image URL | checksum-file URL | default SSH user |
# prerequisite-install command (run in the guest; uses sudo when user != root).
#
# The prerequisite command installs python3 + git + pipx — install.sh installs
# none of them — so `test-ready` (and the `prereqs` sub-command) drive
# install.sh down its recommended pipx path. Bump the URLs here when a distro
# publishes a newer build.
declare -A IMAGE_URL SUM_URL SSH_USER PREREQ
IMAGE_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
SUM_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/SHA256SUMS"
SSH_USER[ubuntu]="ubuntu"
PREREQ[ubuntu]="sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3 git pipx"
IMAGE_URL[fedora]="https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2"
SUM_URL[fedora]="https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-44-1.7-x86_64-CHECKSUM"
SSH_USER[fedora]="fedora"
PREREQ[fedora]="sudo dnf install -y python3 git pipx"
IMAGE_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-cloudimg.qcow2"
SUM_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-cloudimg.qcow2.SHA256"
SSH_USER[arch]="arch"
PREREQ[arch]="sudo pacman -Sy --noconfirm python git python-pipx"
# Use the *nocloud_* Alpine variant, not generic_: the generic image probes
# network datasources and ignores the local NoCloud seed, so cloud-init never
# runs and the SSH key is never injected. (Only published at .0 patch levels.)
IMAGE_URL[alpine]="https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/nocloud_alpine-3.21.0-x86_64-bios-cloudinit-r0.qcow2"
SUM_URL[alpine]="" # Alpine cloud images ship .sha512 only; this verifier is sha256.
SSH_USER[alpine]="alpine"
PREREQ[alpine]="sudo apk add --no-cache python3 git pipx"
# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
# AMIs), so the harness BUILDS one with nixos-generators — see build_nixos_image
# and linux-install-test-nixos.nix. It's externally-managed in its own way (no
# FHS ~/.local on PATH by default); `nix profile install` provisions the
# prerequisites into root's profile (the image enables flakes for this).
IMAGE_URL[nixos]="nix:build" # sentinel: ensure_base_image builds instead of downloading
SUM_URL[nixos]=""
SSH_USER[nixos]="root"
# Pin to a stable release: the guest's default `nixpkgs` registry is unstable,
# where pipx isn't in the binary cache and builds from source (its test suite
# currently fails to build). nixos-24.11 has these cached as substitutes.
PREREQ[nixos]="nix profile install nixpkgs/nixos-24.11#python3 nixpkgs/nixos-24.11#git nixpkgs/nixos-24.11#pipx"
ALL_DISTROS=(ubuntu fedora arch alpine nixos)
# --- guards ----------------------------------------------------------
require_linux() {
[ "$(uname -s)" = "Linux" ] \
|| { echo "error: this harness is Linux-only (uname is $(uname -s))" >&2; exit 1; }
}
require_kvm() {
[ -e /dev/kvm ] && [ -r /dev/kvm ] && [ -w /dev/kvm ] \
|| { echo "error: /dev/kvm is missing or not accessible (add yourself to the 'kvm' group)" >&2; exit 1; }
}
require_tools() {
local missing=()
command -v qemu-system-x86_64 >/dev/null 2>&1 || missing+=(qemu-system-x86_64)
command -v qemu-img >/dev/null 2>&1 || missing+=(qemu-img)
command -v cloud-localds >/dev/null 2>&1 || missing+=(cloud-localds)
command -v ssh >/dev/null 2>&1 || missing+=(ssh)
command -v curl >/dev/null 2>&1 || missing+=(curl)
if [ "${#missing[@]}" -ne 0 ]; then
echo "error: missing required tools: ${missing[*]}" >&2
echo " on NixOS: nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#openssh nixpkgs#curl" >&2
exit 1
fi
}
known_distro() {
[ -n "${IMAGE_URL[$DISTRO]:-}" ] \
|| { echo "error: unknown distro '$DISTRO' (known: ${ALL_DISTROS[*]})" >&2; exit 1; }
}
# --- run-dir bookkeeping ---------------------------------------------
# The marker lets prereqs/run/status/down in separate invocations find the VM
# that `up` started. The test cycles set RUN_DIR themselves and never write it.
_marker() { echo "${TMPDIR:-/tmp}/bot-bottle-install-test.$DISTRO.run"; }
_ensure_run_dir() {
if [ -z "$RUN_DIR" ]; then
RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")"
fi
mkdir -p "$RUN_DIR"
}
_load_run_dir() {
if [ -z "$RUN_DIR" ] && [ -f "$(_marker)" ]; then
RUN_DIR="$(cat "$(_marker)")"
fi
[ -n "$RUN_DIR" ] && [ -d "$RUN_DIR" ]
}
_ssh_key() { echo "$RUN_DIR/id_ed25519"; }
_overlay() { echo "$RUN_DIR/overlay.qcow2"; }
_seed() { echo "$RUN_DIR/seed.iso"; }
_pidfile() { echo "$RUN_DIR/qemu.pid"; }
_serial() { echo "$RUN_DIR/serial.log"; }
# --- ssh helpers -----------------------------------------------------
_ssh_opts() {
# No host-key pinning: the guest is thrown away every run.
printf '%s\0' \
-i "$(_ssh_key)" \
-p "$SSH_PORT" \
-o StrictHostKeyChecking=no \
-o UserKnownHostsFile=/dev/null \
-o LogLevel=ERROR \
-o ConnectTimeout=8 \
-o BatchMode=yes
}
guest() {
local -a opts
mapfile -d '' -t opts < <(_ssh_opts)
ssh "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1" "$@"
}
wait_for_ssh() {
local deadline=$(( SECONDS + BOOT_TIMEOUT ))
echo "== waiting for SSH on 127.0.0.1:$SSH_PORT (up to ${BOOT_TIMEOUT}s) =="
while [ "$SECONDS" -lt "$deadline" ]; do
if guest true 2>/dev/null; then
echo " guest is up"
return 0
fi
# Bail early if QEMU has died — no point waiting out the timeout.
if [ -f "$(_pidfile)" ] && ! kill -0 "$(cat "$(_pidfile)")" 2>/dev/null; then
echo "error: QEMU exited before SSH came up; see $(_serial)" >&2
return 1
fi
sleep 3
done
echo "error: timed out waiting for SSH; see $(_serial)" >&2
return 1
}
# --- image cache -----------------------------------------------------
_base_image() {
# One cached file per distro. NixOS is built (not downloaded), so it has a
# fixed cache name; the rest are keyed by the image's basename so a URL bump
# lands as a new cache entry rather than a stale hit.
if [ "$DISTRO" = nixos ]; then
echo "$CACHE_DIR/nixos-built.qcow2"
return
fi
local url="${IMAGE_URL[$DISTRO]}"
echo "$CACHE_DIR/$DISTRO-$(basename "$url")"
}
# NixOS has no upstream cloud qcow2; build one with nixos-generators and copy it
# out of the (immutable, GC-able) store into the cache.
build_nixos_image() {
local out; out="$(_base_image)"
[ -f "$out" ] && { echo "== nixos base image cached: $out =="; return 0; }
command -v nixos-generate >/dev/null 2>&1 || {
echo "error: 'nixos-generate' is required to build the NixOS image" >&2
echo " run inside: nix shell nixpkgs#nixos-generators nixpkgs#qemu nixpkgs#cloud-utils" >&2
return 1
}
echo "== building NixOS cloud image with nixos-generators (first run is slow) =="
local link="$CACHE_DIR/nixos-result"
nixos-generate -f qcow --system x86_64-linux \
-c "$_SCRIPT_DIR/linux-install-test-nixos.nix" -o "$link"
cp -L "$link"/*.qcow2 "$out"
rm -f "$link"
echo " built + cached: $out"
}
verify_checksum() {
local file="$1" sums_url="${SUM_URL[$DISTRO]}" base
base="$(basename "${IMAGE_URL[$DISTRO]}")"
if [ "${BB_TEST_SKIP_VERIFY:-0}" = "1" ] || [ -z "$sums_url" ]; then
echo " checksum: SKIPPED (${sums_url:+set BB_TEST_SKIP_VERIFY=0 to enable}${sums_url:-no sums URL for $DISTRO})" >&2
return 0
fi
local want
# Vendors publish either "HASH filename" tables or a bare "HASH" (or a
# "SHA256 (file) = HASH" BSD line, e.g. Fedora). Cover all three.
local sums; sums="$(curl -fsSL "$sums_url")"
want="$(printf '%s\n' "$sums" | awk -v f="$base" '
$0 ~ f && $1 ~ /^[0-9a-fA-F]{64}$/ { print $1; exit } # GNU "hash file"
$1=="SHA256" && $0 ~ f { gsub(/[()]/,""); print $NF; exit } # BSD "SHA256 (file) = hash"
')"
[ -z "$want" ] && want="$(printf '%s\n' "$sums" | awk '/^[0-9a-fA-F]{64}$/ {print $1; exit}')"
[ -n "$want" ] || { echo "error: could not find a sha256 for $base in $sums_url" >&2; return 1; }
local got; got="$(sha256sum "$file" | awk '{print $1}')"
if [ "$want" != "$got" ]; then
echo "error: checksum mismatch for $base" >&2
echo " want $want" >&2
echo " got $got" >&2
return 1
fi
echo " checksum: OK"
}
ensure_base_image() {
mkdir -p "$CACHE_DIR"
if [ "$DISTRO" = nixos ]; then
build_nixos_image
return
fi
local base; base="$(_base_image)"
if [ -f "$base" ]; then
echo "== base image cached: $base =="
return 0
fi
echo "== downloading $DISTRO cloud image =="
echo " ${IMAGE_URL[$DISTRO]}"
# Download to a temp name and rename on success so an interrupted pull
# never poisons the cache with a truncated image.
local tmp="$base.partial"
curl -fSL --retry 3 -o "$tmp" "${IMAGE_URL[$DISTRO]}"
verify_checksum "$tmp"
mv "$tmp" "$base"
echo " cached: $base"
}
# --- cloud-init seed -------------------------------------------------
make_seed() {
ssh-keygen -t ed25519 -N '' -f "$(_ssh_key)" -q
local pub; pub="$(cat "$(_ssh_key).pub")"
local user="${SSH_USER[$DISTRO]}"
local user_data="$RUN_DIR/user-data"
if [ "$user" = "root" ]; then
# NixOS' cloud-init lands the key straight on root; no sudo needed.
cat > "$user_data" <<EOF
#cloud-config
ssh_authorized_keys:
- $pub
EOF
elif [ "$DISTRO" = alpine ]; then
# Alpine needs three things the systemd distros don't: cloud-init locks
# the account (lock_passwd), but Alpine's non-PAM sshd then refuses
# pubkey auth for a locked account — so give it a throwaway password;
# and OpenRC does not auto-start sshd after the key is injected, so
# start it via runcmd.
cat > "$user_data" <<EOF
#cloud-config
users:
- name: $user
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/sh
lock_passwd: false
ssh_authorized_keys:
- $pub
chpasswd:
expire: false
list: |
$user:bbtest
packages:
- sudo
runcmd:
- [ sh, -c, "rc-service sshd start 2>/dev/null || true" ]
EOF
else
cat > "$user_data" <<EOF
#cloud-config
users:
- name: $user
sudo: ALL=(ALL) NOPASSWD:ALL
shell: /bin/sh
lock_passwd: true
ssh_authorized_keys:
- $pub
EOF
fi
# NoCloud wants a meta-data with an instance-id, or cloud-init may not treat
# the seed as a new instance (the Alpine nocloud image is strict about this).
printf 'instance-id: bbtest-%s\nlocal-hostname: bbtest-%s\n' "$DISTRO" "$DISTRO" \
> "$RUN_DIR/meta-data"
cloud-localds "$(_seed)" "$user_data" "$RUN_DIR/meta-data"
}
# --- commands --------------------------------------------------------
cmd_up() {
require_linux; require_kvm; require_tools; known_distro
_ensure_run_dir
ensure_base_image
# Throwaway copy-on-write overlay: the cached base is read-only backing,
# all guest writes land in the overlay, and `down` deletes it. Resize so
# pipx + a git build have headroom (cloud-init grows the rootfs to fit).
qemu-img create -q -f qcow2 -F qcow2 -b "$(_base_image)" "$(_overlay)" "$DISK"
make_seed
echo "== booting $DISTRO VM (mem=${MEM_MB}M cpus=$CPUS, ssh -> :$SSH_PORT) =="
qemu-system-x86_64 \
-machine accel=kvm -cpu host -smp "$CPUS" -m "$MEM_MB" \
-display none -daemonize \
-pidfile "$(_pidfile)" \
-serial "file:$(_serial)" \
-drive "file=$(_overlay),if=virtio,format=qcow2" \
-drive "file=$(_seed),if=virtio,format=raw" \
-netdev "user,id=n0,hostfwd=tcp:127.0.0.1:$SSH_PORT-:22" \
-device virtio-net-pci,netdev=n0
# Only publish the marker (so a later prereqs/run/down finds this VM) when
# we aren't inside a test cycle, which manages its own RUN_DIR + teardown.
[ "$IN_TEST" = 1 ] || echo "$RUN_DIR" > "$(_marker)"
wait_for_ssh
if [ "$IN_TEST" != 1 ]; then
echo "== VM is up. Prepare it with: BB_TEST_DISTRO=$DISTRO $0 prereqs (or go straight to 'run') =="
fi
}
# Install install.sh's toolchain prerequisites (python3 + git + pipx) into the
# running VM. This is what separates `test-ready` from `test`, and it is a
# distinct sub-command so a manual up/prereqs/run/down cycle is possible.
cmd_prereqs() {
require_linux
_load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
echo "== installing prerequisites (python3 + git + pipx) on $DISTRO =="
# Runs via the guest login shell; PREREQ is a client-side table value.
guest "${PREREQ[$DISTRO]}"
}
cmd_run() {
require_linux
_load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
local spec_env=""
[ -n "${BOT_BOTTLE_INSTALL_SPEC:-}" ] \
&& spec_env="BOT_BOTTLE_INSTALL_SPEC='$BOT_BOTTLE_INSTALL_SPEC' "
echo "== installing bot-bottle as ${SSH_USER[$DISTRO]} =="
# Capture install.sh's exit code and full output rather than aborting on
# non-zero: on a bare host a clean prerequisite *decline* is sound, so the
# verdict step — not set -e — decides the outcome.
local rc
set +e
if [ -n "${BB_TEST_INSTALL_URL:-}" ]; then
guest "curl -fsSL '$BB_TEST_INSTALL_URL' | ${spec_env}sh" 2>&1 | tee "$RUN_DIR/install.log"
else
# Feed THIS checkout's install.sh in over stdin — the same `curl … | sh`
# shape a real user runs, and nothing is staged in the guest to leak.
guest "${spec_env}sh -s" < "$_REPO_ROOT/install.sh" 2>&1 | tee "$RUN_DIR/install.log"
fi
rc="${PIPESTATUS[0]}"
set -e
printf '%s\n' "$rc" > "$RUN_DIR/install.rc"
echo "== install.sh exited $rc =="
[ "$IN_TEST" = 1 ] \
|| echo "== verdict anytime with: BB_TEST_DISTRO=$DISTRO $0 status =="
}
# Quietly report whether a runnable bot-bottle entry point exists for the
# guest user, checking the pipx/pip locations install.sh may leave off PATH.
entry_point_runnable() {
# shellcheck disable=SC2016 # expand in the GUEST shell.
guest '
for bb in "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle" "$(command -v bot-bottle 2>/dev/null)"; do
[ -n "$bb" ] && [ -x "$bb" ] || continue
# --help exits 0 before any DB/migration/network work; it is the
# cheapest proof the package imports and the shim runs. (bot-bottle
# has no --version: an unknown arg would die non-zero.)
"$bb" --help >/dev/null 2>&1 && exit 0
done
exit 1
' >/dev/null 2>&1
}
# `bot-bottle doctor` in the guest, classified. doctor's own exit code
# conflates "is the install sound" with "is a backend ready to run a bottle" —
# and inside a plain VM no backend can be ready (no nested KVM/Docker), so the
# raw exit code is non-zero by design. This separates the two: an unhandled
# traceback, or a missing python/config line, is an install defect and fails;
# a not-ready backend is reported, not fatal (unless BB_TEST_REQUIRE_BACKEND=1,
# for a nested-virt host that can actually satisfy it).
doctor_in_guest() {
local out rc=0 bad=0
out="$(mktemp "${TMPDIR:-/tmp}/bb-doctor.XXXXXX")"
# shellcheck disable=SC2016 # $HOME/$bb must expand in the GUEST shell.
guest '
for bb in bot-bottle "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle"; do
if command -v "$bb" >/dev/null 2>&1; then
case "$bb" in
bot-bottle) : ;;
*) echo " (not on PATH — running $bb directly, as install.sh advises)" ;;
esac
exec "$bb" doctor
fi
done
echo " no bot-bottle entry point found for this user" >&2
exit 1
' >"$out" 2>&1 || rc=$?
cat "$out"
# An unhandled exception is always an install/product defect, never an
# environment fact — doctor's non-zero exit alone would not distinguish it.
if grep -q 'Traceback (most recent call last)' "$out"; then
echo " doctor crashed (traceback above) — a defect, not a missing prerequisite" >&2
bad=1
fi
grep -qE '^ok: +python:' "$out" \
|| { echo " doctor never reported a usable python" >&2; bad=1; }
grep -qE '^ok: +config:' "$out" \
|| { echo " doctor never reported a usable config dir" >&2; bad=1; }
if [ "$rc" -ne 0 ] && ! grep -qE '^(fail|warn): +backend' "$out"; then
echo " doctor failed for something other than backend readiness" >&2
bad=1
fi
local backend_ready=1
grep -qE '^fail: +backend' "$out" && backend_ready=0
rm -f "$out"
[ "$bad" -eq 0 ] || return 1
if [ "${BB_TEST_REQUIRE_BACKEND:-0}" = "1" ] && [ "$backend_ready" -eq 0 ]; then
echo " backend is not ready and BB_TEST_REQUIRE_BACKEND=1 — failing" >&2
return 1
fi
if [ "$backend_ready" -eq 1 ]; then
echo " doctor: install sound; a backend is ready"
else
echo " doctor: install sound; no backend ready (expected in a plain VM — install gate only)"
fi
return 0
}
# The prerequisite-decline messages install.sh prints via die(). On a bare host
# ANY of these means install.sh correctly refused rather than half-installing —
# a sound outcome for `test`.
PREREQ_ERR_RE='is required but was not found|or newer is required|git is required to install from|neither pipx nor a usable|externally managed \(PEP 668\)|is not on PATH'
# The verdict: is the install sound? Reads install.sh's captured exit code and
# output (from cmd_run) plus the guest's resulting state.
# - installed & runnable -> the verdict is doctor's soundness classification.
# - no entry point, but a recognized prerequisite decline, and declines are
# allowed (bare `test`, _REQUIRE_INSTALL=0) -> sound.
# - anything else -> not sound.
install_verdict() {
local rc=""
[ -f "$RUN_DIR/install.rc" ] && rc="$(cat "$RUN_DIR/install.rc")"
if [ "${rc:-1}" = 0 ] && entry_point_runnable; then
echo "doctor (in guest):"
doctor_in_guest
return $?
fi
if [ "${_REQUIRE_INSTALL:-0}" != "1" ] \
&& [ -n "$rc" ] && [ "$rc" != 0 ] \
&& [ -f "$RUN_DIR/install.log" ] \
&& grep -Eiq "$PREREQ_ERR_RE" "$RUN_DIR/install.log"; then
echo " install.sh declined with an actionable prerequisite error (rc=$rc)"
echo " — sound on a bare host; run 'test-ready' (or 'prereqs') to install."
return 0
fi
if [ "${_REQUIRE_INSTALL:-0}" = "1" ]; then
echo " prerequisites were provisioned, but install.sh left no runnable entry point (rc=${rc:-?})" >&2
else
echo " install.sh neither installed nor gave a recognized prerequisite error (rc=${rc:-?})" >&2
fi
return 1
}
cmd_status() {
require_linux
if ! _load_run_dir; then
echo "vm: no running VM for $DISTRO"
return 0
fi
if [ -f "$(_pidfile)" ] && kill -0 "$(cat "$(_pidfile)")" 2>/dev/null; then
echo "vm: $DISTRO running (pid $(cat "$(_pidfile)"), ssh :$SSH_PORT)"
else
echo "vm: $DISTRO run-dir present but QEMU not alive"
return 1
fi
if install_verdict; then
echo "OK[$DISTRO]: the install is sound"
return 0
fi
echo "FAIL[$DISTRO]: the install is not sound (see above)" >&2
return 1
}
cmd_down() {
require_linux
if ! _load_run_dir; then
echo "$DISTRO: nothing running"
return 0
fi
if [ -f "$(_pidfile)" ]; then
local pid; pid="$(cat "$(_pidfile)")"
if kill -0 "$pid" 2>/dev/null; then
kill "$pid" 2>/dev/null || true
for _ in 1 2 3 4 5; do kill -0 "$pid" 2>/dev/null || break; sleep 1; done
kill -9 "$pid" 2>/dev/null || true
fi
fi
# Deleting the overlay + seed is the reset; the read-only base stays cached.
rm -f "$(_overlay)" "$(_seed)" "$(_ssh_key)" "$(_ssh_key).pub" \
"$RUN_DIR/user-data" "$RUN_DIR/meta-data" \
"$RUN_DIR/install.rc" "$RUN_DIR/install.log" "$(_serial)"
# Only remove a scratch dir we created (leave a user-provided one alone).
[ -n "${BB_TEST_RUN_DIR:-}" ] || rmdir "$RUN_DIR" 2>/dev/null || true
rm -f "$(_marker)"
echo "removed $DISTRO VM and its overlay — install surface is clean."
}
cmd_ssh() {
require_linux
_load_run_dir || { echo "error: no running VM for $DISTRO" >&2; return 1; }
local -a opts
mapfile -d '' -t opts < <(_ssh_opts)
exec ssh -t "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1"
}
# Teardown half of the test cycles, armed the moment the VM exists so a failure
# or a Ctrl-C still leaves nothing running.
_test_teardown() {
local rc=$?
trap - EXIT INT TERM
if [ "${BB_TEST_KEEP:-0}" = "1" ]; then
echo
echo "== [$_STEPS/$_STEPS] down: SKIPPED (BB_TEST_KEEP=1) =="
echo " VM still up; remove with: BB_TEST_DISTRO=$DISTRO $0 down"
echo " ssh in with: BB_TEST_RUN_DIR=$RUN_DIR BB_TEST_DISTRO=$DISTRO $0 ssh"
exit "$rc"
fi
echo
echo "== [$_STEPS/$_STEPS] down =="
cmd_down || rc=1
if [ "$rc" -eq 0 ]; then
echo
echo "PASS[$DISTRO]: $_PASS_CLAIM"
else
echo
echo "FAIL[$DISTRO]: see above (the VM was torn down regardless)." >&2
fi
exit "$rc"
}
# The two variants differ only in whether the prerequisites get installed
# before install.sh runs, which is exactly the question each one asks:
#
# test a bare host — assert install.sh is SOUND (installs cleanly, or
# declines with an actionable prerequisite error).
# test-ready prerequisites satisfied — assert install.sh actually LANDS.
_test_cycle() {
local with_prereqs="$1"
require_linux; require_kvm; require_tools; known_distro
IN_TEST=1
RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")"
# Arm teardown BEFORE cmd_up: its wait_for_ssh can fail after QEMU is
# already running (e.g. a guest that never opens SSH), and without the trap
# in place that would leak the VM.
trap _test_teardown EXIT INT TERM
echo "== [1/$_STEPS] up ($DISTRO) =="
cmd_up
local step=2
if [ "$with_prereqs" = 1 ]; then
echo
echo "== [$step/$_STEPS] prereqs =="
cmd_prereqs || { echo "error: could not install prerequisites (see above)." >&2; return 1; }
step=$(( step + 1 ))
fi
echo
echo "== [$step/$_STEPS] run =="
cmd_run
step=$(( step + 1 ))
echo
echo "== [$step/$_STEPS] verdict =="
# install.sh exits 0 even when doctor reports unmet prerequisites, so the
# install succeeding is not the verdict — this is.
cmd_status || {
echo "error: the install is not sound for $DISTRO (see above)." >&2
echo " re-run with BB_TEST_KEEP=1 to keep the VM and dig in." >&2
return 1
}
}
cmd_test() {
_STEPS=4
_PASS_CLAIM="on a bare $DISTRO host, install.sh behaves soundly."
_test_cycle 0
}
cmd_test_ready() {
_STEPS=5
_PASS_CLAIM="a $DISTRO host with prerequisites satisfied installs bot-bottle cleanly."
# Prerequisites are provisioned, so a graceful decline is no longer an
# acceptable outcome — the install must actually land.
_REQUIRE_INSTALL=1
_test_cycle 1
}
cmd_test_all() {
require_linux; require_kvm; require_tools
local -a passed=() failed=()
local port="$SSH_PORT"
local script
script="$_SCRIPT_DIR/$(basename "${BASH_SOURCE[0]}")"
# Every distro × both variants. Each cell gets its own forwarded port so a
# leftover from a prior cell can't collide, and its own subshell so one
# cell's failure (or teardown trap) doesn't abort the matrix.
for sub in test test-ready; do
for d in "${ALL_DISTROS[@]}"; do
echo
echo "########################################################"
echo "# $d ($sub)"
echo "########################################################"
if ( DISTRO="$d" SSH_PORT="$port" \
BB_TEST_DISTRO="$d" BB_TEST_SSH_PORT="$port" \
bash "$script" "$sub" ); then
passed+=("$d/$sub")
else
failed+=("$d/$sub")
fi
port=$(( port + 1 ))
done
done
echo
echo "== matrix summary =="
echo " PASS: ${passed[*]:-(none)}"
echo " FAIL: ${failed[*]:-(none)}"
[ "${#failed[@]}" -eq 0 ]
}
case "${1:-}" in
test) cmd_test ;;
test-ready) cmd_test_ready ;;
test-all) cmd_test_all ;;
up) cmd_up ;;
prereqs) cmd_prereqs ;;
run) cmd_run ;;
status) cmd_status ;;
down) cmd_down ;;
ssh) cmd_ssh ;;
*) echo "usage: $0 {test|test-ready|test-all|up|prereqs|run|status|down|ssh} (distro via BB_TEST_DISTRO)" >&2; exit 2 ;;
esac

Some files were not shown because too many files have changed in this diff Show More