feat(prd-0081): reprovision CA, git-gate, and egress tokens on gateway bring-up

On a Firecracker gateway cold boot, reconcile every live agent VM against
the fresh gateway: push the new mitmproxy CA into each agent's trust store,
re-provision git-gate repos/creds from the persisted upstreams snapshot,
and restore egress tokens. Per-bottle failures are logged and skipped so
one unreachable VM does not block the rest.

New modules / changes:
- backend/firecracker/reconcile.py: attach_bottled_agents_to_gateway,
  _push_ca, _reprovision_git_gate, _guest_ip_from_config
- git_gate/provision.py: write upstreams.json after key provisioning so
  the bring-up reconcile can reconstruct the upstream table without the
  manifest
- backend/firecracker/infra.py: call attach_bottled_agents_to_gateway in
  the cold-boot branch of ensure_running()
- backend/base.py: no-op default on BottleBackend
- backend/firecracker/consolidated_launch.py: remove superseded
  _reprovision_running_bottles / _guest_ip_from_config
- orchestrator: OrchestratorCore.update_agent_secret + POST
  /bottles/<id>/secret + client.update_agent_secret (single-secret
  in-place update, the reusable primitive from the 2026-07-26 hotfix)
- tests: 15 new tests in test_firecracker_reconcile.py; updated
  test_firecracker_infra.py cold-boot cases; stale FC reprovision tests
  removed from test_backend_secret_reprovision.py

Closes #516.

docs/ci.md

@@ -45,23 +45,6 @@ report; macOS reports advisory coverage in its own job. The macOS infra
 container is a singleton, so its job uses a concurrency group and always tears
 the service down.
 
-## Artifact publication and promotion
-
-[`publish-artifacts.yml`](../.gitea/workflows/publish-artifacts.yml) is the
-manual deployment-test boundary. Dispatch it with any branch, tag, or commit;
-it resolves that input once, publishes the orchestrator, gateway, Claude,
-Codex, Pi, and Firecracker artifacts, then publishes only the wheel verified
-against those identities. Install the reported commit with
-`BOT_BOTTLE_REF=<sha> sh install.sh`; the installer warns that snapshots may
-not have completed release qualification.
-
-[`release.yml`](../.gitea/workflows/release.yml) handles promotion tags.
-`vX.Y.Z-rc.N` must be reachable from `staging`; `vX.Y.Z` must be reachable
-from `production`. It runs the complete pre-release workflow, ensures that the
-same commit bundle is published, verifies a clean install, creates the
-qualified release pointer, and advances the matching channel. Promotion
-reuses immutable bundle bytes and never rebuilds them.
-
 ## Scheduled canary
 
 [`.gitea/workflows/canaries.yml`](../.gitea/workflows/canaries.yml) runs weekly

docs/prds/0081-reprovision-gateway-state-on-bringup.md

@@ -0,0 +1,117 @@
+# PRD 0081: Reprovision gateway-dependent state on gateway bring-up
+
+- **Status:** Active
+- **Author:** claude
+- **Created:** 2026-07-26
+- **Issue:** #516
+
+## Summary
+
+When the gateway is (re)built, reconcile every already-running bottle against the
+fresh gateway instead of persisting the gateway's state. On a gateway cold boot,
+the gateway reconciles all live bottles in one flow: **replace** each agent's
+trusted CA with the freshly-minted gateway CA, **re-provision** each bottle's
+git-gate repos + creds onto the gateway, and **restore** each bottle's egress
+tokens. One mechanism across all three services; the CA rotates for free on every
+bring-up.
+
+## Problem
+
+A gateway rebuild/restart silently breaks every already-running bottle:
+
+- **CA (#510).** The gateway's mitmproxy mints a new CA on a fresh rootfs; agents
+  still trust the old one, so egress fails TLS verification (`SSL certificate
+  verification failed`).
+- **git-gate (#512).** Per-bottle bare repos (`/git/<id>`) + deploy creds
+  (`/git-gate/creds/<id>`) live in the gateway's ephemeral rootfs; a rebuild wipes
+  them and the agent 404s on fetch/push.
+
+Both are the same root cause: per-bottle gateway-dependent state is provisioned
+**once, at bottle launch**, and nothing restores it for already-running bottles
+when the gateway comes back. The existing launch-time reprovision
+(`reprovision_bottles`) restores **only** egress tokens, and only as a side effect
+of the *next* launch.
+
+Persisting the state (a host bind-mount / a per-VM data volume per service) was
+prototyped and rejected: it differs per service (a volume for the CA, another for
+git-gate, reprovision for tokens), it pins the CA static forever (no rotation),
+and it adds volume surface that `docker volume prune` / a wiped cache can silently
+destroy (the original #450 failure mode).
+
+## Goals / Success Criteria
+
+- A gateway (re)boot restores **all** running bottles' gateway-dependent state
+  with no manual step and no relaunch — the agent's next egress / fetch / push
+  just works.
+- **One** reconcile flow covering CA, git-gate, and egress tokens, rather than a
+  different mechanism per service.
+- The CA **rotates** on every gateway bring-up (no long-lived CA), distributed to
+  running agents by the same reconcile.
+- Per-bottle failures are tolerated: one unreachable or malformed bottle does not
+  block the others or the gateway coming up.
+- **All backends** (Firecracker, docker, macOS) reconcile through the *same*
+  contract — an abstract method on the backend base class, so a new backend
+  cannot forget to implement it and none drifts onto a bespoke mechanism.
+
+## Non-goals
+
+- Deliberate mid-session CA rotation *without* a gateway restart — `rotate_ca`
+  stays for that operator action.
+- Changing source-IP attribution, `/resolve`, or the plane split (#469).
+
+## Design
+
+**The contract — `attach_bottled_agents_to_gateway()` on the backend ABC.**
+Reconciling running bottles against the current gateway is a backend
+responsibility (only the backend can enumerate its agents and reach them —
+firecracker over SSH, docker/macOS over `exec`/`cp`), so it is an
+`@abc.abstractmethod` on `BottleBackend` (`backend/base.py`). Every backend
+implements it; the host calls it whenever the gateway is (re)brought up. This is
+what makes the fix cross-backend by construction rather than a per-backend
+follow-up. It reprovisions **all** registered bottles' gateway-dependent state:
+CA, git-gate, and egress tokens.
+
+**Trigger — the gateway bring-up path.** The host calls
+`attach_bottled_agents_to_gateway()` only when the gateway was actually
+(re)brought up — the cold-boot branch of the infra bring-up (e.g.
+`FirecrackerInfraService.ensure_running` after it boots a fresh pair), never on an
+adopt of a healthy, current gateway (state intact). So it fires exactly when the
+gateway was (re)booted — including orchestrator restarts, since the pair boots
+together — and there is no bare-restart path that bypasses bring-up.
+
+**Per-backend implementation.** Each backend's `attach_bottled_agents_to_gateway`
+enumerates its live bottles and, for each, reconciles the three services against
+the current gateway. The firecracker implementation, once the gateway VM is up
+and its CA is available:
+
+1. Map each live bottle's guest IP → `bottle_id` from the orchestrator registry
+   (`list_bottles`).
+2. Install the **current** shared git-gate hooks (`git_gate_render_hook` /
+   `git_gate_render_access_hook`) into the fresh gateway once — rendered from
+   code, never from a bottle's possibly-stale state dir.
+3. For each live agent VM (enumerated from its run dir):
+   - **CA:** SSH the current gateway CA into the agent's trust store and run
+     `update-ca-certificates` (unconditional replace — there is one gateway, so no
+     fingerprint match is needed).
+   - **git-gate:** rebuild the bottle's upstreams from its persisted git-gate
+     state dir (deploy key, known_hosts, upstream URL) and re-init its bare repos
+     + per-repo creds under `/git/<bottle_id>`.
+   - **egress token:** read the agent's `ENV_VAR_SECRET` and feed
+     `reprovision_bottles`, restoring the orchestrator's in-memory tokens.
+4. Every per-bottle step is wrapped so one failure is logged and skipped.
+
+**Retire the persistence prototype.** No CA data volume, no git-gate data volume
+(the abandoned PRs #511 / #513). The launch-time `_reprovision_running_bottles`
+call folds into this bring-up reconcile, so egress tokens are restored on the same
+cold-boot trigger (an adopt needs no restore — the orchestrator never restarted).
+
+**CA rotation.** Because the gateway rootfs is ephemeral, every cold boot mints a
+fresh CA; reconcile is what distributes it, so a routine gateway rebuild doubles
+as a CA rotation with zero extra machinery.
+
+## Open questions
+
+- Docker's existing host-bind-mounted CA (`host_gateway_ca_dir`): once docker's
+  `attach_bottled_agents_to_gateway` pushes the CA to running agents on bring-up,
+  the bind-mount is redundant — drop it (so docker rotates like firecracker) or
+  keep it as belt-and-suspenders? Leaning drop, for one behaviour across backends.

docs/prds/0083-packaged-infra-artifacts.md

@@ -1,374 +0,0 @@
-# PRD 0083: Packaged infrastructure artifacts
-
-- **Status:** Active
-- **Author:** Codex
-- **Created:** 2026-07-27
-- **Issue:** #536
-
-## Summary
-
-Publish the orchestrator and gateway infrastructure before packaging
-bot-bottle, record their immutable identities in the application package, and
-make every production backend acquire those exact artifacts instead of
-building infrastructure during bottle startup. Docker and Apple Container pull
-digest-pinned OCI images; Firecracker pulls checksum-verified rootfs artifacts
-selected by the same packaged release manifest.
-
-## Problem
-
-Starting a bottle currently invokes an infrastructure build on Docker and
-Apple Container even when a healthy orchestrator is already running. Both
-infra composers call `ensure_built()` before their health/currentness checks,
-and both implementations unconditionally invoke their OCI builder. Layer
-caching can make this less expensive, but startup still depends on a working
-builder, build context, base-image registry, and package installation network.
-
-The Apple Container orchestrator also bind-mounts the installed source over the
-package baked into its image. A pinned image alone would therefore not pin the
-code that actually runs.
-
-Firecracker already downloads fixed rootfs artifacts, but it derives their
-versions from the package contents at launch. There is no package-level record
-that says which already-published infrastructure release belongs to an
-installed bot-bottle release.
-
-The result is three different delivery contracts:
-
-- Docker and Apple Container build mutable `:latest` images on the launch host.
-- Firecracker computes an artifact version locally and downloads it.
-- An installed wheel contains Dockerfiles specifically so production startup
-  can rebuild its own infrastructure.
-
-That makes startup slower and less reliable, prevents a packaged application
-from identifying the infrastructure it was tested with, and weakens the
-supply-chain boundary between release production and runtime.
-
-## Goals / Success criteria
-
-- A packaged bot-bottle release contains one validated release manifest with
-  immutable orchestrator and gateway identities.
-- Docker and Apple Container acquire and run the manifest's digest-pinned OCI
-  images without invoking `docker build` or `container build`.
-- The shipped Claude, Codex, and Pi agent images are built, smoke-tested,
-  published, and selected by digest as part of the same commit bundle.
-- Firecracker acquires the manifest's versioned, checksum-verified
-  orchestrator and gateway rootfs artifacts rather than deriving versions at
-  launch.
-- All three backends expose the same backend-neutral `ensure_available`
-  lifecycle contract.
-- A healthy running orchestrator is retained only when its recorded release
-  identity matches the installed package.
-- Apple Container runs the package baked into the orchestrator image; no host
-  source overlays production code.
-- Missing, mutable, malformed, or internally inconsistent release metadata
-  fails closed before infrastructure starts.
-- Source-checkout development retains an explicit local-build mode.
-- Publishing produces the infrastructure first and the installable Python
-  package last, so the package cannot name artifacts that were not published.
-- Any branch, tag, or commit can be resolved once to a full source SHA and
-  published as one immutable, commit-addressed artifact bundle.
-- The installer defaults to the production channel, can select staging or an
-  exact commit, and warns for commit snapshots that have not been qualified as
-  a release.
-- Staging and production releases promote the same tested artifact bytes;
-  promotion never rebuilds them.
-- A release is published only after the complete pre-release suite and a clean
-  install from the published bundle succeed.
-
-## Non-goals
-
-- Prebuilding user-supplied Dockerfiles. Custom agent images continue to build
-  through the backend-specific agent-image paths.
-- Installing Docker, Apple Container, Firecracker, or other host prerequisites.
-- Replacing Gitea's OCI or generic-package registries.
-- Adding image signing or a transparency log. Immutable digests and existing
-  Firecracker SHA-256 verification are the integrity boundary for this version.
-- Forcing a network download when an immutable, verified artifact is already
-  cached locally. "Pull" means acquire the packaged identity, with safe cache
-  reuse.
-- Supporting live source overlays in packaged production installs.
-- Defining the project's version-numbering policy beyond stable
-  `vX.Y.Z` and staging `vX.Y.Z-rc.N` tag shapes.
-
-## Design
-
-### Release manifest
-
-`bot_bottle/release-manifest.json` is generated during package production and
-shipped as package data:
-
-```json
-{
-  "schema": 1,
-  "source_commit": "<40 lowercase hex>",
-  "oci": {
-    "orchestrator": "gitea.dideric.is/didericis/bot-bottle-orchestrator@sha256:<64 hex>",
-    "gateway": "gitea.dideric.is/didericis/bot-bottle-gateway@sha256:<64 hex>",
-    "agent_claude": "gitea.dideric.is/didericis/bot-bottle-claude@sha256:<64 hex>",
-    "agent_codex": "gitea.dideric.is/didericis/bot-bottle-codex@sha256:<64 hex>",
-    "agent_pi": "gitea.dideric.is/didericis/bot-bottle-pi@sha256:<64 hex>"
-  },
-  "firecracker": {
-    "orchestrator": {
-      "version": "<artifact version>",
-      "sha256": "<rootfs.ext4.gz sha256>"
-    },
-    "gateway": {
-      "version": "<artifact version>",
-      "sha256": "<rootfs.ext4.gz sha256>"
-    }
-  }
-}
-```
-
-The runtime loader accepts only schema 1, digest-qualified OCI references,
-non-empty Firecracker versions, 64-character lowercase SHA-256 values, and a
-full source commit. It returns immutable value objects rather than passing raw
-dictionaries through backend code.
-
-The repository carries no pretend production pins. A source checkout or ad-hoc
-Git wheel carries a `development: true` marker and selects local infrastructure
-builds, preserving the contributor and current `install.sh` paths. The release
-packaging workflow must replace that marker with the validated schema above;
-it refuses mutable or incomplete release inputs.
-
-### Packaging order
-
-The release job operates on one tested commit:
-
-1. Build and smoke-test the multi-architecture orchestrator, gateway, Claude,
-   Codex, and Pi OCI images.
-2. Push them and resolve their registry manifest digests.
-3. Build Firecracker's orchestrator and gateway rootfs artifacts from the same
-   source and pinned OCI bases.
-4. Publish both generic-package artifacts and retain their versions and
-   compressed-file checksums.
-5. Generate `release-manifest.json` from those published identities.
-6. Build the wheel with that manifest. Do not produce a source distribution:
-   rebuilding an sdist outside this release boundary could replace the pins
-   with the development manifest.
-7. Install the wheel in a clean environment and assert that every manifest
-   identity is valid and retrievable.
-8. Publish only that verified wheel.
-
-The build hook receives the manifest as a file input. It must not query a
-registry or choose versions itself: package builds stay deterministic and
-network-independent once their release inputs exist.
-
-### Commit-addressed bundle index
-
-Every publication resolves its requested branch, tag, or commit to one
-40-character source SHA before doing any work. The publisher writes an
-immutable external bundle index under that SHA. The index contains:
-
-- schema version and source SHA;
-- wheel URL and SHA-256;
-- orchestrator and gateway OCI digest references;
-- Claude, Codex, and Pi agent OCI digest references;
-- Firecracker package versions and compressed-file SHA-256 values;
-- producing workflow/run identity and publication timestamp; and
-- producing workflow provenance. Qualification is recorded separately in an
-  immutable tag pointer so the commit bundle itself never changes.
-
-The embedded wheel manifest contains the runtime subset of the same identities.
-Publication fails if the embedded manifest and external index differ.
-
-Bundle coordinates are immutable. Re-running publication for a SHA verifies
-and reuses a byte-identical complete bundle; it never replaces one artifact or
-mixes outputs from different runs. A partial existing bundle is an error and
-requires an explicit administrative cleanup before retrying.
-
-Commit bundles are durable release inputs, not expiring CI artifacts. The
-project may garbage-collect unqualified snapshots only under a documented
-retention policy; qualified staging and production bundles are retained.
-
-### Branches, tags, and promotion
-
-The protected promotion topology is:
-
-```text
-feature -> main -> staging -> production
-                       |           |
-                 vX.Y.Z-rc.N     vX.Y.Z
-```
-
-Changes reach `staging` and `production` through promotion pull requests; those
-branches do not accept direct development commits. A staging tag must point to
-a commit reachable from `staging` and uses `vX.Y.Z-rc.N`. A production tag must
-point to a commit reachable from `production` and uses `vX.Y.Z`.
-
-Channels are small, mutable pointers to immutable qualified bundles:
-
-- `production` points to the newest promoted stable tag;
-- `staging` points to the newest promoted release-candidate tag; and
-- `main` is not a release channel; its commits are installable snapshots.
-
-Channel updates are serialized and monotonic. Moving a channel to an older
-release requires a distinct, explicit rollback operation that records the
-reason and target. Deleting or moving a Git tag does not mutate a published
-bundle or channel silently.
-
-Promotion reuses the exact bundle selected by source SHA. It does not rebuild
-the wheel, OCI images, or Firecracker artifacts. Thus production runs the same
-bytes qualified in staging.
-
-### Publication workflows
-
-`publish-artifacts` is manually dispatchable for any branch, tag, or commit.
-It resolves the input to a source SHA, checks out that SHA, builds and publishes
-all infrastructure artifacts, generates the bundle index and embedded
-manifest, builds the wheel, installs the wheel in a clean environment, and
-publishes the verified wheel. It does not create a release or update staging or
-production. Its output is an unqualified snapshot and publication is
-serialized per source SHA.
-
-After required CI succeeds, `main` may invoke the same publication operation
-automatically. Manual publication remains available so a branch can be tested
-before merge. Automatic and manual runs converge on the same idempotent bundle.
-
-The release workflow is triggered by an eligible staging or production tag:
-
-1. Resolve and validate the tag, source SHA, tag shape, and branch reachability.
-2. Run the complete pre-release suite against that SHA, including Docker,
-   Firecracker, and Apple Container.
-3. After the suite passes, ensure the immutable commit bundle exists. Build it
-   then if absent; otherwise verify every existing identity and checksum.
-4. Install using the published installer and bundle index on clean supported
-   hosts, and run the post-install smoke test against the acquired artifacts.
-5. Publish an immutable qualified tag pointer, create the Gitea release, and
-   advance the matching channel pointer.
-
-No release artifact or channel update occurs before qualification succeeds.
-Jobs use per-SHA and per-channel concurrency locks so concurrent dispatches
-cannot publish twice or race a promotion.
-
-### Installer selection and warnings
-
-The installer installs the latest `production` channel by default. It accepts:
-
-- `BOT_BOTTLE_CHANNEL=production` or `staging` to select the latest release in
-  that channel;
-- `BOT_BOTTLE_REF=<40-character-sha>` to install an exact commit bundle; and
-- `BOT_BOTTLE_VERSION=<tag>` to install an exact qualified release.
-
-Selectors are mutually exclusive. Branch names are accepted by the publication
-workflow but not by the installer because they are mutable; callers resolve
-and publish them first, then install the reported SHA.
-
-An exact-commit install prints a prominent warning that the snapshot may not
-have passed release qualification, even if the same SHA is later associated
-with a tag. A tag or channel install suppresses that warning only when its
-validated pointer is marked qualified and selects the same immutable bundle
-SHA.
-
-The installer downloads the wheel named by the external index, verifies its
-SHA-256 before invoking pipx or pip, and reports the selected channel/tag,
-source SHA, and bundle identity. It never builds a Git checkout for a packaged
-install and never trusts a mutable channel response without resolving it to an
-immutable bundle index.
-
-### Backend-neutral lifecycle
-
-Rename the host-side control-plane lifecycle operation from `ensure_built()` to
-`ensure_available()`. The production meaning is "make the package-selected
-artifact locally available and verified." Infrastructure composers call it
-before starting either plane.
-
-The gateway receives the same contract so startup does not continue to rebuild
-the adjacent half of the fixed infrastructure pair.
-
-A source checkout is itself a development boundary and selects the existing
-builders. `BOT_BOTTLE_INFRA_BUILD=local` provides the same explicit override
-for release debugging. Both paths deliberately bypass release metadata.
-Packaged production startup never silently falls back from a failed pull to a
-local build.
-
-### Docker
-
-The Docker backend runs `docker pull` with each digest-qualified manifest
-reference. Docker may reuse its content-addressed local store. Containers are
-created from the immutable reference, and the running container's image ID is
-compared with the locally resolved ID for currentness.
-
-The orchestrator source-hash label is replaced by a release-identity label.
-The label is diagnostic; the image ID comparison remains authoritative.
-
-### Apple Container
-
-The Apple Container backend pulls the same multi-architecture OCI references.
-Because Apple Container's accepted syntax for pull, tag, inspect, and run is
-not identical to Docker's, one utility owns normalization from a digest
-reference to the local runnable name. Integration coverage on the macOS runner
-guards that adapter.
-
-Production orchestrator launch removes the build-root bind mount, `PYTHONPATH`,
-and `BOT_BOTTLE_SOURCE_HASH`. It runs only the code baked into the pulled
-image. Currentness is based on the packaged release identity/image ID.
-
-### Firecracker
-
-The publishing tool may continue computing content-derived artifact versions;
-that is a producer concern. Runtime launch instead reads the version and
-expected compressed checksum from the release manifest.
-
-The artifact cache validates the packaged expected checksum even when a
-previous `.verified` marker exists. The marker records the checksum it
-validated, rather than an unqualified `ok`, so changing package metadata cannot
-adopt an artifact verified against a different expectation.
-
-The existing candidate-directory and explicit local-build paths remain for CI
-and development. Candidate metadata must match the packaged identity unless
-the caller selected local development mode.
-
-### Overrides and failure behavior
-
-`BOT_BOTTLE_ORCHESTRATOR_IMAGE` and `BOT_BOTTLE_GATEWAY_IMAGE` remain useful
-for testing but production accepts them only when they are digest-qualified.
-Mutable tag overrides require explicit local-build mode.
-
-Artifact acquisition errors identify the packaged reference and backend. No
-backend catches an acquisition failure and rebuilds from local source.
-
-## Testing strategy
-
-- Unit-test manifest parsing, immutability validation, missing-manifest errors,
-  package-data inclusion, and deterministic build-hook injection.
-- Assert Docker and Apple production preparation pulls and never builds.
-- Assert explicit local mode retains existing build behavior.
-- Assert the Apple orchestrator command has no source bind mount or
-  `PYTHONPATH`.
-- Assert running-image mismatches recreate infrastructure while matching,
-  healthy instances are retained.
-- Assert Firecracker uses packaged versions/checksums and rejects mismatches in
-  downloaded, cached, and candidate artifacts.
-- Extend wheel-install coverage to inspect the installed manifest.
-- Run the full unit and Docker integration gates; run advisory Firecracker and
-  macOS integration before publishing a release.
-- Test ref resolution, tag-shape and branch-reachability enforcement, bundle
-  idempotency, partial-publication rejection, and per-SHA concurrency.
-- Test installer channel, tag, and exact-commit selection; checksum rejection;
-  mutually exclusive selectors; and snapshot warning behavior.
-- Test that release qualification happens before publication and that a failed
-  suite or clean-install smoke test cannot create a release or move a channel.
-- Test promotion reuses the qualified bundle byte-for-byte and that monotonic
-  channel movement rejects an accidental rollback.
-
-## Rollout
-
-Land runtime, installer, bundle publication, and release qualification support
-together, but do not move the production installer away from its current
-source-install path until an end-to-end staging release has passed on Docker,
-Firecracker, and Apple Container.
-
-Create and protect `staging` and `production`, then promote one commit from
-`main` through both branches. Publish its commit bundle, qualify an RC tag,
-perform clean installs through the staging channel, promote the same SHA, and
-qualify a stable tag. Until that first stable qualification exists, the
-installer's production default fails closed because there is no production
-pointer. The stable promotion must demonstrate that every production artifact
-identity matches the staging-qualified bundle.
-
-Existing source checkouts use explicit local mode. Existing mutable local
-images are ignored by production acquisition and may be pruned independently.
-Snapshot retention and rollback procedures must be documented before automatic
-publication from every successful `main` commit is enabled.

docs/research/testing-clean-install-on-linux.md

@@ -1,183 +0,0 @@
-# Testing a clean bot-bottle install on Linux
-
-How do you exercise `install.sh` the way a brand-new user would — on a
-pristine Linux environment you can throw away afterward — *without*
-polluting your daily-driver host, and across the several package-management
-regimes Linux fragments into? This is the Linux counterpart to
-[`testing-clean-install-on-macos.md`](testing-clean-install-on-macos.md);
-the conclusion is different because Linux gives us a boundary macOS doesn't.
-
-## Summary
-
-On macOS the honest options were a throwaway user or a VM, and the throwaway
-user won on pragmatics (nested virtualization is gated to M3+). On Linux the
-calculus flips: a **disposable KVM virtual machine, booted from a distro
-cloud image and deleted per run, is both the cleanest boundary and the one
-that lets a single harness cover Ubuntu, Fedora, Arch, Alpine, and NixOS**.
-The host already requires KVM for the Firecracker backend, so the VM is cheap
-here.
-
-The harness lives at [`scripts/linux-install-test.sh`](../../scripts/linux-install-test.sh).
-Per run it caches one read-only base image, boots a throwaway copy-on-write
-overlay (`qemu-img create -b base`), installs the distro's prerequisites,
-pipes *this checkout's* `install.sh` into the guest exactly as `curl … | sh`
-would, asserts the CLI installed, and deletes the overlay — the Linux
-equivalent of `docker run --rm`, for a whole machine.
-
-## Why a VM, not a container or a throwaway user
-
-| Mechanism | Why it's the wrong boundary here |
-|---|---|
-| **Container** (`docker run --rm`) | Shares the host kernel and ships a deliberately minimal userland — no systemd, a stubbed-out package manager story, and (crucially) it doesn't reproduce the *externally-managed Python* (PEP 668) that real desktop/server installs put in front of the user. It tests "does install.sh run in a container," not "does it run on a real distro." |
-| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
-| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
-
-## Two variants: `test` (bare host) and `test-ready` (prepared host)
-
-`install.sh` never installs a backend, and never installs its own toolchain
-prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
-runs `doctor`, which *reports* what's missing
-([`install.sh`](../../install.sh) header,
-[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
-That leaves two distinct things worth testing, split into two subcommands that
-mirror the macOS harness's `test` / `test-ready` convention (there the split is
-the backend service; here it is the toolchain the installer needs):
-
-- **`test`** — `install.sh` runs on the **bare cloud image**, prerequisites and
-  all left as the vendor ships them. This exercises `install.sh`'s own
-  prerequisite-guard logic — the entire first half of the script (python
-  version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
-- **`test-ready`** — the harness installs python3 + git + pipx first (the
-  `prereqs` step), then runs `install.sh`. This is the *prepared-host happy
-  path*: does a clean install actually land and produce a working CLI?
-
-**Pass criteria differ by variant:**
-
-| Variant | PASS when |
-|---|---|
-| `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
-| `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
-
-Neither variant requires a green `doctor`: inside the VM there is no nested KVM
-or Docker, so **the backend is correctly reported not-ready** — install.sh does
-not install a backend and cannot regress one, and this harness does not
-provision the Docker backend. This is where Linux necessarily diverges from the
-macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
-makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
-verdict instead classifies `doctor`'s output the way the macOS harness does — a
-`Traceback` is an install defect (fail), a missing `python`/`config` line is a
-fail, a not-ready backend is reported — so a genuine installer regression (a
-broken shim, an import error, a botched PATH) stays visible.
-
-`test-all` runs the full matrix — every distro × both variants — each cell in
-its own throwaway VM, and prints a per-cell PASS/FAIL summary.
-
-## The distro matrix is the point
-
-Each distro exercises a different corner of the installer:
-
-| Distro | Cloud image | What it stresses |
-|---|---|---|
-| **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. |
-| **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. |
-| **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. |
-| **Alpine** | Alpine `nocloud_` (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
-| **NixOS** | locally built with `nixos-generators` | No FHS `~/.local` on PATH by default; `nix profile install` prereqs; pipx laying a self-contained venv on a non-FHS host. |
-
-### Validation run (2026-07-27) — full green
-
-Full matrix on the delphi KVM host, QEMU 11.0.2, both variants × all five
-distros passing:
-
-| Distro | `test` (bare) | `test-ready` (prepared) |
-|---|---|---|
-| Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green |
-| Fedora 44 | ✅ declines at git gate | ✅ installs |
-| Arch (latest) | ✅ declines at git gate | ✅ installs |
-| Alpine 3.21 | ✅ declines at git gate | ✅ installs |
-| NixOS 24.11 | ✅ declines (no python3) | ✅ installs |
-
-The bare `test` sees `install.sh` decline soundly — exit 1 at the
-git-for-git-specs gate on the Debian/Fedora/Arch/Alpine images (they ship
-python3 but not git), and at the python3 gate on NixOS (no python3 on PATH) —
-and `test-ready` installs cleanly with `doctor` reporting a usable python and
-config (backends all not-ready, as expected in a plain VM).
-
-Getting to green surfaced and fixed a series of real defects:
-
-- **Fedora 41 was EOL/404** → bumped to 44.
-- The liveness probe used `bot-bottle --version`, which the CLI does not
-  implement (unknown args die non-zero), so every *successful* install was
-  misreported as failed → switched to `bot-bottle --help`.
-- **Alpine** needed three fixes: the `generic_` image ignores a NoCloud seed
-  (switched to the `nocloud_` variant); OpenRC does not auto-start sshd after
-  cloud-init injects the key (start it via `runcmd`); and Alpine's non-PAM
-  sshd refuses pubkey auth for a cloud-init-*locked* account (give it a
-  throwaway password). It also has no `sudo` by default (install it via
-  cloud-init `packages:`).
-- **NixOS** publishes no downloadable cloud qcow2 (its cloud images are
-  Hydra-built AMIs), so the harness builds one with `nixos-generators`
-  ([`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)):
-  cloud-init for the key, flakes enabled, deliberately no python/git/pipx. The
-  `test-ready` prereq install pins `nixpkgs/nixos-24.11` because the guest's
-  default unstable registry builds pipx from source (and its test suite
-  currently fails to build).
-- Two harness-hygiene bugs also fixed: `cmd_down` left `serial.log` behind
-  (orphaned run dirs), and the teardown trap was armed after `cmd_up`, leaking
-  a VM when `wait_for_ssh` timed out.
-
-In `test-ready` the harness installs `python3 + git + pipx` first on each
-distro (install.sh installs none of them), so all five drive the recommended
-pipx path. `test` then removes that scaffolding and lets each distro's bare
-image collide with install.sh's guards — on most cloud images python3 is
-present (cloud-init needs it) but git and pipx are not, so install.sh is
-expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
-an actionable message. Both are legitimate, and the two variants together cover
-the whole first half of the installer as well as the happy path.
-
-## What a clean install touches (the footprint that decides "wipeable")
-
-| Artifact | Location | In the guest's `$HOME`? | Survives VM teardown? |
-|---|---|---|---|
-| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
-| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
-| pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
-| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix profile` | ❌ | ❌ **overlay deleted** |
-
-Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
-footprint *survives*), **every row here dies with the overlay** — that is the
-VM's whole advantage. The cached base image is read-only backing and is the
-only thing that persists between runs, on purpose.
-
-## Design notes baked into the harness
-
-- **User-mode networking** (`-netdev user,hostfwd=tcp:127.0.0.1:PORT-:22`):
-  no root, no bridge, no host network state touched. Only SSH is forwarded.
-- **cloud-init seed ISO** (`cloud-localds`) injects an ephemeral SSH keypair
-  and a passwordless-sudo login. The keypair is generated per run and deleted
-  on teardown; the guest can't be logged into after it's gone.
-- **Copy-on-write overlay**: the cached base is never mutated, so a corrupt or
-  interrupted run can't poison the cache; downloads land at `*.partial` and
-  are renamed only after checksum verification.
-- **Checksums**: verified against each vendor's published sums file at
-  download time (GNU `hash  file`, bare-hash, and Fedora's BSD
-  `SHA256 (file) = hash` formats are all handled). Alpine ships `.sha512` only
-  (this verifier is sha256) so it is skipped; NixOS is built locally, not
-  downloaded, so there is nothing to verify.
-- **NixOS is built, not downloaded**: `ensure_base_image` runs
-  `nixos-generate -f qcow` against
-  [`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)
-  once and caches the result; the per-run seed/overlay flow is otherwise
-  identical to the downloaded distros.
-- **`test-all`** runs every distro × both variants (`test` and `test-ready`),
-  each cell in its own subshell on its own forwarded port, so one cell's
-  failure (or teardown trap) can't abort the matrix; it prints a per-cell
-  PASS/FAIL summary.
-
-## Not wired into PR CI
-
-Like the macOS harness, the runtime is host-specific (needs `/dev/kvm`,
-`qemu`, and `cloud-localds`) and is not exercised by the Linux pull-request
-runner. It is validated statically (`bash -n`, `shellcheck`) and run by hand
-on a KVM-capable host. The cloud-image URLs in the `DISTRO` table are the one
-place to bump when a distro cuts a newer build.