feat(prd-0081): reprovision CA, git-gate, and egress tokens on gateway bring-up

On a Firecracker gateway cold boot, reconcile every live agent VM against
the fresh gateway: push the new mitmproxy CA into each agent's trust store,
re-provision git-gate repos/creds from the persisted upstreams snapshot,
and restore egress tokens. Per-bottle failures are logged and skipped so
one unreachable VM does not block the rest.

New modules / changes:
- backend/firecracker/reconcile.py: attach_bottled_agents_to_gateway,
  _push_ca, _reprovision_git_gate, _guest_ip_from_config
- git_gate/provision.py: write upstreams.json after key provisioning so
  the bring-up reconcile can reconstruct the upstream table without the
  manifest
- backend/firecracker/infra.py: call attach_bottled_agents_to_gateway in
  the cold-boot branch of ensure_running()
- backend/base.py: no-op default on BottleBackend
- backend/firecracker/consolidated_launch.py: remove superseded
  _reprovision_running_bottles / _guest_ip_from_config
- orchestrator: OrchestratorCore.update_agent_secret + POST
  /bottles/<id>/secret + client.update_agent_secret (single-secret
  in-place update, the reusable primitive from the 2026-07-26 hotfix)
- tests: 15 new tests in test_firecracker_reconcile.py; updated
  test_firecracker_infra.py cold-boot cases; stale FC reprovision tests
  removed from test_backend_secret_reprovision.py

Closes #516.

docs/research/testing-clean-install-on-linux.md

@@ -1,183 +0,0 @@
-# Testing a clean bot-bottle install on Linux
-
-How do you exercise `install.sh` the way a brand-new user would — on a
-pristine Linux environment you can throw away afterward — *without*
-polluting your daily-driver host, and across the several package-management
-regimes Linux fragments into? This is the Linux counterpart to
-[`testing-clean-install-on-macos.md`](testing-clean-install-on-macos.md);
-the conclusion is different because Linux gives us a boundary macOS doesn't.
-
-## Summary
-
-On macOS the honest options were a throwaway user or a VM, and the throwaway
-user won on pragmatics (nested virtualization is gated to M3+). On Linux the
-calculus flips: a **disposable KVM virtual machine, booted from a distro
-cloud image and deleted per run, is both the cleanest boundary and the one
-that lets a single harness cover Ubuntu, Fedora, Arch, Alpine, and NixOS**.
-The host already requires KVM for the Firecracker backend, so the VM is cheap
-here.
-
-The harness lives at [`scripts/linux-install-test.sh`](../../scripts/linux-install-test.sh).
-Per run it caches one read-only base image, boots a throwaway copy-on-write
-overlay (`qemu-img create -b base`), installs the distro's prerequisites,
-pipes *this checkout's* `install.sh` into the guest exactly as `curl … | sh`
-would, asserts the CLI installed, and deletes the overlay — the Linux
-equivalent of `docker run --rm`, for a whole machine.
-
-## Why a VM, not a container or a throwaway user
-
-| Mechanism | Why it's the wrong boundary here |
-|---|---|
-| **Container** (`docker run --rm`) | Shares the host kernel and ships a deliberately minimal userland — no systemd, a stubbed-out package manager story, and (crucially) it doesn't reproduce the *externally-managed Python* (PEP 668) that real desktop/server installs put in front of the user. It tests "does install.sh run in a container," not "does it run on a real distro." |
-| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
-| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
-
-## Two variants: `test` (bare host) and `test-ready` (prepared host)
-
-`install.sh` never installs a backend, and never installs its own toolchain
-prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
-runs `doctor`, which *reports* what's missing
-([`install.sh`](../../install.sh) header,
-[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
-That leaves two distinct things worth testing, split into two subcommands that
-mirror the macOS harness's `test` / `test-ready` convention (there the split is
-the backend service; here it is the toolchain the installer needs):
-
-- **`test`** — `install.sh` runs on the **bare cloud image**, prerequisites and
-  all left as the vendor ships them. This exercises `install.sh`'s own
-  prerequisite-guard logic — the entire first half of the script (python
-  version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
-- **`test-ready`** — the harness installs python3 + git + pipx first (the
-  `prereqs` step), then runs `install.sh`. This is the *prepared-host happy
-  path*: does a clean install actually land and produce a working CLI?
-
-**Pass criteria differ by variant:**
-
-| Variant | PASS when |
-|---|---|
-| `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
-| `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
-
-Neither variant requires a green `doctor`: inside the VM there is no nested KVM
-or Docker, so **the backend is correctly reported not-ready** — install.sh does
-not install a backend and cannot regress one, and this harness does not
-provision the Docker backend. This is where Linux necessarily diverges from the
-macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
-makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
-verdict instead classifies `doctor`'s output the way the macOS harness does — a
-`Traceback` is an install defect (fail), a missing `python`/`config` line is a
-fail, a not-ready backend is reported — so a genuine installer regression (a
-broken shim, an import error, a botched PATH) stays visible.
-
-`test-all` runs the full matrix — every distro × both variants — each cell in
-its own throwaway VM, and prints a per-cell PASS/FAIL summary.
-
-## The distro matrix is the point
-
-Each distro exercises a different corner of the installer:
-
-| Distro | Cloud image | What it stresses |
-|---|---|---|
-| **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. |
-| **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. |
-| **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. |
-| **Alpine** | Alpine `nocloud_` (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
-| **NixOS** | locally built with `nixos-generators` | No FHS `~/.local` on PATH by default; `nix profile install` prereqs; pipx laying a self-contained venv on a non-FHS host. |
-
-### Validation run (2026-07-27) — full green
-
-Full matrix on the delphi KVM host, QEMU 11.0.2, both variants × all five
-distros passing:
-
-| Distro | `test` (bare) | `test-ready` (prepared) |
-|---|---|---|
-| Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green |
-| Fedora 44 | ✅ declines at git gate | ✅ installs |
-| Arch (latest) | ✅ declines at git gate | ✅ installs |
-| Alpine 3.21 | ✅ declines at git gate | ✅ installs |
-| NixOS 24.11 | ✅ declines (no python3) | ✅ installs |
-
-The bare `test` sees `install.sh` decline soundly — exit 1 at the
-git-for-git-specs gate on the Debian/Fedora/Arch/Alpine images (they ship
-python3 but not git), and at the python3 gate on NixOS (no python3 on PATH) —
-and `test-ready` installs cleanly with `doctor` reporting a usable python and
-config (backends all not-ready, as expected in a plain VM).
-
-Getting to green surfaced and fixed a series of real defects:
-
-- **Fedora 41 was EOL/404** → bumped to 44.
-- The liveness probe used `bot-bottle --version`, which the CLI does not
-  implement (unknown args die non-zero), so every *successful* install was
-  misreported as failed → switched to `bot-bottle --help`.
-- **Alpine** needed three fixes: the `generic_` image ignores a NoCloud seed
-  (switched to the `nocloud_` variant); OpenRC does not auto-start sshd after
-  cloud-init injects the key (start it via `runcmd`); and Alpine's non-PAM
-  sshd refuses pubkey auth for a cloud-init-*locked* account (give it a
-  throwaway password). It also has no `sudo` by default (install it via
-  cloud-init `packages:`).
-- **NixOS** publishes no downloadable cloud qcow2 (its cloud images are
-  Hydra-built AMIs), so the harness builds one with `nixos-generators`
-  ([`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)):
-  cloud-init for the key, flakes enabled, deliberately no python/git/pipx. The
-  `test-ready` prereq install pins `nixpkgs/nixos-24.11` because the guest's
-  default unstable registry builds pipx from source (and its test suite
-  currently fails to build).
-- Two harness-hygiene bugs also fixed: `cmd_down` left `serial.log` behind
-  (orphaned run dirs), and the teardown trap was armed after `cmd_up`, leaking
-  a VM when `wait_for_ssh` timed out.
-
-In `test-ready` the harness installs `python3 + git + pipx` first on each
-distro (install.sh installs none of them), so all five drive the recommended
-pipx path. `test` then removes that scaffolding and lets each distro's bare
-image collide with install.sh's guards — on most cloud images python3 is
-present (cloud-init needs it) but git and pipx are not, so install.sh is
-expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
-an actionable message. Both are legitimate, and the two variants together cover
-the whole first half of the installer as well as the happy path.
-
-## What a clean install touches (the footprint that decides "wipeable")
-
-| Artifact | Location | In the guest's `$HOME`? | Survives VM teardown? |
-|---|---|---|---|
-| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
-| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
-| pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
-| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix profile` | ❌ | ❌ **overlay deleted** |
-
-Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
-footprint *survives*), **every row here dies with the overlay** — that is the
-VM's whole advantage. The cached base image is read-only backing and is the
-only thing that persists between runs, on purpose.
-
-## Design notes baked into the harness
-
-- **User-mode networking** (`-netdev user,hostfwd=tcp:127.0.0.1:PORT-:22`):
-  no root, no bridge, no host network state touched. Only SSH is forwarded.
-- **cloud-init seed ISO** (`cloud-localds`) injects an ephemeral SSH keypair
-  and a passwordless-sudo login. The keypair is generated per run and deleted
-  on teardown; the guest can't be logged into after it's gone.
-- **Copy-on-write overlay**: the cached base is never mutated, so a corrupt or
-  interrupted run can't poison the cache; downloads land at `*.partial` and
-  are renamed only after checksum verification.
-- **Checksums**: verified against each vendor's published sums file at
-  download time (GNU `hash  file`, bare-hash, and Fedora's BSD
-  `SHA256 (file) = hash` formats are all handled). Alpine ships `.sha512` only
-  (this verifier is sha256) so it is skipped; NixOS is built locally, not
-  downloaded, so there is nothing to verify.
-- **NixOS is built, not downloaded**: `ensure_base_image` runs
-  `nixos-generate -f qcow` against
-  [`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)
-  once and caches the result; the per-run seed/overlay flow is otherwise
-  identical to the downloaded distros.
-- **`test-all`** runs every distro × both variants (`test` and `test-ready`),
-  each cell in its own subshell on its own forwarded port, so one cell's
-  failure (or teardown trap) can't abort the matrix; it prints a per-cell
-  PASS/FAIL summary.
-
-## Not wired into PR CI
-
-Like the macOS harness, the runtime is host-specific (needs `/dev/kvm`,
-`qemu`, and `cloud-localds`) and is not exercised by the Linux pull-request
-runner. It is validated statically (`bash -n`, `shellcheck`) and run by hand
-on a KVM-capable host. The cloud-image URLs in the `DISTRO` table are the one
-place to bump when a distro cuts a newer build.