fix: name an absolute path in the sudo re-run hint

The firecracker setup message told users to run sudo bot-bottle backend setup --backend=firecracker which fails for exactly the users who followed the documented install. sudo replaces PATH with sudoers' secure_path — /usr/local/sbin:/usr/local/bin: /usr/sbin:/usr/bin:/sbin:/bin on Debian and Ubuntu — which deliberately excludes user-writable directories. Both supported install paths land in one: pipx uses ~/.local/bin, and install.sh's venv fallback symlinks there. So the hint works for anyone who installed system-wide and breaks with "command not found" for everyone else, which is how it survives a read-through. Add bot_bottle/invocation.py: self_path() resolves the running entry point to an absolute path, and sudo_command() builds the copy-pasteable form. The one sudo recommendation in the tree now uses it. Non-sudo hints keep the bare `bot-bottle`, which is correct — the user reached them by running it. self_path() falls back to the bare name when argv[0] cannot be resolved (a `python -m` style invocation), because a slightly wrong hint beats a traceback raised while reporting some unrelated problem. Tested behaviourally rather than by scanning source: the first version of the test grepped the module and failed on the comment explaining why the bare form is wrong. It now drives _setup_systemd() as non-root and asserts what is actually printed. Verified the guard bites by restoring the bare form and watching it fail. Not verified end to end: this message only prints on the systemd path, so it is unreachable on macOS, where the rest of this work was tested.
feat: get Alpine and NixOS cells to green
2026-07-27 12:35:51 -04:00 · 2026-07-27 12:24:31 -04:00 · 2026-07-27 12:24:31 -04:00 · 2026-07-27 12:24:31 -04:00 · 2026-07-27 12:24:31 -04:00 · 2026-07-27 12:24:31 -04:00
51 changed files with 1177 additions and 1086 deletions
@@ -21,7 +21,7 @@ from abc import ABC, abstractmethod
 from contextlib import AbstractContextManager, contextmanager
 from dataclasses import dataclass
 from pathlib import Path
-from typing import TYPE_CHECKING, Generator, Generic, Sequence, TypeVar
+from typing import Generator, Generic, Sequence, TypeVar

 from ..agent_provider import AgentProvisionPlan, get_provider
 from ..egress import EgressPlan
@@ -35,9 +35,6 @@ from ..workspace import WorkspacePlan, workspace_plan
 from .print_util import print_multi, visible_agent_env_names
 from .util import host_skill_dir

-if TYPE_CHECKING:
-    from .gateway_attach import GatewayAttachResources
-

 class BackendStatus(enum.IntEnum):
    """Return codes for BottleBackend.status(). READY == 0 so callsites
@@ -273,17 +270,6 @@ class Bottle(ABC):

 PlanT = TypeVar("PlanT", bound=BottlePlan)
 CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
-# The backend-specific handle for one running bottle the reconcile attaches to
-# the gateway (a run dir for firecracker, a container name for docker/macOS).
-# Registry-style callers that don't care about the handle bind it to `Any`
-# (`BottleBackend[Any, Any, Any]`).
-AttachTargetT = TypeVar("AttachTargetT")
-
-
-class InfraLaunchError(RuntimeError):
-    """A per-host infra (gateway + orchestrator) launch/reconcile step could not
-    complete. Shared across backends so the base backend can raise + catch one
-    type; each backend's ``infra_launch`` module re-exports it."""


@dataclass(frozen=True)
@@ -298,7 +284,7 @@ class BottleImages:
    sidecar: str | Path = ""


-class BottleBackend(ABC, Generic[PlanT, CleanupT, AttachTargetT]):
+class BottleBackend(ABC, Generic[PlanT, CleanupT]):
    """Abstract base for selectable bottle backends. Concrete subclasses
    (e.g. DockerBottleBackend) own their own prepare/launch impls.
    Parameterized over the backend's concrete plan + cleanup-plan types
@@ -526,31 +512,6 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT, AttachTargetT]):
        compose ls`; firecracker cross-references its running gateway
        containers against per-bottle metadata."""

-    def attach_bottled_agents_to_gateway(self) -> None:
-        """Reconcile every running bottle against the freshly-(re)booted gateway
-        on the cold-boot bring-up path (PRD 0081). The shared flow + fail-hard
-        policy live in `gateway_attach`; backends override only the three
-        primitives below (ADR 0006)."""
-        from .gateway_attach import reconcile_running_bottles
-        reconcile_running_bottles(self)
-
-    @abstractmethod
-    def _gateway_attach_resources(self) -> "GatewayAttachResources":
-        """Gather what every bottle needs to (re)attach to the current gateway
-        (the CA now). Raises if the gateway isn't reachable (aborts reconcile)."""
-
-    @abstractmethod
-    def _running_bottles(self) -> Sequence[AttachTargetT]:
-        """The live bottles as backend handles for `_attach_bottle_to_gateway`.
-        Raises if the live set can't be determined (fail hard, no partial set)."""
-
-    @abstractmethod
-    def _attach_bottle_to_gateway(
-        self, bottle: AttachTargetT, resources: "GatewayAttachResources",
-    ) -> None:
-        """(Re)attach one bottle to the current gateway (SSH / `exec`+`cp`).
-        Raises `InfraLaunchError` naming the bottle on failure."""
-
    @classmethod
    @abstractmethod
    def is_available(cls) -> bool:
@@ -23,7 +23,7 @@ import shutil
 import io
 from contextlib import contextmanager, redirect_stderr
 from pathlib import Path
-from typing import TYPE_CHECKING, Generator, Sequence
+from typing import Generator, Sequence

 from ...supervisor.types import SUPERVISE_HOSTNAME, SUPERVISE_PORT
 from ...agent_provider import AgentProvisionPlan
@@ -33,7 +33,6 @@ from ...git_gate import GitGatePlan
 from ...supervisor.plan import SupervisePlan
 from ...manifest import Manifest
 from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
-from ..gateway_attach import GatewayAttachResources
 from . import cleanup as _cleanup
 from . import enumerate as _enumerate
 from . import launch as _launch
@@ -41,27 +40,12 @@ from . import resolve_plan as _resolve_plan
 from .bottle import DockerBottle
 from .bottle_cleanup_plan import DockerBottleCleanupPlan
 from .bottle_plan import DockerBottlePlan
-
-if TYPE_CHECKING:
-    from .infra import DockerInfraService
-
-
-class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanupPlan", str]):
+class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanupPlan"]):
    """Docker backend implementation. Selected by BOT_BOTTLE_BACKEND
    when set to `docker`; retained as a legacy/example backend."""

    name = "docker"

-    def __init__(self, *, infra: "DockerInfraService | None" = None) -> None:
-        # The infra service whose gateway just cold-booted, passed in on the
-        # bring-up reconcile path (PRD 0081) so `_gateway_attach_resources`
-        # reads THAT gateway's CA rather than a fresh default-named service —
-        # otherwise a non-default instance (an isolated integration test's
-        # `-itest-` gateway) reads the default `bot-bottle-orch-gateway`, which
-        # doesn't exist for it (#519 review). None for ordinary construction:
-        # falls back to the per-host default singleton.
-        self._infra = infra
-
    @classmethod
    def is_available(cls) -> bool:
        """`docker` on PATH is sufficient; we don't probe `docker info`
@@ -156,22 +140,3 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup

    def enumerate_active(self) -> Sequence[ActiveAgent]:
        return _enumerate.enumerate_active()
-
-    def _gateway_attach_resources(self) -> GatewayAttachResources:
-        from .infra import DockerInfraService
-        infra = self._infra if self._infra is not None else DockerInfraService()
-        return GatewayAttachResources(ca_pem=infra.gateway().ca_cert_pem())
-
-    def _running_bottles(self) -> Sequence[str]:
-        from .infra import DockerInfraService
-        from .infra_launch import running_agent_containers
-        infra = self._infra if self._infra is not None else DockerInfraService()
-        return running_agent_containers(
-            network=infra.network, gateway_name=infra.gateway().name,
-        )
-
-    def _attach_bottle_to_gateway(
-        self, bottle: str, resources: GatewayAttachResources,
-    ) -> None:
-        from .infra_launch import push_ca_to_container
-        push_ca_to_container(bottle, resources.ca_pem)
@@ -13,7 +13,6 @@ orchestrator-facing wiring so that sequence stays testable in isolation.

 from __future__ import annotations

-import tempfile
 from dataclasses import dataclass

 from ... import log
@@ -25,13 +24,15 @@ from ...gateway import GATEWAY_NETWORK
 from .infra import INFRA_NAME, DockerInfraService
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
 from ...orchestrator.reprovision import reprovision_bottles
-from ..base import InfraLaunchError
 from ..provision_bottle import deprovision_bottle, provision_bottle
-from ..util import AGENT_CA_PATH
 from .gateway_transport import DockerGatewayTransport
 from .gateway_net import next_free_ip


+class ConsolidatedLaunchError(RuntimeError):
+    """The consolidated register/provision sequence could not complete."""
+
+
@dataclass(frozen=True)
 class LaunchContext:
    """What the agent container needs to join the shared gateway."""
@@ -53,7 +54,7 @@ def _network_cidr(network: str) -> str:
    ])
    cidr = proc.stdout.strip()
    if proc.returncode != 0 or not cidr:
-        raise InfraLaunchError(
+        raise ConsolidatedLaunchError(
            f"gateway network {network} has no subnet: {proc.stderr.strip()}"
        )
    return cidr
@@ -69,7 +70,7 @@ def _network_container_ips(network: str) -> list[str]:
    ])
    if proc.returncode != 0:
        detail = proc.stderr.strip() or f"exit {proc.returncode}"
-        raise InfraLaunchError(
+        raise ConsolidatedLaunchError(
            f"could not inspect addresses on gateway network {network}: {detail}"
        )
    ips: list[str] = []
@@ -78,64 +79,6 @@ def _network_container_ips(network: str) -> list[str]:
    return ips


-def running_agent_containers(
-    network: str = GATEWAY_NETWORK, gateway_name: str = INFRA_NAME,
-) -> list[str]:
-    """Every running agent container on the gateway `network` (the gateway
-    itself excluded) — the bottles the bring-up reconcile attaches to the fresh
-    gateway (PRD 0081). Raises `OSError` if `docker` can't be run (fail hard: a
-    reconcile that can't list its bottles must not look like "no bottles")."""
-    proc = run_docker([
-        "docker", "network", "inspect", "--format",
-        "{{range .Containers}}{{.Name}}\n{{end}}", network,
-    ])
-    if proc.returncode != 0:
-        detail = proc.stderr.strip() or f"exit {proc.returncode}"
-        raise InfraLaunchError(
-            f"could not enumerate bottles on gateway network {network}: {detail}"
-        )
-    return [
-        name for name in (line.strip() for line in proc.stdout.splitlines())
-        if name and name != gateway_name
-    ]
-
-
-def push_ca_to_container(container: str, ca_pem: str) -> None:
-    """(Re)attach one running agent container to the current gateway: replace
-    its trusted gateway CA with `ca_pem` and rebuild its trust store via
-    `docker cp` + `docker exec` (PRD 0081). Unconditional install — there is one
-    gateway, so no fingerprint match is needed.
-
-    Fail hard: raises `InfraLaunchError` (naming the container) on any failure —
-    the base reconcile aggregates and surfaces it rather than leaving the agent
-    silently unable to reach the gateway."""
-    mkdir = run_docker(
-        ["docker", "exec", container, "mkdir", "-p",
-         "/usr/local/share/ca-certificates"]
-    )
-    if mkdir.returncode != 0:
-        raise InfraLaunchError(
-            f"CA push to {container} failed (mkdir): {mkdir.stderr.strip()}"
-        )
-    with tempfile.NamedTemporaryFile("w", suffix=".crt") as tmp:
-        tmp.write(ca_pem)
-        tmp.flush()
-        cp = run_docker(["docker", "cp", tmp.name, f"{container}:{AGENT_CA_PATH}"])
-    if cp.returncode != 0:
-        raise InfraLaunchError(
-            f"CA push to {container} failed (cp): {cp.stderr.strip()}"
-        )
-    ex = run_docker(
-        ["docker", "exec", container, "sh", "-c",
-         f"chmod 644 {AGENT_CA_PATH} && update-ca-certificates"]
-    )
-    if ex.returncode != 0:
-        raise InfraLaunchError(
-            f"CA push to {container} failed (update-ca-certificates): "
-            f"{ex.stderr.strip()}"
-        )
-
-
 def _reprovision_running_bottles(
    orchestrator_url: str,
    network: str = GATEWAY_NETWORK,
@@ -242,8 +185,6 @@ def deprovision_consolidated(
 __all__ = [
    "LaunchContext",
    "launch_consolidated",
-    "running_agent_containers",
-    "push_ca_to_container",
    "deprovision_consolidated",
-    "InfraLaunchError",
+    "ConsolidatedLaunchError",
 ]
@@ -200,7 +200,7 @@ class DockerGateway(Gateway):
                f"gateway network {self.network} failed to create: {proc.stderr.strip()}"
            )

-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        # Bind to this orchestrator (PRD 0070): stash the URL its daemons resolve
        # policy against + the pre-minted `gateway` token they present, then bring
        # the container up. The gateway never mints, so it holds no signing key —
@@ -227,7 +227,7 @@ class DockerGateway(Gateway):
        # just when the container is absent.
        self._ensure_network()
        if self.is_running() and self._running_image_is_current():
-            return False
+            return
        # Clear any stale (stopped OR outdated-image) container holding the
        # fixed name, then start fresh. `rm --force` on an absent name is a
        # tolerated no-op.
@@ -271,9 +271,6 @@ class DockerGateway(Gateway):
        # pre-connect window (they retry /resolve per request).
        if self._control_network:
            self._connect_control_network()
-        # (Re)created a fresh container — signal a cold boot so the caller
-        # reconciles running bottles against it (PRD 0081).
-        return True

    def _connect_control_network(self) -> None:
        """Ensure the `--internal` control network exists and attach the gateway
@@ -142,16 +142,9 @@ class DockerInfraService(InfraService):
        # mints the role-scoped `gateway` JWT here and hands it to the gateway,
        # which never sees the key (#469). `connect_to_orchestrator` is
        # idempotent.
-        cold_booted = gateway.connect_to_orchestrator(
+        gateway.connect_to_orchestrator(
            orchestrator.gateway_url(), orchestrator.mint_gateway_token(),
        )
-        # A (re)created gateway container minted/mounted its CA afresh and lost
-        # every bottle's per-bottle state; reconcile the already-running bottles
-        # against it (PRD 0081). Skipped when a healthy current gateway was left
-        # untouched — no cold boot, nothing to reconcile.
-        if cold_booted:
-            from .backend import DockerBottleBackend
-            DockerBottleBackend(infra=self).attach_bottled_agents_to_gateway()
        return orchestrator.url()

    def stop(self) -> None:
@@ -61,7 +61,7 @@ from .compose import (
 )
 from .consolidated_compose import consolidated_agent_compose
 from ...orchestrator.store.config_store import resolve_teardown_timeout
-from .infra_launch import launch_consolidated, deprovision_consolidated
+from .consolidated_launch import launch_consolidated, deprovision_consolidated
 from .infra import INFRA_NAME
 from .gateway import DockerGateway
 from ... import resources
@@ -20,7 +20,6 @@ from ...git_gate import GitGatePlan
 from ...manifest import Manifest
 from ...supervisor.plan import SupervisePlan
 from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
-from ..gateway_attach import GatewayAttachResources
 from . import cleanup as _cleanup
 from . import enumerate as _enumerate
 from . import launch as _launch
@@ -32,7 +31,7 @@ from .bottle_plan import FirecrackerBottlePlan


 class FirecrackerBottleBackend(
-    BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan", "Path"]
+    BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan"]
 ):
    name = "firecracker"

@@ -120,19 +119,6 @@ class FirecrackerBottleBackend(
    def enumerate_active(self) -> Sequence[ActiveAgent]:
        return _enumerate.enumerate_active()

-    def _gateway_attach_resources(self) -> GatewayAttachResources:
-        from .gateway import FirecrackerGateway
-        return GatewayAttachResources(ca_pem=FirecrackerGateway().ca_cert_pem())
-
-    def _running_bottles(self) -> Sequence[Path]:
-        return _cleanup.live_run_dirs()
-
-    def _attach_bottle_to_gateway(
-        self, bottle: Path, resources: GatewayAttachResources,
-    ) -> None:
-        from .infra_launch import attach_ca_to_agent_vm
-        attach_ca_to_agent_vm(bottle, resources.ca_pem)
-
    def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
        return plan.agent_supervise_url

@@ -40,9 +40,7 @@ from ...orchestrator.lifecycle import (
 )
 from ...orchestrator.reprovision import reprovision_bottles
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
-from ..base import InfraLaunchError
 from ..provision_bottle import deprovision_bottle, provision_bottle
-from ..util import AGENT_CA_PATH
 from . import cleanup, util
 from .gateway import FirecrackerGateway
 from .infra import FirecrackerInfraService
@@ -50,6 +48,10 @@ from .infra import FirecrackerInfraService
 _ENV_VAR_SECRET_PATH = "/run/bot-bottle/env-var-secret"


+class ConsolidatedLaunchError(RuntimeError):
+    """The consolidated register/provision sequence could not complete."""
+
+
@dataclass(frozen=True)
 class LaunchContext:
    """What the Firecracker launch needs from the consolidated sequence."""
@@ -81,48 +83,12 @@ def persist_env_var_secret(private_key: Path, guest_ip: str, secret: str) -> Non
        input=secret, capture_output=True, text=True, check=False,
    )
    if proc.returncode != 0:
-        raise InfraLaunchError(
+        raise ConsolidatedLaunchError(
            f"failed to persist {ENV_VAR_SECRET_NAME} in agent VM: "
            f"{proc.stderr.strip() or '<no stderr>'}"
        )


-def attach_ca_to_agent_vm(run_dir: Path, ca_pem: str) -> None:
-    """(Re)attach one running agent VM (identified by its `run_dir`) to the
-    current gateway: replace its trusted gateway CA with `ca_pem` and rebuild
-    its trust store over SSH (PRD 0081). Unconditional install — there is one
-    gateway, so no fingerprint match is needed. Bare-pipe input keeps the cert
-    off argv.
-
-    Fail hard: raises `InfraLaunchError` (naming the bottle) if the run dir is
-    malformed or the push fails — the base reconcile aggregates and surfaces it
-    rather than leaving the agent silently unable to reach the gateway."""
-    guest_ip = _guest_ip_from_config(run_dir / "config.json")
-    private_key = run_dir / "bottle_id_ed25519"
-    if not guest_ip or not private_key.is_file():
-        raise InfraLaunchError(
-            f"{run_dir.name}: cannot resolve guest IP / SSH key to attach it "
-            f"to the gateway"
-        )
-    install = (
-        "umask 022; mkdir -p /usr/local/share/ca-certificates && "
-        f"cat > {AGENT_CA_PATH} && chmod 644 {AGENT_CA_PATH} && "
-        "update-ca-certificates"
-    )
-    try:
-        proc = subprocess.run(
-            util.ssh_base_argv(private_key, guest_ip) + [install],
-            input=ca_pem, capture_output=True, text=True, check=False,
-        )
-    except OSError as exc:
-        raise InfraLaunchError(f"{run_dir.name} ({guest_ip}): CA push failed: {exc}") from exc
-    if proc.returncode != 0:
-        raise InfraLaunchError(
-            f"{run_dir.name} ({guest_ip}): CA push failed: "
-            f"{proc.stderr.strip() or '<no stderr>'}"
-        )
-
-
 def _reprovision_running_bottles(client: OrchestratorClient) -> None:
    """Read keys from live agent VMs and restore the restarted gateway."""
    try:
@@ -195,8 +161,7 @@ def deprovision_consolidated(
 __all__ = [
    "LaunchContext",
    "launch_consolidated",
-    "attach_ca_to_agent_vm",
    "deprovision_consolidated",
-    "InfraLaunchError",
+    "ConsolidatedLaunchError",
    "OrchestratorStartError",
 ]
@@ -69,17 +69,13 @@ class FirecrackerGateway(Gateway):
        self._orchestrator_url = ""
        self._gateway_token = ""

-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        """Boot the gateway VM on its link resolving policy against
        `orchestrator_url`, then seed `gateway_token` over SSH. The orchestrator's
        guest IP is parsed from the URL and passed on the cmdline (`bb_orch=`), so
        the baked init stays IP-independent. Boot the orchestrator first — the
        gateway daemons reach it at startup. The gateway never mints, so it holds
-        no signing key, only this token (#469).
-
-        Always returns True: the firecracker gateway VM is booted fresh here
-        (this runs only on the infra cold-boot path), so it is always a cold
-        boot that minted a new CA — the caller reconciles running bottles."""
+        no signing key, only this token (#469)."""
        self._orchestrator_url = orchestrator_url
        self._gateway_token = gateway_token
        if not self._orchestrator_url:
@@ -109,7 +105,6 @@ class FirecrackerGateway(Gateway):
            "the gateway JWT to the gateway VM (its data plane will not start)",
        )
        self._vm = vm
-        return True

    def is_running(self) -> bool:
        return infra_vm._pidfile_alive(infra_vm._gw_dir())
@@ -67,17 +67,9 @@ class FirecrackerInfraService(InfraService):
            # holds the signing key) mints the role-scoped `gateway` JWT for the
            # gateway, which never sees the key (#469).
            orchestrator.ensure_running(startup_timeout=startup_timeout)
-            cold_booted = self.gateway().connect_to_orchestrator(
+            self.gateway().connect_to_orchestrator(
                orchestrator.gateway_url(), orchestrator.mint_gateway_token())
            infra_vm.record_booted_version(want)
-            # The fresh gateway rootfs minted a new CA and lost every bottle's
-            # git-gate/token state; reconcile the already-running bottles against
-            # it so a gateway rebuild doesn't silently break their egress
-            # (PRD 0081). Cold-boot only — the adopt fast-paths above never reach
-            # here, so a healthy current gateway is left untouched.
-            if cold_booted:
-                from .backend import FirecrackerBottleBackend
-                FirecrackerBottleBackend().attach_bottled_agents_to_gateway()
            return url

    def stop(self) -> None:
@@ -51,7 +51,7 @@ from .bottle import FirecrackerBottle
 from .bottle_plan import FirecrackerBottlePlan
 from ...orchestrator.store.config_store import resolve_teardown_timeout
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
-from .infra_launch import (
+from .consolidated_launch import (
    launch_consolidated,
    persist_env_var_secret,
    deprovision_consolidated,
@@ -20,6 +20,7 @@ import subprocess
 import sys
 from pathlib import Path

+from ... import invocation
 from ... import resources
 from . import netpool
 from . import util
@@ -179,9 +180,11 @@ def _setup_systemd() -> None:
        f"sudo systemctl daemon-reload\n"
        f"sudo systemctl enable --now {netpool.SYSTEMD_UNIT}\n"
    )
+    # Absolute path, not `sudo bot-bottle`: sudo's secure_path drops
+    # ~/.local/bin, where both pipx and install.sh put the entry point.
    sys.stderr.write(
-        f"\n(Or re-run this as root to install it directly: "
-        f"sudo bot-bottle backend setup --backend=firecracker)\n"
+        f"\n(Or re-run this as root to install it directly:\n"
+        f"    {invocation.sudo_command('backend', 'setup', '--backend=firecracker')})\n"
    )


@@ -1,54 +0,0 @@
-"""The shared gateway-attach reconcile flow (PRD 0081).
-
-`BottleBackend.attach_bottled_agents_to_gateway()` delegates here so every
-backend reconciles running bottles against a freshly-booted gateway *the same
-way* — the control flow + fail-hard policy live in one place and backends
-override only the primitives (see ADR 0006). Kept out of `backend/base.py` so
-the backend contract module stays lean (the base.py size guardrail).
-"""
-
-from __future__ import annotations
-
-from dataclasses import dataclass
-from typing import TYPE_CHECKING, Any
-
-from .base import InfraLaunchError
-
-if TYPE_CHECKING:
-    from .base import BottleBackend
-
-
-@dataclass(frozen=True)
-class GatewayAttachResources:
-    """Everything a running bottle needs to (re)attach to the current gateway on
-    a cold boot (PRD 0081). Gathered once per reconcile and handed to every
-    bottle. The CA now; egress secrets and git-gate state join as the reconcile
-    grows (#516)."""
-
-    ca_pem: str
-
-
-def reconcile_running_bottles(backend: "BottleBackend[Any, Any, Any]") -> None:
-    """Reconcile every already-running bottle against the freshly-(re)booted
-    gateway: gather the attach resources once, then attach every live bottle.
-
-    Fail hard, never skip: a bottle that silently can't reach the fresh gateway
-    (its egress just starts failing TLS) is worse than a loud bring-up failure,
-    so any attach failure aborts bring-up. Every bottle is attempted first and
-    the failures are raised together, so one bring-up surfaces the full blast
-    radius rather than one bottle at a time."""
-    resources = backend._gateway_attach_resources()
-    failures: list[str] = []
-    for bottle in backend._running_bottles():
-        try:
-            backend._attach_bottle_to_gateway(bottle, resources)
-        except InfraLaunchError as exc:
-            failures.append(str(exc))
-    if failures:
-        raise InfraLaunchError(
-            f"could not attach {len(failures)} running bottle(s) to the "
-            f"freshly-booted gateway:\n  " + "\n  ".join(failures)
-        )
-
-
-__all__ = ["GatewayAttachResources", "reconcile_running_bottles"]
@@ -14,7 +14,6 @@ from ...git_gate import GitGatePlan
 from ...supervisor.plan import SupervisePlan
 from ...manifest import Manifest
 from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
-from ..gateway_attach import GatewayAttachResources
 from . import cleanup as _cleanup
 from . import enumerate as _enumerate
 from . import launch as _launch
@@ -26,7 +25,7 @@ from .bottle_plan import MacosContainerBottlePlan


 class MacosContainerBottleBackend(
-    BottleBackend["MacosContainerBottlePlan", "MacosContainerBottleCleanupPlan", str]
+    BottleBackend["MacosContainerBottlePlan", "MacosContainerBottleCleanupPlan"]
 ):
    """Apple Container backend. Selected by
    `BOT_BOTTLE_BACKEND=macos-container` or
@@ -118,19 +117,5 @@ class MacosContainerBottleBackend(
    def enumerate_active(self) -> Sequence[ActiveAgent]:
        return _enumerate.enumerate_active()

-    def _gateway_attach_resources(self) -> GatewayAttachResources:
-        from .infra import MacosInfraService
-        return GatewayAttachResources(ca_pem=MacosInfraService().ca_cert_pem())
-
-    def _running_bottles(self) -> Sequence[str]:
-        from .infra_launch import running_agent_containers
-        return running_agent_containers()
-
-    def _attach_bottle_to_gateway(
-        self, bottle: str, resources: GatewayAttachResources,
-    ) -> None:
-        from .infra_launch import push_ca_to_container
-        push_ca_to_container(bottle, resources.ca_pem)
-
    def supervise_mcp_url(self, plan: MacosContainerBottlePlan) -> str:
        return plan.agent_supervise_url
@@ -34,7 +34,6 @@ every real command arrives through `container exec`.

 from __future__ import annotations

-import tempfile
 from dataclasses import dataclass

 from ...egress import EgressPlan
@@ -43,9 +42,7 @@ from ...log import info
 from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
 from ...orchestrator.reprovision import reprovision_bottles
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
-from ..base import InfraLaunchError
 from ..provision_bottle import deprovision_bottle, provision_bottle
-from ..util import AGENT_CA_PATH
 from . import util as container_mod
 from .enumerate import CONTAINER_NAME_PREFIX, EnumerationError, enumerate_active
 from .gateway import GATEWAY_NETWORK
@@ -53,6 +50,10 @@ from .gateway_transport import MacosGatewayTransport
 from .infra import MacosInfraService, OrchestratorStartError


+class ConsolidatedLaunchError(RuntimeError):
+    """The consolidated register/provision sequence could not complete."""
+
+
@dataclass(frozen=True)
 class GatewayEndpoint:
    """What the agent `container run` needs to reach the shared gateway.
@@ -97,52 +98,6 @@ def ensure_gateway(
    return endpoint


-def running_agent_containers() -> list[str]:
-    """Every running agent container's name — the bottles the bring-up reconcile
-    attaches to the fresh gateway (PRD 0081). Raises `EnumerationError` if the
-    live set can't be determined (fail hard rather than reconcile a partial
-    set)."""
-    return [f"{CONTAINER_NAME_PREFIX}{agent.slug}" for agent in enumerate_active()]
-
-
-def push_ca_to_container(name: str, ca_pem: str) -> None:
-    """(Re)attach one running agent container to the current gateway: replace
-    its trusted gateway CA with `ca_pem` and rebuild its trust store via
-    `container cp` + `container exec` (PRD 0081). Unconditional install — there
-    is one gateway, so no fingerprint match is needed.
-
-    Fail hard: raises `InfraLaunchError` (naming the container) on any failure —
-    the base reconcile aggregates and surfaces it rather than leaving the agent
-    silently unable to reach the gateway."""
-    mkdir = container_mod.run_container_argv(
-        ["container", "exec", name, "mkdir", "-p",
-         "/usr/local/share/ca-certificates"]
-    )
-    if mkdir.returncode != 0:
-        raise InfraLaunchError(
-            f"CA push to {name} failed (mkdir): {(mkdir.stderr or '').strip()}"
-        )
-    with tempfile.NamedTemporaryFile("w", suffix=".crt") as tmp:
-        tmp.write(ca_pem)
-        tmp.flush()
-        cp = container_mod.run_container_argv(
-            ["container", "cp", tmp.name, f"{name}:{AGENT_CA_PATH}"]
-        )
-    if cp.returncode != 0:
-        raise InfraLaunchError(
-            f"CA push to {name} failed (cp): {(cp.stderr or '').strip()}"
-        )
-    ex = container_mod.run_container_argv(
-        ["container", "exec", name, "sh", "-c",
-         f"chmod 644 {AGENT_CA_PATH} && update-ca-certificates"]
-    )
-    if ex.returncode != 0:
-        raise InfraLaunchError(
-            f"CA push to {name} failed (update-ca-certificates): "
-            f"{(ex.stderr or '').strip()}"
-        )
-
-
 def _reprovision_running_bottles(endpoint: GatewayEndpoint) -> None:
    """Recover keys from live Apple containers and restore gateway tokens."""
    try:
@@ -245,12 +200,10 @@ __all__ = [
    "GatewayEndpoint",
    "LaunchContext",
    "ensure_gateway",
-    "running_agent_containers",
-    "push_ca_to_container",
    "live_source_ips",
    "register_agent",
    "deprovision_consolidated",
-    "InfraLaunchError",
+    "ConsolidatedLaunchError",
    "OrchestratorStartError",
    "GATEWAY_NETWORK",
 ]
@@ -104,15 +104,11 @@ class MacosGateway(Gateway):
        container_mod.build_image(
            self.image_ref, str(self._repo_root), dockerfile="Dockerfile.gateway")

-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        """Bind the gateway to this orchestrator and (re)start it, dual-homed on
        the agent + control networks, resolving policy against `orchestrator_url`
        (the orchestrator's control-network address — Apple has no container
-        DNS) and presenting `gateway_token`.
-
-        Always returns True: this recreates the gateway container unconditionally
-        (a cold boot), so the caller reconciles running bottles against it
-        (PRD 0081)."""
+        DNS) and presenting `gateway_token`."""
        self._orchestrator_url = orchestrator_url
        self._gateway_token = gateway_token
        # Fail closed on a missing policy source or token: the resolver-only data
@@ -160,7 +156,6 @@ class MacosGateway(Gateway):
                f"gateway container failed to start: "
                f"{(result.stderr or '').strip() or '<no stderr>'}"
            )
-        return True

    def is_running(self) -> bool:
        return container_mod.container_is_running(self.name)
@@ -138,15 +138,7 @@ class MacosInfraService(InfraService):
        # `gateway` JWT and hands it to the gateway, which never sees the key
        # (#469).
        url = orchestrator.url()
-        cold_booted = gateway.connect_to_orchestrator(
-            url, orchestrator.mint_gateway_token())
-        # A (re)created gateway container minted/mounted its CA afresh and lost
-        # every bottle's per-bottle state; reconcile the already-running bottles
-        # against it (PRD 0081). Skipped when a healthy current gateway was left
-        # untouched — no cold boot, nothing to reconcile.
-        if cold_booted:
-            from .backend import MacosContainerBottleBackend
-            MacosContainerBottleBackend().attach_bottled_agents_to_gateway()
+        gateway.connect_to_orchestrator(url, orchestrator.mint_gateway_token())
        return url

    def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
@@ -5,7 +5,7 @@ proxies egress through the one per-host gateway, replacing the per-bottle
 companion container removed in #385.

 The order differs from docker's, forced by Apple Container 1.0.0 having no
-`--ip` (see `infra_launch`): the agent is started *before* it is
+`--ip` (see `consolidated_launch`): the agent is started *before* it is
 registered, because its DHCP-assigned address — the attribution key — does not
 exist until then.

@@ -64,7 +64,7 @@ from . import nested_containers as nested_containers_mod
 from .bottle_plan import MacosContainerBottlePlan
 from ...orchestrator.store.config_store import resolve_teardown_timeout
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME, new_env_var_secret
-from .infra_launch import (
+from .consolidated_launch import (
    GatewayEndpoint,
    ensure_gateway,
    register_agent,
@@ -2,7 +2,7 @@

 Register a bottle with the orchestrator and provision its git-gate state into
 the shared gateway (`provision_bottle`), and the inverse teardown
-(`deprovision_bottle`). Backend-neutral — each backend's infra_launch
+(`deprovision_bottle`). Backend-neutral — each backend's consolidated_launch
 drives these through its own `GatewayTransport` rather than re-implementing
 them. The git-gate half of the work lives in `provision_gateway`.
 """
@@ -26,10 +26,10 @@ from .base import ActiveAgent, BackendStatus, BottleBackend
 # Tests may replace _backends with a {name: fake} dict via patch.object;
 # _get_backends() returns the current module-level value as-is when it
 # is not None, so test fakes take effect without triggering real imports.
-_backends: dict[str, BottleBackend[Any, Any, Any]] | None = None
+_backends: dict[str, BottleBackend[Any, Any]] | None = None


-def _get_backends() -> dict[str, BottleBackend[Any, Any, Any]]:
+def _get_backends() -> dict[str, BottleBackend[Any, Any]]:
    """Return the registry of all backend instances, loading lazily on first call."""
    global _backends  # pylint: disable=global-statement
    if _backends is None:
@@ -48,7 +48,7 @@ def get_bottle_backend(
    name: str | None = None,
    *,
    prompt: bool = True,
-) -> BottleBackend[Any, Any, Any]:
+) -> BottleBackend[Any, Any]:
    """Resolve the bottle backend.

    `name` precedence:
@@ -122,18 +122,12 @@ class Gateway(abc.ABC):
        return

    @abc.abstractmethod
-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        """Bind the gateway to this orchestrator and bring it up: store the URL +
        the pre-minted `gateway` token as instance state, then (re)start the
        gateway unit carrying the mitmproxy CA + that token, resolving policy
        against `orchestrator_url`. Idempotent — a healthy, current gateway on
-        the same binding is left alone; a changed binding reconciles it.
-
-        Returns True when the gateway was actually (re)brought up (a cold boot,
-        so its mitmproxy minted a fresh CA and lost its per-bottle state), False
-        when a healthy current gateway was left untouched. The bring-up flow
-        uses this as the cold-boot signal that gates the running-bottle
-        reconcile (`attach_bottled_agents_to_gateway`, PRD 0081)."""
+        the same binding is left alone; a changed binding reconciles it."""

    @abc.abstractmethod
    def is_running(self) -> bool:
@@ -0,0 +1,48 @@
+"""How to tell a user to re-run this CLI.
+
+`bot-bottle …` is the right thing to print for anything the user runs as
+themselves — it is on their PATH, since that is how they got here.
+
+Under `sudo` it is not. sudo replaces PATH with sudoers' `secure_path`
+(`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` on Debian and
+Ubuntu, similar elsewhere), which deliberately excludes user-writable
+directories. Both supported install paths put the entry point in one of those:
+pipx uses `~/.local/bin`, and `install.sh`'s venv fallback symlinks there too.
+So `sudo bot-bottle …` fails with "command not found" for exactly the users who
+followed the documented install, while working for anyone who happened to
+install system-wide — which is why it survives review so easily.
+
+Naming the absolute path sidesteps secure_path entirely.
+"""
+
+from __future__ import annotations
+
+import os
+import shutil
+import sys
+
+
+def self_path() -> str:
+    """Absolute path to this CLI's entry point.
+
+    Falls back to the bare name when the entry point cannot be resolved (an
+    unusual invocation such as `python -m`), because a slightly wrong hint is
+    better than a traceback while reporting an unrelated problem.
+    """
+    argv0 = sys.argv[0] or "bot-bottle"
+    resolved = shutil.which(argv0) or argv0
+    if not os.path.isabs(resolved):
+        if os.path.exists(resolved):
+            resolved = os.path.abspath(resolved)
+        else:
+            return "bot-bottle"
+    return resolved
+
+
+def sudo_command(*args: str) -> str:
+    """A copy-pasteable `sudo …` invocation of this CLI.
+
+    >>> sudo_command("backend", "setup", "--backend=firecracker")
+    'sudo /home/u/.local/bin/bot-bottle backend setup --backend=firecracker'
+    """
+    return " ".join(["sudo", self_path(), *args])
@@ -50,12 +50,6 @@ class ReprovisionBody(_StrictModel):
    env_var_secret: StrictStr


-class SecretBody(_StrictModel):
-    name: StrictStr
-    value: StrictStr
-    env_var_secret: StrictStr
-
-
 class ReconcileBody(_StrictModel):
    live_source_ips: list[StrictStr]
    grace_seconds: float | None = None
@@ -265,21 +259,6 @@ def create_app(orch: OrchestratorCore, *, signing_key: str) -> FastAPI:
            return {"reprovisioned": True}
        raise HTTPException(404, "no stored secrets for this bottle")

-    @app.post("/bottles/{bottle_id}/secret")
-    def update_secret(bottle_id: str, body: SecretBody) -> dict[str, object]:
-        # Update ONE egress token for a running bottle in place — the
-        # single-secret form of reprovision, for refreshing a short-lived host
-        # credential (e.g. the Codex access token) without a relaunch. cli-only
-        # (not in _GATEWAY_ROUTES): a bottle must never set its own tokens.
-        if orch.update_agent_secret(
-            bottle_id,
-            _required(body.name, "name"),
-            _required(body.value, "value"),
-            _required(body.env_var_secret, "env_var_secret"),
-        ):
-            return {"updated": True}
-        raise HTTPException(404, "no such bottle")
-
    @app.delete("/bottles/{bottle_id}")
    def teardown(bottle_id: str) -> dict[str, object]:
        if orch.teardown_bottle(bottle_id):
@@ -184,27 +184,6 @@ class OrchestratorClient:
            )
        return True

-    def update_agent_secret(
-        self, bottle_id: str, name: str, value: str, env_var_secret: str,
-    ) -> bool:
-        """Update ONE egress token for a running bottle in place
-        (`POST /bottles/<id>/secret`) — the single-secret form of
-        `reprovision_gateway`, for pushing a freshly-refreshed host credential
-        into a bottle without a relaunch. Returns True on success, False when the
-        orchestrator doesn't know the bottle (404)."""
-        status, _ = self._request(
-            "POST",
-            f"/bottles/{bottle_id}/secret",
-            {"name": name, "value": value, "env_var_secret": env_var_secret},
-        )
-        if status == 404:
-            return False
-        if not 200 <= status < 300:
-            raise OrchestratorClientError(
-                f"update_agent_secret {bottle_id}: HTTP {status}"
-            )
-        return True
-
    def teardown_bottle(self, bottle_id: str) -> bool:
        """Tear a bottle down (`DELETE /bottles/<id>`). False if the
        orchestrator didn't know it (404) — idempotent for cleanup paths."""
@@ -379,30 +379,6 @@ class OrchestratorCore:
        self._tokens[bottle_id] = decrypted
        return True

-    def update_agent_secret(
-        self, bottle_id: str, name: str, value: str, env_var_secret: str,
-    ) -> bool:
-        """Update ONE egress token for a known bottle in place — the
-        single-secret form of ``reprovision_from_secret``.
-
-        Sets the in-memory token AND upserts the single re-encrypted row under
-        *env_var_secret* (the same key the rest of the rows are encrypted with, so
-        the whole set stays decryptable by a later ``reprovision_from_secret``),
-        leaving every other token untouched. Returns False if the bottle is
-        unknown.
-
-        Unlike ``reprovision_from_secret`` (which restores the values captured at
-        launch), this pushes a *caller-supplied* value — used to refresh a
-        short-lived host credential (e.g. the Codex access token) into a
-        still-running bottle without a relaunch."""
-        from .store.secret_store import encrypt_value
-        if self.registry.get(bottle_id) is None:
-            return False
-        self._tokens.setdefault(bottle_id, {})[name] = value
-        self.registry.store_agent_secret(
-            bottle_id, name, encrypt_value(env_var_secret, value))
-        return True
-
    # --- consolidated gateway ----------------------------------------------

    def gateway_status(self) -> dict[str, object]:
@@ -370,30 +370,6 @@ class RegistryStore(DbStore):
            )
        self._chmod()

-    def store_agent_secret(
-        self,
-        bottle_id: str,
-        key: str,
-        encrypted_value: str,
-        secret_type: str = "injected_env_var",
-    ) -> None:
-        """Upsert ONE encrypted secret row (env-var name → ciphertext) for
-        *bottle_id*, leaving the bottle's other secrets untouched — the per-key
-        counterpart of ``store_agent_secrets``' replace-all. Delete-then-insert
-        because the table carries no unique constraint to `ON CONFLICT` against."""
-        with self._connection() as conn:
-            conn.execute(
-                "DELETE FROM bottled_agent_secrets "
-                "WHERE bottled_agent_id = ? AND key = ? AND type = ?",
-                (bottle_id, key, secret_type),
-            )
-            conn.execute(
-                "INSERT INTO bottled_agent_secrets "
-                "(bottled_agent_id, key, value, type) VALUES (?, ?, ?, ?)",
-                (bottle_id, key, encrypted_value, secret_type),
-            )
-        self._chmod()
-
    def get_agent_secrets(
        self,
        bottle_id: str,
@@ -1,65 +0,0 @@
-# ADR 0006: Shared behaviour lives in the base backend
-
- **Status:** Accepted
- **Date:** 2026-07-27
- **Deciders:** didericis
-
-## Context
-
-`BottleBackend` has three concrete implementations (Firecracker, docker,
-macOS-container). Cross-backend operations can be expressed two ways:
-
-1. **Abstract method per backend** — the base declares `@abstractmethod foo()`
-   and each backend writes its own `foo()` end to end.
-2. **Template method** — the base owns the *control flow* of `foo()` as a
-   concrete method and each backend overrides small **primitives** it calls.
-
-Form 1 gives each backend total freedom, which is exactly the problem: the three
-implementations drift. The bring-up reconcile (PRD 0081) first shipped as form 1
-and immediately diverged — each backend re-implemented "gather CA → list running
-bottles → push to each", and one backend's copy silently skipped failures while
-another's aborted, an inconsistency caught only in review (#519). The behaviour
-that must be identical (the loop, the error policy, the ordering) was duplicated
-in the place least likely to stay in sync.
-
-## Decision
-
-Prefer the template-method form for cross-backend behaviour: **encode shared
-control flow and policy in the base backend; backends override primitives, not
-control flow.** Lean toward *less* freedom in the concrete backends and *more*
-consistency in the base.
-
-Concretely, a cross-backend operation on `BottleBackend` should be a concrete
-method that:
-
- owns the sequencing, iteration, and error policy (e.g. fail-hard vs.
-  best-effort, fail-fast vs. attempt-all-then-aggregate); and
- delegates only the irreducibly backend-specific steps to `@abstractmethod`
-  primitives with narrow, well-typed signatures.
-
-The first application is `attach_bottled_agents_to_gateway()` (PRD 0081): the
-base gathers resources once, attempts every running bottle, and raises an
-aggregate `InfraLaunchError` if any failed; backends supply only
-`_gateway_attach_resources()`, `_running_bottles()`, and
-`_attach_bottle_to_gateway()`.
-
-A backend-specific `@abstractmethod` with no shared control flow (e.g.
-`_launch_impl`, `enumerate_active`) stays as form 1 — this ADR is about
-operations whose *shape* should be uniform, not about banning abstract methods.
-
-## Consequences
-
- The error policy, ordering, and loop for a cross-backend operation have one
-  source of truth; a new backend cannot forget them or diverge.
- New backends implement smaller, more focused primitives instead of re-deriving
-  a whole flow.
- The base carries more logic and needs generic-enough primitive signatures
-  (e.g. a backend-specific attach-target type parameter on `BottleBackend`).
- Genuinely backend-specific one-offs still use a plain abstract method; the
-  guidance is a default, not an absolute.
-
-## Links
-
- PRD 0081 (`docs/prds/0081-reprovision-gateway-state-on-bringup.md`).
- Review that prompted this: PR #519.
- `bot_bottle/backend/base.py` (`BottleBackend.attach_bottled_agents_to_gateway`).
@@ -55,13 +55,3 @@ agent has two observable states:

 Shorthand for an active (running) Bottled Agent. Used in the supervisor TUI
 and discovery layer to mean "a bottle whose agent runtime is currently up."
-
-## Infra
-
-The per-host pair of shared services every bottle attaches to: the
-**orchestrator** (control plane — holds the signing key, owns the registry DB,
-mints tokens) and the **gateway** (data plane — TLS-inspecting egress proxy,
-git-gate, supervise). Brought up as an idempotent singleton pair by each
-backend's `InfraService` (two containers on docker/macOS, two microVMs on
-Firecracker). The per-backend `infra_launch` module composes the bring-up and
-bottle register/provision sequence against this pair.
@@ -1,136 +0,0 @@
-# PRD 0081: Reprovision gateway-dependent state on gateway bring-up
-
- **Status:** Active
- **Author:** claude
- **Created:** 2026-07-26
- **Issue:** #516
-
-## Summary
-
-When the gateway is (re)built, reconcile every already-running bottle against the
-fresh gateway instead of persisting the gateway's state. On a gateway cold boot,
-the gateway reconciles all live bottles in one flow: **replace** each agent's
-trusted CA with the freshly-minted gateway CA, **re-provision** each bottle's
-git-gate repos + creds onto the gateway, and **restore** each bottle's egress
-tokens. One mechanism across all three services; the CA rotates for free on every
-bring-up.
-
-## Problem
-
-A gateway rebuild/restart silently breaks every already-running bottle:
-
- **CA (#510).** The gateway's mitmproxy mints a new CA on a fresh rootfs; agents
-  still trust the old one, so egress fails TLS verification (`SSL certificate
-  verification failed`).
- **git-gate (#512).** Per-bottle bare repos (`/git/<id>`) + deploy creds
-  (`/git-gate/creds/<id>`) live in the gateway's ephemeral rootfs; a rebuild wipes
-  them and the agent 404s on fetch/push.
-
-Both are the same root cause: per-bottle gateway-dependent state is provisioned
-**once, at bottle launch**, and nothing restores it for already-running bottles
-when the gateway comes back. The existing launch-time reprovision
-(`reprovision_bottles`) restores **only** egress tokens, and only as a side effect
-of the *next* launch.
-
-Persisting the state (a host bind-mount / a per-VM data volume per service) was
-prototyped and rejected: it differs per service (a volume for the CA, another for
-git-gate, reprovision for tokens), it pins the CA static forever (no rotation),
-and it adds volume surface that `docker volume prune` / a wiped cache can silently
-destroy (the original #450 failure mode).
-
-## Goals / Success Criteria
-
- A gateway (re)boot restores **all** running bottles' gateway-dependent state
-  with no manual step and no relaunch — the agent's next egress / fetch / push
-  just works.
- **One** reconcile flow covering CA, git-gate, and egress tokens, rather than a
-  different mechanism per service.
- The CA **rotates** on every gateway bring-up (no long-lived CA), distributed to
-  running agents by the same reconcile.
- Reconcile failures are **loud, never hidden**: if a running bottle can't be
-  re-attached to the fresh gateway, the bring-up fails and names the bottle
-  rather than leaving it silently unable to reach the gateway (a bottle whose
-  egress just starts failing TLS is worse than a loud failure). Every bottle is
-  attempted first and the failures are raised together, so one bring-up surfaces
-  the full blast radius. (Reversed from an earlier "tolerate per-bottle
-  failures" draft after #519 review.)
- **All backends** (Firecracker, docker, macOS) reconcile through the *same*
-  contract — an abstract method on the backend base class, so a new backend
-  cannot forget to implement it and none drifts onto a bespoke mechanism.
-
-## Non-goals
-
- Deliberate mid-session CA rotation *without* a gateway restart — `rotate_ca`
-  stays for that operator action.
- Changing source-IP attribution, `/resolve`, or the plane split (#469).
-
-## Design
-
-**The contract — `attach_bottled_agents_to_gateway()`, a concrete flow on the
-backend base.** Reconciling running bottles against the current gateway is a
-backend responsibility (only the backend can enumerate its agents and reach them
-— firecracker over SSH, docker/macOS over `exec`/`cp`). To keep every backend
-reconciling identically, the *control flow* lives in a concrete
-`attach_bottled_agents_to_gateway()` on `BottleBackend` (`backend/base.py`) and
-each backend overrides only three primitives:
-
-1. `_gateway_attach_resources()` — gather what a bottle needs to attach (the
-   gateway CA now; egress secrets + git-gate state as this grows).
-2. `_running_bottles()` — enumerate the live bottles as backend handles.
-3. `_attach_bottle_to_gateway(bottle, resources)` — push the resources into one
-   bottle, raising `InfraLaunchError` (naming it) on failure.
-
-The base gathers resources once, attempts every bottle, and — fail hard, never
-skip — raises an aggregate `InfraLaunchError` if any failed, so a backend can't
-drift onto a bespoke loop or a silent skip. (Encoding shared behaviour in the
-base backend, with subclasses overriding primitives rather than control flow, is
-[ADR 0006](../decisions/0006-shared-behavior-in-base-backend.md).) The host
-calls it whenever the gateway is (re)brought up. It reprovisions **all**
-registered bottles' gateway-dependent state: CA, git-gate, and egress tokens.
-
-**Trigger — the gateway bring-up path.** The host calls
-`attach_bottled_agents_to_gateway()` only when the gateway was actually
-(re)brought up — the cold-boot branch of the infra bring-up (e.g.
-`FirecrackerInfraService.ensure_running` after it boots a fresh pair), never on an
-adopt of a healthy, current gateway (state intact). So it fires exactly when the
-gateway was (re)booted — including orchestrator restarts, since the pair boots
-together — and there is no bare-restart path that bypasses bring-up.
-
-**Per-backend implementation.** Each backend's `attach_bottled_agents_to_gateway`
-enumerates its live bottles and, for each, reconciles the three services against
-the current gateway. The firecracker implementation, once the gateway VM is up
-and its CA is available:
-
-1. Map each live bottle's guest IP → `bottle_id` from the orchestrator registry
-   (`list_bottles`).
-2. Install the **current** shared git-gate hooks (`git_gate_render_hook` /
-   `git_gate_render_access_hook`) into the fresh gateway once — rendered from
-   code, never from a bottle's possibly-stale state dir.
-3. For each live agent VM (enumerated from its run dir):
-   - **CA:** SSH the current gateway CA into the agent's trust store and run
-     `update-ca-certificates` (unconditional replace — there is one gateway, so no
-     fingerprint match is needed).
-   - **git-gate:** rebuild the bottle's upstreams from its persisted git-gate
-     state dir (deploy key, known_hosts, upstream URL) and re-init its bare repos
-     + per-repo creds under `/git/<bottle_id>`.
-   - **egress token:** read the agent's `ENV_VAR_SECRET` and feed
-     `reprovision_bottles`, restoring the orchestrator's in-memory tokens.
-4. If any per-bottle step fails, the reconcile attempts the rest and then raises
-   an aggregate naming every bottle that failed — bring-up fails loudly rather
-   than leaving a bottle silently unable to reach the gateway.
-
-**Retire the persistence prototype.** No CA data volume, no git-gate data volume
-(the abandoned PRs #511 / #513). The launch-time `_reprovision_running_bottles`
-call folds into this bring-up reconcile, so egress tokens are restored on the same
-cold-boot trigger (an adopt needs no restore — the orchestrator never restarted).
-
-**CA rotation.** Because the gateway rootfs is ephemeral, every cold boot mints a
-fresh CA; reconcile is what distributes it, so a routine gateway rebuild doubles
-as a CA rotation with zero extra machinery.
-
-## Open questions
-
- Docker's existing host-bind-mounted CA (`host_gateway_ca_dir`): once docker's
-  `attach_bottled_agents_to_gateway` pushes the CA to running agents on bring-up,
-  the bind-mount is redundant — drop it (so docker rotates like firecracker) or
-  keep it as belt-and-suspenders? Leaning drop, for one behaviour across backends.
@@ -0,0 +1,183 @@
+# Testing a clean bot-bottle install on Linux
+
+How do you exercise `install.sh` the way a brand-new user would — on a
+pristine Linux environment you can throw away afterward — *without*
+polluting your daily-driver host, and across the several package-management
+regimes Linux fragments into? This is the Linux counterpart to
+[`testing-clean-install-on-macos.md`](testing-clean-install-on-macos.md);
+the conclusion is different because Linux gives us a boundary macOS doesn't.
+
+## Summary
+
+On macOS the honest options were a throwaway user or a VM, and the throwaway
+user won on pragmatics (nested virtualization is gated to M3+). On Linux the
+calculus flips: a **disposable KVM virtual machine, booted from a distro
+cloud image and deleted per run, is both the cleanest boundary and the one
+that lets a single harness cover Ubuntu, Fedora, Arch, Alpine, and NixOS**.
+The host already requires KVM for the Firecracker backend, so the VM is cheap
+here.
+
+The harness lives at [`scripts/linux-install-test.sh`](../../scripts/linux-install-test.sh).
+Per run it caches one read-only base image, boots a throwaway copy-on-write
+overlay (`qemu-img create -b base`), installs the distro's prerequisites,
+pipes *this checkout's* `install.sh` into the guest exactly as `curl … | sh`
+would, asserts the CLI installed, and deletes the overlay — the Linux
+equivalent of `docker run --rm`, for a whole machine.
+
+## Why a VM, not a container or a throwaway user
+
+| Mechanism | Why it's the wrong boundary here |
+|---|---|
+| **Container** (`docker run --rm`) | Shares the host kernel and ships a deliberately minimal userland — no systemd, a stubbed-out package manager story, and (crucially) it doesn't reproduce the *externally-managed Python* (PEP 668) that real desktop/server installs put in front of the user. It tests "does install.sh run in a container," not "does it run on a real distro." |
+| **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
+| **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
+
+## Two variants: `test` (bare host) and `test-ready` (prepared host)
+
+`install.sh` never installs a backend, and never installs its own toolchain
+prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
+runs `doctor`, which *reports* what's missing
+([`install.sh`](../../install.sh) header,
+[`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
+That leaves two distinct things worth testing, split into two subcommands that
+mirror the macOS harness's `test` / `test-ready` convention (there the split is
+the backend service; here it is the toolchain the installer needs):
+
+- **`test`** — `install.sh` runs on the **bare cloud image**, prerequisites and
+  all left as the vendor ships them. This exercises `install.sh`'s own
+  prerequisite-guard logic — the entire first half of the script (python
+  version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
+- **`test-ready`** — the harness installs python3 + git + pipx first (the
+  `prereqs` step), then runs `install.sh`. This is the *prepared-host happy
+  path*: does a clean install actually land and produce a working CLI?
+
+**Pass criteria differ by variant:**
+
+| Variant | PASS when |
+|---|---|
+| `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
+| `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
+
+Neither variant requires a green `doctor`: inside the VM there is no nested KVM
+or Docker, so **the backend is correctly reported not-ready** — install.sh does
+not install a backend and cannot regress one, and this harness does not
+provision the Docker backend. This is where Linux necessarily diverges from the
+macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
+makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
+verdict instead classifies `doctor`'s output the way the macOS harness does — a
+`Traceback` is an install defect (fail), a missing `python`/`config` line is a
+fail, a not-ready backend is reported — so a genuine installer regression (a
+broken shim, an import error, a botched PATH) stays visible.
+
+`test-all` runs the full matrix — every distro × both variants — each cell in
+its own throwaway VM, and prints a per-cell PASS/FAIL summary.
+
+## The distro matrix is the point
+
+Each distro exercises a different corner of the installer:
+
+| Distro | Cloud image | What it stresses |
+|---|---|---|
+| **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. |
+| **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. |
+| **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. |
+| **Alpine** | Alpine `nocloud_` (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
+| **NixOS** | locally built with `nixos-generators` | No FHS `~/.local` on PATH by default; `nix profile install` prereqs; pipx laying a self-contained venv on a non-FHS host. |
+
+### Validation run (2026-07-27) — full green
+
+Full matrix on the delphi KVM host, QEMU 11.0.2, both variants × all five
+distros passing:
+
+| Distro | `test` (bare) | `test-ready` (prepared) |
+|---|---|---|
+| Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green |
+| Fedora 44 | ✅ declines at git gate | ✅ installs |
+| Arch (latest) | ✅ declines at git gate | ✅ installs |
+| Alpine 3.21 | ✅ declines at git gate | ✅ installs |
+| NixOS 24.11 | ✅ declines (no python3) | ✅ installs |
+
+The bare `test` sees `install.sh` decline soundly — exit 1 at the
+git-for-git-specs gate on the Debian/Fedora/Arch/Alpine images (they ship
+python3 but not git), and at the python3 gate on NixOS (no python3 on PATH) —
+and `test-ready` installs cleanly with `doctor` reporting a usable python and
+config (backends all not-ready, as expected in a plain VM).
+
+Getting to green surfaced and fixed a series of real defects:
+
+- **Fedora 41 was EOL/404** → bumped to 44.
+- The liveness probe used `bot-bottle --version`, which the CLI does not
+  implement (unknown args die non-zero), so every *successful* install was
+  misreported as failed → switched to `bot-bottle --help`.
+- **Alpine** needed three fixes: the `generic_` image ignores a NoCloud seed
+  (switched to the `nocloud_` variant); OpenRC does not auto-start sshd after
+  cloud-init injects the key (start it via `runcmd`); and Alpine's non-PAM
+  sshd refuses pubkey auth for a cloud-init-*locked* account (give it a
+  throwaway password). It also has no `sudo` by default (install it via
+  cloud-init `packages:`).
+- **NixOS** publishes no downloadable cloud qcow2 (its cloud images are
+  Hydra-built AMIs), so the harness builds one with `nixos-generators`
+  ([`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)):
+  cloud-init for the key, flakes enabled, deliberately no python/git/pipx. The
+  `test-ready` prereq install pins `nixpkgs/nixos-24.11` because the guest's
+  default unstable registry builds pipx from source (and its test suite
+  currently fails to build).
+- Two harness-hygiene bugs also fixed: `cmd_down` left `serial.log` behind
+  (orphaned run dirs), and the teardown trap was armed after `cmd_up`, leaking
+  a VM when `wait_for_ssh` timed out.
+
+In `test-ready` the harness installs `python3 + git + pipx` first on each
+distro (install.sh installs none of them), so all five drive the recommended
+pipx path. `test` then removes that scaffolding and lets each distro's bare
+image collide with install.sh's guards — on most cloud images python3 is
+present (cloud-init needs it) but git and pipx are not, so install.sh is
+expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
+an actionable message. Both are legitimate, and the two variants together cover
+the whole first half of the installer as well as the happy path.
+
+## What a clean install touches (the footprint that decides "wipeable")
+
+| Artifact | Location | In the guest's `$HOME`? | Survives VM teardown? |
+|---|---|---|---|
+| Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
+| pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
+| pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
+| **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix profile` | ❌ | ❌ **overlay deleted** |
+
+Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
+footprint *survives*), **every row here dies with the overlay** — that is the
+VM's whole advantage. The cached base image is read-only backing and is the
+only thing that persists between runs, on purpose.
+
+## Design notes baked into the harness
+
+- **User-mode networking** (`-netdev user,hostfwd=tcp:127.0.0.1:PORT-:22`):
+  no root, no bridge, no host network state touched. Only SSH is forwarded.
+- **cloud-init seed ISO** (`cloud-localds`) injects an ephemeral SSH keypair
+  and a passwordless-sudo login. The keypair is generated per run and deleted
+  on teardown; the guest can't be logged into after it's gone.
+- **Copy-on-write overlay**: the cached base is never mutated, so a corrupt or
+  interrupted run can't poison the cache; downloads land at `*.partial` and
+  are renamed only after checksum verification.
+- **Checksums**: verified against each vendor's published sums file at
+  download time (GNU `hash  file`, bare-hash, and Fedora's BSD
+  `SHA256 (file) = hash` formats are all handled). Alpine ships `.sha512` only
+  (this verifier is sha256) so it is skipped; NixOS is built locally, not
+  downloaded, so there is nothing to verify.
+- **NixOS is built, not downloaded**: `ensure_base_image` runs
+  `nixos-generate -f qcow` against
+  [`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)
+  once and caches the result; the per-run seed/overlay flow is otherwise
+  identical to the downloaded distros.
+- **`test-all`** runs every distro × both variants (`test` and `test-ready`),
+  each cell in its own subshell on its own forwarded port, so one cell's
+  failure (or teardown trap) can't abort the matrix; it prints a per-cell
+  PASS/FAIL summary.
+
+## Not wired into PR CI
+
+Like the macOS harness, the runtime is host-specific (needs `/dev/kvm`,
+`qemu`, and `cloud-localds`) and is not exercised by the Linux pull-request
+runner. It is validated statically (`bash -n`, `shellcheck`) and run by hand
+on a KVM-capable host. The cloud-image URLs in the `DISTRO` table are the one
+place to bump when a distro cuts a newer build.
@@ -0,0 +1,24 @@
+# NixOS image for scripts/linux-install-test.sh.
+#
+# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
+# AMIs), so the harness BUILDS this one with nixos-generators (-f qcow). It is
+# deliberately minimal — no python3/git/pipx — so the bare `test` variant is
+# genuinely under-provisioned and exercises install.sh's guards; `test-ready`
+# provisions them with `nix profile install` (hence flakes below).
+{ lib, ... }:
+{
+  # Consume the same NoCloud seed the other distros use: cloud-init injects the
+  # per-run ephemeral SSH key for root. Leave networking to NixOS's default
+  # dhcpcd (QEMU user-mode NAT) — enabling cloud-init's networkd here conflicts
+  # with dhcpcd and can drop the guest's network.
+  services.cloud-init.enable = true;
+  services.cloud-init.network.enable = false;
+
+  services.openssh.enable = true;
+  services.openssh.settings.PermitRootLogin = lib.mkForce "prohibit-password";
+
+  # Flakes so the `test-ready` prereq step can `nix profile install nixpkgs#...`.
+  nix.settings.experimental-features = [ "nix-command" "flakes" ];
+
+  system.stateVersion = "24.11";
+}
@@ -0,0 +1,753 @@
+#!/usr/bin/env bash
+# Clean-install test harness for the Linux path.
+#
+# Exercises install.sh the way a brand-new user would, inside a THROWAWAY
+# QEMU/KVM virtual machine that is booted from a distro cloud image and
+# deleted afterward. install.sh's entire footprint is user-home-local (the
+# pipx venv under ~/.local, the ~/.bot-bottle config dir, and a printed PATH
+# hint), so a fresh VM's fresh $HOME is the clean surface we want — and unlike
+# a throwaway user account, tearing the VM down also wipes any OS-level
+# prerequisites installed into it, so the reset is total. Full rationale in
+# docs/research/testing-clean-install-on-linux.md.
+#
+# Why a VM and not a container or a throwaway user: a container shares the
+# host kernel and cannot exercise a genuinely pristine OS (systemd, the distro
+# package manager, PEP 668 externally-managed Python) the way a real guest
+# does, and a throwaway user leaves every system package it installs behind.
+# The host already needs KVM for the Firecracker backend, so a per-run,
+# copy-on-write VM is cheap here: one cached base image, a throwaway overlay
+# per run (`qemu-img create -b base`), deleted on teardown — the Linux
+# equivalent of `docker run --rm`, but for a whole machine.
+#
+# Usage:
+#   ./scripts/linux-install-test.sh test        # up -> run -> verdict -> down (bare host)
+#   ./scripts/linux-install-test.sh test-ready  # ... with prerequisites installed first
+#   ./scripts/linux-install-test.sh test-all    # every distro × both variants, with a summary
+#   ./scripts/linux-install-test.sh up          # fetch base image, boot a fresh VM
+#   ./scripts/linux-install-test.sh prereqs     # install python3 + git + pipx in the VM
+#   ./scripts/linux-install-test.sh run         # pipe install.sh into the VM
+#   ./scripts/linux-install-test.sh status      # VM reachable? is the install sound?
+#   ./scripts/linux-install-test.sh down        # kill the VM, delete overlay + seed (the reset)
+#   ./scripts/linux-install-test.sh ssh         # open an interactive shell in the running VM
+#
+# TWO TEST VARIANTS, because "does install.sh handle an unprepared host" and
+# "does a prepared host get a clean install" are different questions (mirrors
+# the macOS harness's test / test-ready split):
+#
+#   test        A Linux system WITHOUT the prerequisites set up — the default
+#               state of a stock cloud image (python3 is usually present for
+#               cloud-init, but git and pipx are not). This exercises
+#               install.sh's own prerequisite-guard logic. It is SOUND — a
+#               PASS — when install.sh EITHER installs cleanly (the image
+#               already had enough) OR declines with one of its own recognized,
+#               actionable errors (missing python3/git, no usable pip, PEP 668
+#               externally-managed). A crash or an unrecognized failure fails.
+#
+#   test-ready  A Linux system WITH the prerequisites satisfied — the harness
+#               installs python3 + git + pipx first (see the DISTRO table),
+#               then runs install.sh. A graceful decline is no longer good
+#               enough here: the install MUST land, the entry point must run,
+#               and doctor must report a usable python and config without
+#               crashing.
+#
+# Neither variant requires a green backend. install.sh does not install a
+# backend and cannot regress one, and inside a plain VM there is no nested KVM
+# for Firecracker; the harness does not provision the Docker backend either. So
+# backend readiness is reported, not required (this is where Linux necessarily
+# diverges from the macOS test-ready, which reaches the host backend).
+# BB_TEST_REQUIRE_BACKEND=1 makes it fatal anyway, for a nested-virt host.
+#
+# Config via env:
+#   BB_TEST_DISTRO         ubuntu | fedora | arch | alpine | nixos (default: ubuntu)
+#   BB_TEST_SSH_PORT       host port forwarded to the guest's :22  (default: 2222)
+#   BB_TEST_CACHE_DIR      where base images are cached            (default: ~/.cache/bot-bottle-install-test)
+#   BB_TEST_RUN_DIR        per-run scratch (overlay, seed, key, …) (default: a mktemp dir)
+#   BB_TEST_MEM_MB         guest RAM                               (default: 2048)
+#   BB_TEST_CPUS           guest vCPUs                             (default: 2)
+#   BB_TEST_DISK           overlay virtual size                   (default: 12G)
+#   BB_TEST_BOOT_TIMEOUT   seconds to wait for SSH after boot      (default: 300)
+#   BB_TEST_KEEP           1 = the test cycles skip teardown, to poke at a failure
+#   BB_TEST_REQUIRE_BACKEND 1 = make a not-ready backend fatal (needs nested virt)
+#   BB_TEST_INSTALL_URL    curl this install.sh in the guest instead of piping the local checkout
+#   BB_TEST_SKIP_VERIFY    1 = skip base-image checksum verification (not recommended)
+#   BOT_BOTTLE_INSTALL_SPEC   passed through to install.sh (pip / git spec)
+#
+# Notes:
+#   * Needs /dev/kvm, qemu-system-x86_64, and cloud-localds (cloud-image-utils
+#     / cloud-utils); the nixos distro additionally needs nixos-generate. On the
+#     NixOS host:  nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#nixos-generators
+#   * Networking is user-mode (`-netdev user,hostfwd`) so the harness needs no
+#     root, no bridge, and touches no host network state. Only SSH is forwarded.
+#   * The cloud-image URLs in the DISTRO table are the one place to bump when a
+#     distro cuts a new build; each is verified against the vendor's published
+#     checksum at download time (guarding against truncated/corrupt pulls).
+
+set -euo pipefail
+
+DISTRO="${BB_TEST_DISTRO:-ubuntu}"
+SSH_PORT="${BB_TEST_SSH_PORT:-2222}"
+CACHE_DIR="${BB_TEST_CACHE_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/bot-bottle-install-test}"
+MEM_MB="${BB_TEST_MEM_MB:-2048}"
+CPUS="${BB_TEST_CPUS:-2}"
+DISK="${BB_TEST_DISK:-12G}"
+BOOT_TIMEOUT="${BB_TEST_BOOT_TIMEOUT:-300}"
+
+_SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+_REPO_ROOT="$(cd "$_SCRIPT_DIR/.." && pwd)"
+
+# Per-run scratch. Persisted across sub-commands (up/prereqs/run/status/down)
+# via a marker file so `up` in one invocation and `down` in the next find the
+# same VM; the test cycles set RUN_DIR themselves and never write the marker.
+RUN_DIR="${BB_TEST_RUN_DIR:-}"
+
+# Set by the test cycles, which chain the steps and suppress the per-step "next
+# command" hints. _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL are the per-variant
+# knobs the shared cycle and teardown read.
+IN_TEST=0
+_STEPS=4
+_PASS_CLAIM=""
+_REQUIRE_INSTALL=0
+
+# --- distro table ----------------------------------------------------
+# For each distro:  cloud-image URL | checksum-file URL | default SSH user |
+# prerequisite-install command (run in the guest; uses sudo when user != root).
+#
+# The prerequisite command installs python3 + git + pipx — install.sh installs
+# none of them — so `test-ready` (and the `prereqs` sub-command) drive
+# install.sh down its recommended pipx path. Bump the URLs here when a distro
+# publishes a newer build.
+declare -A IMAGE_URL SUM_URL SSH_USER PREREQ
+
+IMAGE_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
+SUM_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/SHA256SUMS"
+SSH_USER[ubuntu]="ubuntu"
+PREREQ[ubuntu]="sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3 git pipx"
+
+IMAGE_URL[fedora]="https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2"
+SUM_URL[fedora]="https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-44-1.7-x86_64-CHECKSUM"
+SSH_USER[fedora]="fedora"
+PREREQ[fedora]="sudo dnf install -y python3 git pipx"
+
+IMAGE_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-cloudimg.qcow2"
+SUM_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-cloudimg.qcow2.SHA256"
+SSH_USER[arch]="arch"
+PREREQ[arch]="sudo pacman -Sy --noconfirm python git python-pipx"
+
+# Use the *nocloud_* Alpine variant, not generic_: the generic image probes
+# network datasources and ignores the local NoCloud seed, so cloud-init never
+# runs and the SSH key is never injected. (Only published at .0 patch levels.)
+IMAGE_URL[alpine]="https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/nocloud_alpine-3.21.0-x86_64-bios-cloudinit-r0.qcow2"
+SUM_URL[alpine]=""   # Alpine cloud images ship .sha512 only; this verifier is sha256.
+SSH_USER[alpine]="alpine"
+PREREQ[alpine]="sudo apk add --no-cache python3 git pipx"
+
+# NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
+# AMIs), so the harness BUILDS one with nixos-generators — see build_nixos_image
+# and linux-install-test-nixos.nix. It's externally-managed in its own way (no
+# FHS ~/.local on PATH by default); `nix profile install` provisions the
+# prerequisites into root's profile (the image enables flakes for this).
+IMAGE_URL[nixos]="nix:build"   # sentinel: ensure_base_image builds instead of downloading
+SUM_URL[nixos]=""
+SSH_USER[nixos]="root"
+# Pin to a stable release: the guest's default `nixpkgs` registry is unstable,
+# where pipx isn't in the binary cache and builds from source (its test suite
+# currently fails to build). nixos-24.11 has these cached as substitutes.
+PREREQ[nixos]="nix profile install nixpkgs/nixos-24.11#python3 nixpkgs/nixos-24.11#git nixpkgs/nixos-24.11#pipx"
+
+ALL_DISTROS=(ubuntu fedora arch alpine nixos)
+
+# --- guards ----------------------------------------------------------
+require_linux() {
+    [ "$(uname -s)" = "Linux" ] \
+        || { echo "error: this harness is Linux-only (uname is $(uname -s))" >&2; exit 1; }
+}
+
+require_kvm() {
+    [ -e /dev/kvm ] && [ -r /dev/kvm ] && [ -w /dev/kvm ] \
+        || { echo "error: /dev/kvm is missing or not accessible (add yourself to the 'kvm' group)" >&2; exit 1; }
+}
+
+require_tools() {
+    local missing=()
+    command -v qemu-system-x86_64 >/dev/null 2>&1 || missing+=(qemu-system-x86_64)
+    command -v qemu-img          >/dev/null 2>&1 || missing+=(qemu-img)
+    command -v cloud-localds     >/dev/null 2>&1 || missing+=(cloud-localds)
+    command -v ssh               >/dev/null 2>&1 || missing+=(ssh)
+    command -v curl              >/dev/null 2>&1 || missing+=(curl)
+    if [ "${#missing[@]}" -ne 0 ]; then
+        echo "error: missing required tools: ${missing[*]}" >&2
+        echo "  on NixOS:  nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#openssh nixpkgs#curl" >&2
+        exit 1
+    fi
+}
+
+known_distro() {
+    [ -n "${IMAGE_URL[$DISTRO]:-}" ] \
+        || { echo "error: unknown distro '$DISTRO' (known: ${ALL_DISTROS[*]})" >&2; exit 1; }
+}
+
+# --- run-dir bookkeeping ---------------------------------------------
+# The marker lets prereqs/run/status/down in separate invocations find the VM
+# that `up` started. The test cycles set RUN_DIR themselves and never write it.
+_marker() { echo "${TMPDIR:-/tmp}/bot-bottle-install-test.$DISTRO.run"; }
+
+_ensure_run_dir() {
+    if [ -z "$RUN_DIR" ]; then
+        RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")"
+    fi
+    mkdir -p "$RUN_DIR"
+}
+
+_load_run_dir() {
+    if [ -z "$RUN_DIR" ] && [ -f "$(_marker)" ]; then
+        RUN_DIR="$(cat "$(_marker)")"
+    fi
+    [ -n "$RUN_DIR" ] && [ -d "$RUN_DIR" ]
+}
+
+_ssh_key()  { echo "$RUN_DIR/id_ed25519"; }
+_overlay()  { echo "$RUN_DIR/overlay.qcow2"; }
+_seed()     { echo "$RUN_DIR/seed.iso"; }
+_pidfile()  { echo "$RUN_DIR/qemu.pid"; }
+_serial()   { echo "$RUN_DIR/serial.log"; }
+
+# --- ssh helpers -----------------------------------------------------
+_ssh_opts() {
+    # No host-key pinning: the guest is thrown away every run.
+    printf '%s\0' \
+        -i "$(_ssh_key)" \
+        -p "$SSH_PORT" \
+        -o StrictHostKeyChecking=no \
+        -o UserKnownHostsFile=/dev/null \
+        -o LogLevel=ERROR \
+        -o ConnectTimeout=8 \
+        -o BatchMode=yes
+}
+
+guest() {
+    local -a opts
+    mapfile -d '' -t opts < <(_ssh_opts)
+    ssh "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1" "$@"
+}
+
+wait_for_ssh() {
+    local deadline=$(( SECONDS + BOOT_TIMEOUT ))
+    echo "== waiting for SSH on 127.0.0.1:$SSH_PORT (up to ${BOOT_TIMEOUT}s) =="
+    while [ "$SECONDS" -lt "$deadline" ]; do
+        if guest true 2>/dev/null; then
+            echo "   guest is up"
+            return 0
+        fi
+        # Bail early if QEMU has died — no point waiting out the timeout.
+        if [ -f "$(_pidfile)" ] && ! kill -0 "$(cat "$(_pidfile)")" 2>/dev/null; then
+            echo "error: QEMU exited before SSH came up; see $(_serial)" >&2
+            return 1
+        fi
+        sleep 3
+    done
+    echo "error: timed out waiting for SSH; see $(_serial)" >&2
+    return 1
+}
+
+# --- image cache -----------------------------------------------------
+_base_image() {
+    # One cached file per distro. NixOS is built (not downloaded), so it has a
+    # fixed cache name; the rest are keyed by the image's basename so a URL bump
+    # lands as a new cache entry rather than a stale hit.
+    if [ "$DISTRO" = nixos ]; then
+        echo "$CACHE_DIR/nixos-built.qcow2"
+        return
+    fi
+    local url="${IMAGE_URL[$DISTRO]}"
+    echo "$CACHE_DIR/$DISTRO-$(basename "$url")"
+}
+
+# NixOS has no upstream cloud qcow2; build one with nixos-generators and copy it
+# out of the (immutable, GC-able) store into the cache.
+build_nixos_image() {
+    local out; out="$(_base_image)"
+    [ -f "$out" ] && { echo "== nixos base image cached: $out =="; return 0; }
+    command -v nixos-generate >/dev/null 2>&1 || {
+        echo "error: 'nixos-generate' is required to build the NixOS image" >&2
+        echo "  run inside:  nix shell nixpkgs#nixos-generators nixpkgs#qemu nixpkgs#cloud-utils" >&2
+        return 1
+    }
+    echo "== building NixOS cloud image with nixos-generators (first run is slow) =="
+    local link="$CACHE_DIR/nixos-result"
+    nixos-generate -f qcow --system x86_64-linux \
+        -c "$_SCRIPT_DIR/linux-install-test-nixos.nix" -o "$link"
+    cp -L "$link"/*.qcow2 "$out"
+    rm -f "$link"
+    echo "   built + cached: $out"
+}
+
+verify_checksum() {
+    local file="$1" sums_url="${SUM_URL[$DISTRO]}" base
+    base="$(basename "${IMAGE_URL[$DISTRO]}")"
+    if [ "${BB_TEST_SKIP_VERIFY:-0}" = "1" ] || [ -z "$sums_url" ]; then
+        echo "   checksum: SKIPPED (${sums_url:+set BB_TEST_SKIP_VERIFY=0 to enable}${sums_url:-no sums URL for $DISTRO})" >&2
+        return 0
+    fi
+    local want
+    # Vendors publish either "HASH  filename" tables or a bare "HASH" (or a
+    # "SHA256 (file) = HASH" BSD line, e.g. Fedora). Cover all three.
+    local sums; sums="$(curl -fsSL "$sums_url")"
+    want="$(printf '%s\n' "$sums" | awk -v f="$base" '
+        $0 ~ f && $1 ~ /^[0-9a-fA-F]{64}$/ { print $1; exit }        # GNU "hash  file"
+        $1=="SHA256" && $0 ~ f { gsub(/[()]/,""); print $NF; exit }  # BSD "SHA256 (file) = hash"
+    ')"
+    [ -z "$want" ] && want="$(printf '%s\n' "$sums" | awk '/^[0-9a-fA-F]{64}$/ {print $1; exit}')"
+    [ -n "$want" ] || { echo "error: could not find a sha256 for $base in $sums_url" >&2; return 1; }
+    local got; got="$(sha256sum "$file" | awk '{print $1}')"
+    if [ "$want" != "$got" ]; then
+        echo "error: checksum mismatch for $base" >&2
+        echo "  want $want" >&2
+        echo "  got  $got" >&2
+        return 1
+    fi
+    echo "   checksum: OK"
+}
+
+ensure_base_image() {
+    mkdir -p "$CACHE_DIR"
+    if [ "$DISTRO" = nixos ]; then
+        build_nixos_image
+        return
+    fi
+    local base; base="$(_base_image)"
+    if [ -f "$base" ]; then
+        echo "== base image cached: $base =="
+        return 0
+    fi
+    echo "== downloading $DISTRO cloud image =="
+    echo "   ${IMAGE_URL[$DISTRO]}"
+    # Download to a temp name and rename on success so an interrupted pull
+    # never poisons the cache with a truncated image.
+    local tmp="$base.partial"
+    curl -fSL --retry 3 -o "$tmp" "${IMAGE_URL[$DISTRO]}"
+    verify_checksum "$tmp"
+    mv "$tmp" "$base"
+    echo "   cached: $base"
+}
+
+# --- cloud-init seed -------------------------------------------------
+make_seed() {
+    ssh-keygen -t ed25519 -N '' -f "$(_ssh_key)" -q
+    local pub; pub="$(cat "$(_ssh_key).pub")"
+    local user="${SSH_USER[$DISTRO]}"
+    local user_data="$RUN_DIR/user-data"
+
+    if [ "$user" = "root" ]; then
+        # NixOS' cloud-init lands the key straight on root; no sudo needed.
+        cat > "$user_data" <<EOF
+#cloud-config
+ssh_authorized_keys:
+  - $pub
+EOF
+    elif [ "$DISTRO" = alpine ]; then
+        # Alpine needs three things the systemd distros don't: cloud-init locks
+        # the account (lock_passwd), but Alpine's non-PAM sshd then refuses
+        # pubkey auth for a locked account — so give it a throwaway password;
+        # and OpenRC does not auto-start sshd after the key is injected, so
+        # start it via runcmd.
+        cat > "$user_data" <<EOF
+#cloud-config
+users:
+  - name: $user
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    shell: /bin/sh
+    lock_passwd: false
+    ssh_authorized_keys:
+      - $pub
+chpasswd:
+  expire: false
+  list: |
+    $user:bbtest
+packages:
+  - sudo
+runcmd:
+  - [ sh, -c, "rc-service sshd start 2>/dev/null || true" ]
+EOF
+    else
+        cat > "$user_data" <<EOF
+#cloud-config
+users:
+  - name: $user
+    sudo: ALL=(ALL) NOPASSWD:ALL
+    shell: /bin/sh
+    lock_passwd: true
+    ssh_authorized_keys:
+      - $pub
+EOF
+    fi
+    # NoCloud wants a meta-data with an instance-id, or cloud-init may not treat
+    # the seed as a new instance (the Alpine nocloud image is strict about this).
+    printf 'instance-id: bbtest-%s\nlocal-hostname: bbtest-%s\n' "$DISTRO" "$DISTRO" \
+        > "$RUN_DIR/meta-data"
+    cloud-localds "$(_seed)" "$user_data" "$RUN_DIR/meta-data"
+}
+
+# --- commands --------------------------------------------------------
+cmd_up() {
+    require_linux; require_kvm; require_tools; known_distro
+    _ensure_run_dir
+    ensure_base_image
+
+    # Throwaway copy-on-write overlay: the cached base is read-only backing,
+    # all guest writes land in the overlay, and `down` deletes it. Resize so
+    # pipx + a git build have headroom (cloud-init grows the rootfs to fit).
+    qemu-img create -q -f qcow2 -F qcow2 -b "$(_base_image)" "$(_overlay)" "$DISK"
+    make_seed
+
+    echo "== booting $DISTRO VM (mem=${MEM_MB}M cpus=$CPUS, ssh -> :$SSH_PORT) =="
+    qemu-system-x86_64 \
+        -machine accel=kvm -cpu host -smp "$CPUS" -m "$MEM_MB" \
+        -display none -daemonize \
+        -pidfile "$(_pidfile)" \
+        -serial "file:$(_serial)" \
+        -drive "file=$(_overlay),if=virtio,format=qcow2" \
+        -drive "file=$(_seed),if=virtio,format=raw" \
+        -netdev "user,id=n0,hostfwd=tcp:127.0.0.1:$SSH_PORT-:22" \
+        -device virtio-net-pci,netdev=n0
+
+    # Only publish the marker (so a later prereqs/run/down finds this VM) when
+    # we aren't inside a test cycle, which manages its own RUN_DIR + teardown.
+    [ "$IN_TEST" = 1 ] || echo "$RUN_DIR" > "$(_marker)"
+
+    wait_for_ssh
+    if [ "$IN_TEST" != 1 ]; then
+        echo "== VM is up. Prepare it with: BB_TEST_DISTRO=$DISTRO $0 prereqs (or go straight to 'run') =="
+    fi
+}
+
+# Install install.sh's toolchain prerequisites (python3 + git + pipx) into the
+# running VM. This is what separates `test-ready` from `test`, and it is a
+# distinct sub-command so a manual up/prereqs/run/down cycle is possible.
+cmd_prereqs() {
+    require_linux
+    _load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
+    echo "== installing prerequisites (python3 + git + pipx) on $DISTRO =="
+    # Runs via the guest login shell; PREREQ is a client-side table value.
+    guest "${PREREQ[$DISTRO]}"
+}
+
+cmd_run() {
+    require_linux
+    _load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
+
+    local spec_env=""
+    [ -n "${BOT_BOTTLE_INSTALL_SPEC:-}" ] \
+        && spec_env="BOT_BOTTLE_INSTALL_SPEC='$BOT_BOTTLE_INSTALL_SPEC' "
+
+    echo "== installing bot-bottle as ${SSH_USER[$DISTRO]} =="
+    # Capture install.sh's exit code and full output rather than aborting on
+    # non-zero: on a bare host a clean prerequisite *decline* is sound, so the
+    # verdict step — not set -e — decides the outcome.
+    local rc
+    set +e
+    if [ -n "${BB_TEST_INSTALL_URL:-}" ]; then
+        guest "curl -fsSL '$BB_TEST_INSTALL_URL' | ${spec_env}sh" 2>&1 | tee "$RUN_DIR/install.log"
+    else
+        # Feed THIS checkout's install.sh in over stdin — the same `curl … | sh`
+        # shape a real user runs, and nothing is staged in the guest to leak.
+        guest "${spec_env}sh -s" < "$_REPO_ROOT/install.sh" 2>&1 | tee "$RUN_DIR/install.log"
+    fi
+    rc="${PIPESTATUS[0]}"
+    set -e
+    printf '%s\n' "$rc" > "$RUN_DIR/install.rc"
+    echo "== install.sh exited $rc =="
+    [ "$IN_TEST" = 1 ] \
+        || echo "== verdict anytime with: BB_TEST_DISTRO=$DISTRO $0 status =="
+}
+
+# Quietly report whether a runnable bot-bottle entry point exists for the
+# guest user, checking the pipx/pip locations install.sh may leave off PATH.
+entry_point_runnable() {
+    # shellcheck disable=SC2016  # expand in the GUEST shell.
+    guest '
+        for bb in "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle" "$(command -v bot-bottle 2>/dev/null)"; do
+            [ -n "$bb" ] && [ -x "$bb" ] || continue
+            # --help exits 0 before any DB/migration/network work; it is the
+            # cheapest proof the package imports and the shim runs. (bot-bottle
+            # has no --version: an unknown arg would die non-zero.)
+            "$bb" --help >/dev/null 2>&1 && exit 0
+        done
+        exit 1
+    ' >/dev/null 2>&1
+}
+
+# `bot-bottle doctor` in the guest, classified. doctor's own exit code
+# conflates "is the install sound" with "is a backend ready to run a bottle" —
+# and inside a plain VM no backend can be ready (no nested KVM/Docker), so the
+# raw exit code is non-zero by design. This separates the two: an unhandled
+# traceback, or a missing python/config line, is an install defect and fails;
+# a not-ready backend is reported, not fatal (unless BB_TEST_REQUIRE_BACKEND=1,
+# for a nested-virt host that can actually satisfy it).
+doctor_in_guest() {
+    local out rc=0 bad=0
+    out="$(mktemp "${TMPDIR:-/tmp}/bb-doctor.XXXXXX")"
+    # shellcheck disable=SC2016  # $HOME/$bb must expand in the GUEST shell.
+    guest '
+        for bb in bot-bottle "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle"; do
+            if command -v "$bb" >/dev/null 2>&1; then
+                case "$bb" in
+                    bot-bottle) : ;;
+                    *) echo "  (not on PATH — running $bb directly, as install.sh advises)" ;;
+                esac
+                exec "$bb" doctor
+            fi
+        done
+        echo "  no bot-bottle entry point found for this user" >&2
+        exit 1
+    ' >"$out" 2>&1 || rc=$?
+    cat "$out"
+
+    # An unhandled exception is always an install/product defect, never an
+    # environment fact — doctor's non-zero exit alone would not distinguish it.
+    if grep -q 'Traceback (most recent call last)' "$out"; then
+        echo "  doctor crashed (traceback above) — a defect, not a missing prerequisite" >&2
+        bad=1
+    fi
+    grep -qE '^ok: +python:' "$out" \
+        || { echo "  doctor never reported a usable python" >&2; bad=1; }
+    grep -qE '^ok: +config:' "$out" \
+        || { echo "  doctor never reported a usable config dir" >&2; bad=1; }
+    if [ "$rc" -ne 0 ] && ! grep -qE '^(fail|warn): +backend' "$out"; then
+        echo "  doctor failed for something other than backend readiness" >&2
+        bad=1
+    fi
+
+    local backend_ready=1
+    grep -qE '^fail: +backend' "$out" && backend_ready=0
+    rm -f "$out"
+
+    [ "$bad" -eq 0 ] || return 1
+    if [ "${BB_TEST_REQUIRE_BACKEND:-0}" = "1" ] && [ "$backend_ready" -eq 0 ]; then
+        echo "  backend is not ready and BB_TEST_REQUIRE_BACKEND=1 — failing" >&2
+        return 1
+    fi
+    if [ "$backend_ready" -eq 1 ]; then
+        echo "  doctor: install sound; a backend is ready"
+    else
+        echo "  doctor: install sound; no backend ready (expected in a plain VM — install gate only)"
+    fi
+    return 0
+}
+
+# The prerequisite-decline messages install.sh prints via die(). On a bare host
+# ANY of these means install.sh correctly refused rather than half-installing —
+# a sound outcome for `test`.
+PREREQ_ERR_RE='is required but was not found|or newer is required|git is required to install from|neither pipx nor a usable|externally managed \(PEP 668\)|is not on PATH'
+
+# The verdict: is the install sound? Reads install.sh's captured exit code and
+# output (from cmd_run) plus the guest's resulting state.
+#   - installed & runnable  -> the verdict is doctor's soundness classification.
+#   - no entry point, but a recognized prerequisite decline, and declines are
+#     allowed (bare `test`, _REQUIRE_INSTALL=0) -> sound.
+#   - anything else -> not sound.
+install_verdict() {
+    local rc=""
+    [ -f "$RUN_DIR/install.rc" ] && rc="$(cat "$RUN_DIR/install.rc")"
+
+    if [ "${rc:-1}" = 0 ] && entry_point_runnable; then
+        echo "doctor (in guest):"
+        doctor_in_guest
+        return $?
+    fi
+
+    if [ "${_REQUIRE_INSTALL:-0}" != "1" ] \
+            && [ -n "$rc" ] && [ "$rc" != 0 ] \
+            && [ -f "$RUN_DIR/install.log" ] \
+            && grep -Eiq "$PREREQ_ERR_RE" "$RUN_DIR/install.log"; then
+        echo "  install.sh declined with an actionable prerequisite error (rc=$rc)"
+        echo "  — sound on a bare host; run 'test-ready' (or 'prereqs') to install."
+        return 0
+    fi
+
+    if [ "${_REQUIRE_INSTALL:-0}" = "1" ]; then
+        echo "  prerequisites were provisioned, but install.sh left no runnable entry point (rc=${rc:-?})" >&2
+    else
+        echo "  install.sh neither installed nor gave a recognized prerequisite error (rc=${rc:-?})" >&2
+    fi
+    return 1
+}
+
+cmd_status() {
+    require_linux
+    if ! _load_run_dir; then
+        echo "vm:      no running VM for $DISTRO"
+        return 0
+    fi
+    if [ -f "$(_pidfile)" ] && kill -0 "$(cat "$(_pidfile)")" 2>/dev/null; then
+        echo "vm:      $DISTRO running (pid $(cat "$(_pidfile)"), ssh :$SSH_PORT)"
+    else
+        echo "vm:      $DISTRO run-dir present but QEMU not alive"
+        return 1
+    fi
+    if install_verdict; then
+        echo "OK[$DISTRO]: the install is sound"
+        return 0
+    fi
+    echo "FAIL[$DISTRO]: the install is not sound (see above)" >&2
+    return 1
+}
+
+cmd_down() {
+    require_linux
+    if ! _load_run_dir; then
+        echo "$DISTRO: nothing running"
+        return 0
+    fi
+    if [ -f "$(_pidfile)" ]; then
+        local pid; pid="$(cat "$(_pidfile)")"
+        if kill -0 "$pid" 2>/dev/null; then
+            kill "$pid" 2>/dev/null || true
+            for _ in 1 2 3 4 5; do kill -0 "$pid" 2>/dev/null || break; sleep 1; done
+            kill -9 "$pid" 2>/dev/null || true
+        fi
+    fi
+    # Deleting the overlay + seed is the reset; the read-only base stays cached.
+    rm -f "$(_overlay)" "$(_seed)" "$(_ssh_key)" "$(_ssh_key).pub" \
+          "$RUN_DIR/user-data" "$RUN_DIR/meta-data" \
+          "$RUN_DIR/install.rc" "$RUN_DIR/install.log" "$(_serial)"
+    # Only remove a scratch dir we created (leave a user-provided one alone).
+    [ -n "${BB_TEST_RUN_DIR:-}" ] || rmdir "$RUN_DIR" 2>/dev/null || true
+    rm -f "$(_marker)"
+    echo "removed $DISTRO VM and its overlay — install surface is clean."
+}
+
+cmd_ssh() {
+    require_linux
+    _load_run_dir || { echo "error: no running VM for $DISTRO" >&2; return 1; }
+    local -a opts
+    mapfile -d '' -t opts < <(_ssh_opts)
+    exec ssh -t "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1"
+}
+
+# Teardown half of the test cycles, armed the moment the VM exists so a failure
+# or a Ctrl-C still leaves nothing running.
+_test_teardown() {
+    local rc=$?
+    trap - EXIT INT TERM
+    if [ "${BB_TEST_KEEP:-0}" = "1" ]; then
+        echo
+        echo "== [$_STEPS/$_STEPS] down: SKIPPED (BB_TEST_KEEP=1) =="
+        echo "   VM still up; remove with: BB_TEST_DISTRO=$DISTRO $0 down"
+        echo "   ssh in with: BB_TEST_RUN_DIR=$RUN_DIR BB_TEST_DISTRO=$DISTRO $0 ssh"
+        exit "$rc"
+    fi
+    echo
+    echo "== [$_STEPS/$_STEPS] down =="
+    cmd_down || rc=1
+    if [ "$rc" -eq 0 ]; then
+        echo
+        echo "PASS[$DISTRO]: $_PASS_CLAIM"
+    else
+        echo
+        echo "FAIL[$DISTRO]: see above (the VM was torn down regardless)." >&2
+    fi
+    exit "$rc"
+}
+
+# The two variants differ only in whether the prerequisites get installed
+# before install.sh runs, which is exactly the question each one asks:
+#
+#   test        a bare host — assert install.sh is SOUND (installs cleanly, or
+#               declines with an actionable prerequisite error).
+#   test-ready  prerequisites satisfied — assert install.sh actually LANDS.
+_test_cycle() {
+    local with_prereqs="$1"
+    require_linux; require_kvm; require_tools; known_distro
+    IN_TEST=1
+    RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")"
+
+    # Arm teardown BEFORE cmd_up: its wait_for_ssh can fail after QEMU is
+    # already running (e.g. a guest that never opens SSH), and without the trap
+    # in place that would leak the VM.
+    trap _test_teardown EXIT INT TERM
+    echo "== [1/$_STEPS] up ($DISTRO) =="
+    cmd_up
+
+    local step=2
+    if [ "$with_prereqs" = 1 ]; then
+        echo
+        echo "== [$step/$_STEPS] prereqs =="
+        cmd_prereqs || { echo "error: could not install prerequisites (see above)." >&2; return 1; }
+        step=$(( step + 1 ))
+    fi
+
+    echo
+    echo "== [$step/$_STEPS] run =="
+    cmd_run
+    step=$(( step + 1 ))
+
+    echo
+    echo "== [$step/$_STEPS] verdict =="
+    # install.sh exits 0 even when doctor reports unmet prerequisites, so the
+    # install succeeding is not the verdict — this is.
+    cmd_status || {
+        echo "error: the install is not sound for $DISTRO (see above)." >&2
+        echo "  re-run with BB_TEST_KEEP=1 to keep the VM and dig in." >&2
+        return 1
+    }
+}
+
+cmd_test() {
+    _STEPS=4
+    _PASS_CLAIM="on a bare $DISTRO host, install.sh behaves soundly."
+    _test_cycle 0
+}
+
+cmd_test_ready() {
+    _STEPS=5
+    _PASS_CLAIM="a $DISTRO host with prerequisites satisfied installs bot-bottle cleanly."
+    # Prerequisites are provisioned, so a graceful decline is no longer an
+    # acceptable outcome — the install must actually land.
+    _REQUIRE_INSTALL=1
+    _test_cycle 1
+}
+
+cmd_test_all() {
+    require_linux; require_kvm; require_tools
+    local -a passed=() failed=()
+    local port="$SSH_PORT"
+    local script
+    script="$_SCRIPT_DIR/$(basename "${BASH_SOURCE[0]}")"
+    # Every distro × both variants. Each cell gets its own forwarded port so a
+    # leftover from a prior cell can't collide, and its own subshell so one
+    # cell's failure (or teardown trap) doesn't abort the matrix.
+    for sub in test test-ready; do
+        for d in "${ALL_DISTROS[@]}"; do
+            echo
+            echo "########################################################"
+            echo "# $d  ($sub)"
+            echo "########################################################"
+            if ( DISTRO="$d" SSH_PORT="$port" \
+                    BB_TEST_DISTRO="$d" BB_TEST_SSH_PORT="$port" \
+                    bash "$script" "$sub" ); then
+                passed+=("$d/$sub")
+            else
+                failed+=("$d/$sub")
+            fi
+            port=$(( port + 1 ))
+        done
+    done
+    echo
+    echo "== matrix summary =="
+    echo "  PASS: ${passed[*]:-(none)}"
+    echo "  FAIL: ${failed[*]:-(none)}"
+    [ "${#failed[@]}" -eq 0 ]
+}
+
+case "${1:-}" in
+    test)       cmd_test ;;
+    test-ready) cmd_test_ready ;;
+    test-all)   cmd_test_all ;;
+    up)         cmd_up ;;
+    prereqs)    cmd_prereqs ;;
+    run)        cmd_run ;;
+    status)     cmd_status ;;
+    down)       cmd_down ;;
+    ssh)        cmd_ssh ;;
+    *) echo "usage: $0 {test|test-ready|test-all|up|prereqs|run|status|down|ssh}  (distro via BB_TEST_DISTRO)" >&2; exit 2 ;;
+esac
@@ -21,7 +21,7 @@ import subprocess
 import time
 import unittest

-from bot_bottle.backend.docker.infra_launch import (
+from bot_bottle.backend.docker.consolidated_launch import (
    _network_cidr,
    _network_container_ips,
 )
@@ -83,13 +83,8 @@ class TestRuntimeModuleSizes(unittest.TestCase):
        self.assertEqual([], violations)

    def test_backend_contract_does_not_absorb_preparation_logic(self) -> None:
-        # base.py holds the backend *contract*; implementation lives elsewhere.
-        # The cap is a tripwire against absorbing preparation/impl logic, not a
-        # ban on new contract surface — bumped 580->615 for the PRD 0081
-        # gateway-attach contract (delegator + 3 abstract primitives; the flow
-        # itself lives in backend/gateway_attach.py, not here).
        caps = {
-            ROOT / "bot_bottle" / "backend" / "base.py": 615,
+            ROOT / "bot_bottle" / "backend" / "base.py": 580,
            ROOT / "bot_bottle" / "backend" / "preparation.py": 160,
        }
        oversized = [
@@ -11,11 +11,9 @@ from types import SimpleNamespace
 from unittest.mock import Mock, patch

 from bot_bottle.orchestrator.reprovision import reprovision_bottles
-from bot_bottle.backend.base import BottleBackend, InfraLaunchError
-from bot_bottle.backend.gateway_attach import GatewayAttachResources
-from bot_bottle.backend.firecracker import infra_launch as fc
-from bot_bottle.backend.macos_container import infra_launch as mac
-from bot_bottle.backend.docker import infra_launch as docker
+from bot_bottle.backend.firecracker import consolidated_launch as fc
+from bot_bottle.backend.macos_container import consolidated_launch as mac
+from bot_bottle.backend.docker import consolidated_launch as docker
 from bot_bottle.orchestrator.client import OrchestratorClientError


@@ -134,7 +132,7 @@ class TestFirecrackerReprovision(unittest.TestCase):
    def test_persist_failure_is_fatal_to_launch(self) -> None:
        with patch.object(fc.util, "ssh_base_argv", return_value=["ssh", "guest"]), \
             patch.object(fc.subprocess, "run", return_value=_proc(1, stderr="denied")):
-            with self.assertRaisesRegex(fc.InfraLaunchError, "denied"):
+            with self.assertRaisesRegex(fc.ConsolidatedLaunchError, "denied"):
                fc.persist_env_var_secret(Path("/key"), "10.0.0.1", "secret")

    def test_reads_live_vm_key_and_reprovisions(self) -> None:
@@ -161,189 +159,5 @@ class TestFirecrackerReprovision(unittest.TestCase):
        restore.assert_called_once_with(client, {})


-class TestAttachFlow(unittest.TestCase):
-    """PRD 0081 / #519 review: the base backend owns the reconcile flow and
-    fails hard — gather resources, attempt every running bottle, then raise an
-    aggregate if any failed (never silently skip)."""
-
-    def _fake(self, **primitives: object) -> SimpleNamespace:
-        base: dict[str, object] = {
-            "_gateway_attach_resources": Mock(
-                return_value=GatewayAttachResources(ca_pem="PEM")),
-            "_running_bottles": Mock(return_value=["a", "b", "c"]),
-            "_attach_bottle_to_gateway": Mock(),
-        }
-        base.update(primitives)
-        return SimpleNamespace(**base)
-
-    def test_attaches_every_running_bottle_once(self) -> None:
-        fake = self._fake()
-        BottleBackend.attach_bottled_agents_to_gateway(fake)  # type: ignore[arg-type]
-        self.assertEqual(3, fake._attach_bottle_to_gateway.call_count)
-        # Each bottle got the single gathered resource set.
-        res = fake._gateway_attach_resources.return_value
-        for call in fake._attach_bottle_to_gateway.call_args_list:
-            self.assertIs(res, call.args[1])
-
-    def test_attempts_all_then_raises_aggregate_on_failure(self) -> None:
-        def attach(bottle: str, _resources: object) -> None:
-            if bottle in ("b", "c"):
-                raise InfraLaunchError(f"{bottle} unreachable")
-        fake = self._fake(_attach_bottle_to_gateway=Mock(side_effect=attach))
-        with self.assertRaises(InfraLaunchError) as ctx:
-            BottleBackend.attach_bottled_agents_to_gateway(fake)  # type: ignore[arg-type]
-        msg = str(ctx.exception)
-        # Every bottle was attempted (not fail-fast), and both failures surface.
-        self.assertEqual(3, fake._attach_bottle_to_gateway.call_count)
-        self.assertIn("b unreachable", msg)
-        self.assertIn("c unreachable", msg)
-        self.assertIn("2", msg)
-
-    def test_resource_gathering_failure_fails_hard_before_enumerating(self) -> None:
-        fake = self._fake(
-            _gateway_attach_resources=Mock(side_effect=RuntimeError("no CA yet")))
-        with self.assertRaises(RuntimeError):
-            BottleBackend.attach_bottled_agents_to_gateway(fake)  # type: ignore[arg-type]
-        fake._running_bottles.assert_not_called()
-
-
-class TestFirecrackerAttachPrimitive(unittest.TestCase):
-    """PRD 0081: attach_ca_to_agent_vm pushes the current gateway CA into one
-    running agent VM over SSH, failing hard."""
-
-    def _run_dir(self, root: Path, ip: str = "10.243.0.3") -> Path:
-        run_dir = root / "demo"
-        run_dir.mkdir()
-        (run_dir / "bottle_id_ed25519").write_text("key")
-        (run_dir / "config.json").write_text(json.dumps({
-            "boot-source": {"boot_args": f"root=/dev/vda ip={ip}::gw:mask::eth0:off"}
-        }))
-        return run_dir
-
-    def test_pushes_ca_over_stdin_and_rebuilds_trust_store(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            run_dir = self._run_dir(Path(tmp))
-            with patch.object(fc.util, "ssh_base_argv", return_value=["ssh", "guest"]), \
-                 patch.object(fc.subprocess, "run", return_value=_proc()) as run:
-                fc.attach_ca_to_agent_vm(run_dir, "PEM-DATA")
-        # The cert goes over stdin (off argv), and the trust store is rebuilt.
-        self.assertEqual("PEM-DATA", run.call_args.kwargs["input"])
-        script = run.call_args.args[0][-1]
-        self.assertIn(fc.AGENT_CA_PATH, script)
-        self.assertIn("update-ca-certificates", script)
-
-    def test_malformed_run_dir_fails_hard(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            empty = Path(tmp) / "no-config"
-            empty.mkdir()
-            with self.assertRaises(fc.InfraLaunchError):
-                fc.attach_ca_to_agent_vm(empty, "PEM")
-
-    def test_push_failure_fails_hard(self) -> None:
-        with tempfile.TemporaryDirectory() as tmp:
-            run_dir = self._run_dir(Path(tmp))
-            with patch.object(fc.util, "ssh_base_argv", return_value=["ssh", "guest"]), \
-                 patch.object(fc.subprocess, "run", return_value=_proc(1, stderr="down")):
-                with self.assertRaisesRegex(fc.InfraLaunchError, "down"):
-                    fc.attach_ca_to_agent_vm(run_dir, "PEM")
-
-
-class TestDockerAttachPrimitives(unittest.TestCase):
-    def test_running_agent_containers_excludes_gateway(self) -> None:
-        # network inspect lists the gateway (excluded) + one agent + a blank line.
-        inspect = _proc(stdout="bot-bottle-orch-gateway\nbot-bottle-a\n\n")
-        with patch.object(docker, "run_docker", return_value=inspect):
-            names = docker.running_agent_containers(
-                network="net", gateway_name="bot-bottle-orch-gateway")
-        self.assertEqual(["bot-bottle-a"], names)
-
-    def test_running_agent_containers_fails_hard_on_docker_error(self) -> None:
-        with patch.object(docker, "run_docker", side_effect=FileNotFoundError("docker")):
-            with self.assertRaises(FileNotFoundError):
-                docker.running_agent_containers()
-
-    def test_running_agent_containers_fails_hard_on_inspect_failure(self) -> None:
-        inspect = _proc(1, stderr="network unavailable")
-        with patch.object(docker, "run_docker", return_value=inspect):
-            with self.assertRaisesRegex(
-                docker.InfraLaunchError, "network unavailable",
-            ):
-                docker.running_agent_containers(network="net")
-
-    def test_push_ca_cp_then_rebuilds_trust_store(self) -> None:
-        with patch.object(
-            docker, "run_docker", side_effect=[_proc(), _proc(), _proc()],
-        ) as run:
-            docker.push_ca_to_container("bot-bottle-a", "PEM")
-        argvs = [c.args[0] for c in run.call_args_list]
-        cp = next(a for a in argvs if a[:2] == ["docker", "cp"])
-        self.assertEqual(f"bot-bottle-a:{docker.AGENT_CA_PATH}", cp[-1])
-        self.assertTrue(any("update-ca-certificates" in a[-1] for a in argvs))
-
-    def test_push_ca_fails_hard(self) -> None:
-        with patch.object(docker, "run_docker", return_value=_proc(1, stderr="gone")):
-            with self.assertRaisesRegex(docker.InfraLaunchError, "gone"):
-                docker.push_ca_to_container("bot-bottle-a", "PEM")
-
-    def test_gateway_attach_resources_reads_the_threaded_infra_gateway(self) -> None:
-        # On the bring-up reconcile path the backend is threaded with the infra
-        # whose gateway just cold-booted, so it reads THAT gateway's CA — not a
-        # fresh default-named DockerInfraService that wouldn't exist for a
-        # non-default instance (#519 review).
-        from bot_bottle.backend.docker.backend import DockerBottleBackend
-        infra = Mock()
-        infra.gateway.return_value.ca_cert_pem.return_value = "ITEST-CA"
-        res = DockerBottleBackend(infra=infra)._gateway_attach_resources()
-        self.assertEqual("ITEST-CA", res.ca_pem)
-        infra.gateway.assert_called_once()
-
-    def test_running_bottles_uses_the_threaded_infra_network(self) -> None:
-        from bot_bottle.backend.docker.backend import DockerBottleBackend
-        infra = Mock(network="itest-network")
-        infra.gateway.return_value.name = "itest-gateway"
-        with patch.object(docker, "running_agent_containers") as running:
-            DockerBottleBackend(infra=infra)._running_bottles()
-        running.assert_called_once_with(
-            network="itest-network", gateway_name="itest-gateway",
-        )
-
-    def test_gateway_attach_resources_defaults_to_the_host_singleton(self) -> None:
-        # Ordinary construction (no infra) falls back to the per-host default.
-        from bot_bottle.backend.docker.backend import DockerBottleBackend
-        with patch("bot_bottle.backend.docker.infra.DockerInfraService") as InfraCls:
-            InfraCls.return_value.gateway.return_value.ca_cert_pem.return_value = "DEFAULT-CA"
-            res = DockerBottleBackend()._gateway_attach_resources()
-        InfraCls.assert_called_once_with()
-        self.assertEqual("DEFAULT-CA", res.ca_pem)
-
-
-class TestMacosAttachPrimitives(unittest.TestCase):
-    def test_running_agent_containers_from_enumeration(self) -> None:
-        with patch.object(mac, "enumerate_active",
-                          return_value=[SimpleNamespace(slug="demo")]):
-            names = mac.running_agent_containers()
-        self.assertEqual([f"{mac.CONTAINER_NAME_PREFIX}demo"], names)
-
-    def test_running_agent_containers_fails_hard(self) -> None:
-        with patch.object(mac, "enumerate_active",
-                          side_effect=mac.EnumerationError("failed")):
-            with self.assertRaises(mac.EnumerationError):
-                mac.running_agent_containers()
-
-    def test_push_ca_cp_then_rebuilds_trust_store(self) -> None:
-        with patch.object(mac.container_mod, "run_container_argv",
-                          return_value=_proc()) as run:
-            mac.push_ca_to_container(f"{mac.CONTAINER_NAME_PREFIX}demo", "PEM")
-        argvs = [c.args[0] for c in run.call_args_list]
-        self.assertTrue(any(mac.AGENT_CA_PATH in " ".join(a) for a in argvs))
-        self.assertTrue(any("update-ca-certificates" in a[-1] for a in argvs))
-
-    def test_push_ca_fails_hard(self) -> None:
-        with patch.object(mac.container_mod, "run_container_argv",
-                          return_value=_proc(1, stderr="no")):
-            with self.assertRaisesRegex(mac.InfraLaunchError, "no"):
-                mac.push_ca_to_container("x", "PEM")
-
-
 if __name__ == "__main__":
    unittest.main()
@@ -6,8 +6,8 @@ import unittest
 from pathlib import Path
 from unittest.mock import MagicMock, Mock, patch

-from bot_bottle.backend.docker.infra_launch import (
-    InfraLaunchError,
+from bot_bottle.backend.docker.consolidated_launch import (
+    ConsolidatedLaunchError,
    _network_container_ips,
    launch_consolidated,
    deprovision_consolidated,
@@ -16,7 +16,7 @@ from bot_bottle.egress import EgressPlan, EgressRoute
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.orchestrator.client import RegisteredBottle

-_MOD = "bot_bottle.backend.docker.infra_launch"
+_MOD = "bot_bottle.backend.docker.consolidated_launch"
 _UTIL = "bot_bottle.backend.provision_bottle"


@@ -93,7 +93,7 @@ class TestNetworkContainerIps(unittest.TestCase):
        result = Mock(returncode=1, stdout="", stderr="daemon unavailable")
        with (
            patch(f"{_MOD}.run_docker", return_value=result),
-            self.assertRaisesRegex(InfraLaunchError, "daemon unavailable"),
+            self.assertRaisesRegex(ConsolidatedLaunchError, "daemon unavailable"),
        ):
            _network_container_ips("bot-bottle-gateway")

@@ -29,20 +29,10 @@ class TestDockerInfraService(unittest.TestCase):
        self.orch.gateway_url.return_value = f"http://{ORCHESTRATOR_NAME}:8099"
        self.orch.mint_gateway_token.return_value = "gw.jwt"
        self.gw = MagicMock()
-        # Default to a cold boot (gateway (re)created) so the reconcile branch
-        # runs; the actual reconcile is stubbed so these composition tests stay
-        # isolated from docker.
-        self.gw.connect_to_orchestrator.return_value = True
        for name, mock in (("orchestrator", self.orch), ("gateway", self.gw)):
            p = patch.object(self.svc, name, return_value=mock)
            p.start()
            self.addCleanup(p.stop)
-        ap = patch(
-            "bot_bottle.backend.docker.backend.DockerBottleBackend"
-            ".attach_bottled_agents_to_gateway"
-        )
-        self.attach = ap.start()
-        self.addCleanup(ap.stop)

    def test_url_delegates_to_the_orchestrator(self) -> None:
        self.assertEqual("http://127.0.0.1:8099", self.svc.url())
@@ -68,20 +58,6 @@ class TestDockerInfraService(unittest.TestCase):
            f"http://{ORCHESTRATOR_NAME}:8099", "gw.jwt")
        self.orch.mint_gateway_token.assert_called_once()

-    def test_cold_boot_reconciles_running_bottles(self) -> None:
-        # A (re)created gateway (connect returns True) triggers the bring-up
-        # reconcile so running bottles re-trust the fresh gateway (PRD 0081).
-        self.gw.connect_to_orchestrator.return_value = True
-        self.svc.ensure_running()
-        self.attach.assert_called_once_with()
-
-    def test_no_reconcile_when_gateway_left_untouched(self) -> None:
-        # A healthy current gateway (connect returns False) is not a cold boot,
-        # so there is nothing to reconcile.
-        self.gw.connect_to_orchestrator.return_value = False
-        self.svc.ensure_running()
-        self.attach.assert_not_called()
-
    def test_stop_removes_both_containers(self) -> None:
        with patch(_RUN) as run:
            self.svc.stop()
@@ -16,7 +16,7 @@ from bot_bottle.agent_provider import AgentProvisionPlan
 from bot_bottle.backend import BottleSpec
 from bot_bottle.backend.docker import launch as launch_mod
 from bot_bottle.backend.docker.bottle_plan import DockerBottlePlan
-from bot_bottle.backend.docker.infra_launch import LaunchContext
+from bot_bottle.backend.docker.consolidated_launch import LaunchContext
 from bot_bottle.egress import EgressPlan
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.log import Die
@@ -19,7 +19,7 @@ from bot_bottle.agent_provider import AgentProvisionPlan
 from bot_bottle.backend import BottleImages, BottleSpec
 from bot_bottle.backend.docker import launch as launch_mod
 from bot_bottle.backend.docker.bottle_plan import DockerBottlePlan
-from bot_bottle.backend.docker.infra_launch import LaunchContext
+from bot_bottle.backend.docker.consolidated_launch import LaunchContext
 from bot_bottle.egress import EgressPlan
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.manifest import ManifestIndex
@@ -61,10 +61,7 @@ class TestFirecrackerGatewayConnect(unittest.TestCase):
        booted = infra_vm.InfraVm(guest_ip="10.243.255.3", private_key=Path("/k"))
        with patch.object(infra_vm, "boot_vm", return_value=booted) as boot, \
                patch.object(infra_vm, "push_secret") as push:
-            cold_booted = gw.connect_to_orchestrator(_ORCH_URL, _TOKEN)
-        # The VM is booted fresh here (cold-boot path only), so it always
-        # signals a cold boot → the caller reconciles running bottles (PRD 0081).
-        self.assertTrue(cold_booted)
+            gw.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        # Booted on the gateway link with the gateway role; the orchestrator's
        # guest IP (parsed off the URL) rides the cmdline as bb_orch.
        kw = boot.call_args.kwargs
@@ -40,21 +40,7 @@ class TestAccessors(unittest.TestCase):
        self.assertEqual(f"http://{ip}:{infra_vm.ORCHESTRATOR_PORT}", svc.url())


-_ATTACH = (
-    "bot_bottle.backend.firecracker.backend.FirecrackerBottleBackend"
-    ".attach_bottled_agents_to_gateway"
-)
-
-
 class TestEnsureRunning(unittest.TestCase):
-    def setUp(self) -> None:
-        # The cold-boot branch reconciles running bottles against the fresh
-        # gateway (PRD 0081). Stub it so these composition tests stay isolated
-        # from SSH; the wiring itself is asserted in test_cold_boot_reconciles*.
-        ap = patch(_ATTACH)
-        self.attach = ap.start()
-        self.addCleanup(ap.stop)
-
    def test_adopts_when_healthy_alive_and_version_matches(self):
        # Healthy control plane + live gateway + existing key + matching version
        # marker -> adopt (no stop/build/boot); returns the orchestrator URL.
@@ -73,8 +59,6 @@ class TestEnsureRunning(unittest.TestCase):
            built.assert_not_called()
            ip = infra_vm.netpool.orch_slot().guest_ip
            self.assertEqual(f"http://{ip}:{infra_vm.ORCHESTRATOR_PORT}", url)
-            # Adopt path — no cold boot, so no running-bottle reconcile.
-            self.attach.assert_not_called()

    def test_reboots_both_when_version_stale(self):
        # Healthy control plane but the running pair booted an OLDER image
@@ -136,8 +120,6 @@ class TestEnsureRunning(unittest.TestCase):
        # reach the control plane at startup).
        orch_cls.return_value.ensure_running.assert_called_once()
        gw_cls.return_value.connect_to_orchestrator.assert_called_once()
-        # A cold boot minted a fresh gateway CA — reconcile running bottles.
-        self.attach.assert_called_once_with()


 if __name__ == "__main__":
@@ -0,0 +1,98 @@
+"""Unit: how the CLI tells users to re-run it under sudo.
+
+`sudo bot-bottle …` is wrong for the users who followed the documented
+install: sudo's secure_path excludes ~/.local/bin, where both pipx and
+install.sh put the entry point. These lock in the absolute-path form.
+"""
+
+from __future__ import annotations
+
+import contextlib
+import io
+import os
+import tempfile
+import unittest
+from pathlib import Path
+from unittest import mock
+
+from bot_bottle import invocation
+
+
+class TestSelfPath(unittest.TestCase):
+    def test_absolute_argv0_is_used_as_is(self):
+        with mock.patch.object(invocation.sys, "argv", ["/opt/venv/bin/bot-bottle"]):
+            self.assertEqual("/opt/venv/bin/bot-bottle", invocation.self_path())
+
+    def test_bare_name_is_resolved_through_path(self):
+        # The case that matters: invoked as `bot-bottle`, installed in a
+        # directory sudo would drop.
+        with tempfile.TemporaryDirectory() as d:
+            entry = Path(d, "bot-bottle")
+            entry.write_text("#!/bin/sh\n")
+            entry.chmod(0o755)
+            with mock.patch.object(invocation.sys, "argv", ["bot-bottle"]), \
+                 mock.patch.dict(os.environ, {"PATH": d}):
+                self.assertEqual(str(entry), invocation.self_path())
+
+    def test_relative_path_is_made_absolute(self):
+        with tempfile.TemporaryDirectory() as d:
+            entry = Path(d, "bot-bottle")
+            entry.write_text("#!/bin/sh\n")
+            entry.chmod(0o755)
+            cwd = os.getcwd()
+            try:
+                os.chdir(d)
+                with mock.patch.object(invocation.sys, "argv", ["./bot-bottle"]):
+                    self.assertTrue(os.path.isabs(invocation.self_path()))
+            finally:
+                os.chdir(cwd)
+
+    def test_unresolvable_entry_point_falls_back_to_the_name(self):
+        # `python -m`-style invocation, or an argv[0] that no longer exists.
+        # A slightly wrong hint beats a traceback raised while reporting some
+        # unrelated problem.
+        with mock.patch.object(invocation.sys, "argv", ["/nonexistent/gone"]), \
+             mock.patch.object(invocation.shutil, "which", return_value=None):
+            self.assertEqual("/nonexistent/gone", invocation.self_path())
+        with mock.patch.object(invocation.sys, "argv", [""]), \
+             mock.patch.object(invocation.shutil, "which", return_value=None):
+            self.assertEqual("bot-bottle", invocation.self_path())
+
+
+class TestSudoCommand(unittest.TestCase):
+    def test_names_an_absolute_path_not_the_bare_command(self):
+        with mock.patch.object(invocation, "self_path",
+                               return_value="/home/u/.local/bin/bot-bottle"):
+            cmd = invocation.sudo_command("backend", "setup", "--backend=firecracker")
+        self.assertEqual(
+            "sudo /home/u/.local/bin/bot-bottle backend setup --backend=firecracker",
+            cmd,
+        )
+        # The regression this exists to prevent.
+        self.assertNotIn("sudo bot-bottle", cmd)
+
+
+class TestFirecrackerSetupUsesIt(unittest.TestCase):
+    def test_root_reinvocation_hint_names_an_absolute_path(self):
+        # The message that prompted all this. Drive the real code path rather
+        # than scanning the source, which would also match the comment
+        # explaining why the bare form is wrong.
+        from bot_bottle.backend.firecracker import setup as fc_setup
+
+        err, out = io.StringIO(), io.StringIO()
+        with mock.patch.object(fc_setup.os, "geteuid", return_value=501), \
+             mock.patch.object(fc_setup.invocation, "self_path",
+                               return_value="/home/u/.local/bin/bot-bottle"), \
+             contextlib.redirect_stderr(err), contextlib.redirect_stdout(out):
+            fc_setup._setup_systemd()
+
+        printed = err.getvalue()
+        self.assertIn(
+            "sudo /home/u/.local/bin/bot-bottle backend setup --backend=firecracker",
+            printed,
+        )
+        self.assertNotIn("sudo bot-bottle", printed)
+
+
+if __name__ == "__main__":
+    unittest.main()
@@ -6,7 +6,7 @@ import unittest
 from pathlib import Path
 from unittest.mock import MagicMock, Mock, patch

-from bot_bottle.backend.macos_container.infra_launch import (
+from bot_bottle.backend.macos_container.consolidated_launch import (
    GatewayEndpoint,
    ensure_gateway,
    register_agent,
@@ -16,7 +16,7 @@ from bot_bottle.egress import EgressPlan, EgressRoute
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.orchestrator.client import RegisteredBottle

-_MOD = "bot_bottle.backend.macos_container.infra_launch"
+_MOD = "bot_bottle.backend.macos_container.consolidated_launch"
 _UTIL = "bot_bottle.backend.provision_bottle"


@@ -145,7 +145,7 @@ class TestLiveSourceIps(unittest.TestCase):
        return [Mock(slug=s) for s in slugs]

    def test_maps_slugs_to_container_addresses(self) -> None:
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        with patch(f"{_MOD}.enumerate_active", return_value=self._agents("a", "b")), \
                patch(f"{_MOD}.container_mod.inspect_container_network_ip",
                      side_effect=["10.0.0.1", "10.0.0.2"]) as ip:
@@ -156,7 +156,7 @@ class TestLiveSourceIps(unittest.TestCase):
    def test_containers_without_an_address_are_skipped(self) -> None:
        """A container that hasn't been given a DHCP address yet contributes
        nothing — the reap's grace window, not this list, protects it."""
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        with patch(f"{_MOD}.enumerate_active", return_value=self._agents("a", "b")), \
                patch(f"{_MOD}.container_mod.inspect_container_network_ip",
                      side_effect=["", "10.0.0.2"]):
@@ -165,7 +165,7 @@ class TestLiveSourceIps(unittest.TestCase):
    def test_container_list_failure_raises(self) -> None:
        """If container list fails, the live set is not authoritative and
        reconciliation must be skipped."""
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        from bot_bottle.backend.macos_container.enumerate import EnumerationError
        with patch(f"{_MOD}.enumerate_active",
                   side_effect=EnumerationError("container list failed")):
@@ -174,7 +174,7 @@ class TestLiveSourceIps(unittest.TestCase):

    def test_per_container_inspect_failure_raises(self) -> None:
        """If any individual inspect fails, the live set is not authoritative."""
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        from bot_bottle.backend.macos_container.enumerate import EnumerationError
        with patch(f"{_MOD}.enumerate_active", return_value=self._agents("a", "b")), \
                patch(f"{_MOD}.container_mod.inspect_container_network_ip",
@@ -16,7 +16,7 @@ from unittest.mock import ANY, patch

 from bot_bottle.backend.macos_container.bottle import MacosContainerBottle
 from bot_bottle.backend.macos_container.bottle_plan import MacosContainerBottlePlan
-from bot_bottle.backend.macos_container.infra_launch import GatewayEndpoint
+from bot_bottle.backend.macos_container.consolidated_launch import GatewayEndpoint
 from bot_bottle.backend.macos_container.gateway_hosts import GATEWAY_HOSTNAME
 from bot_bottle.backend.macos_container.launch import (
    _agent_run_argv,
@@ -44,17 +44,6 @@ class TestMacosGatewayConnect(unittest.TestCase):
    def test_default_name(self) -> None:
        self.assertEqual(GATEWAY_NAME, self.gw.name)

-    def test_connect_signals_cold_boot(self) -> None:
-        # The container is recreated unconditionally, so connect always signals
-        # a cold boot → the caller reconciles running bottles (PRD 0081).
-        run = Mock(return_value=_proc())
-        with patch(f"{_GW}.container_mod") as mod, \
-                patch(f"{_GW}.host_gateway_ca_dir", return_value=Path("/ca")):
-            mod.dns_server.return_value = "1.1.1.1"
-            mod.bind_mount_spec.side_effect = _spec
-            mod.run_container_argv = run
-            self.assertTrue(self.gw.connect_to_orchestrator(_ORCH_URL, _TOKEN))
-
    def test_refuses_without_orchestrator_url(self) -> None:
        with patch(f"{_GW}.container_mod") as mod:
            with self.assertRaises(GatewayError):
@@ -24,20 +24,10 @@ class TestInfraEnsureRunning(unittest.TestCase):
        self.orch.url.return_value = "http://192.168.128.2:8099"
        self.orch.mint_gateway_token.return_value = "gw.jwt"
        self.gw = MagicMock()
-        # Default to a cold boot (gateway (re)created) so the reconcile branch
-        # runs; the actual reconcile is stubbed so these composition tests stay
-        # isolated from the `container` CLI.
-        self.gw.connect_to_orchestrator.return_value = True
        for name, mock in (("orchestrator", self.orch), ("gateway", self.gw)):
            p = patch.object(self.svc, name, return_value=mock)
            p.start()
            self.addCleanup(p.stop)
-        ap = patch(
-            "bot_bottle.backend.macos_container.backend"
-            ".MacosContainerBottleBackend.attach_bottled_agents_to_gateway"
-        )
-        self.attach = ap.start()
-        self.addCleanup(ap.stop)

    def test_composes_networks_builds_and_brings_up_orchestrator_first(self) -> None:
        with patch(f"{_INFRA}.ensure_networks") as nets:
@@ -56,20 +46,6 @@ class TestInfraEnsureRunning(unittest.TestCase):
            "http://192.168.128.2:8099", "gw.jwt")
        self.orch.mint_gateway_token.assert_called_once()

-    def test_cold_boot_reconciles_running_bottles(self) -> None:
-        # A (re)created gateway (connect returns True) triggers the bring-up
-        # reconcile so running bottles re-trust the fresh gateway (PRD 0081).
-        self.gw.connect_to_orchestrator.return_value = True
-        with patch(f"{_INFRA}.ensure_networks"):
-            self.svc.ensure_running()
-        self.attach.assert_called_once_with()
-
-    def test_no_reconcile_when_gateway_left_untouched(self) -> None:
-        self.gw.connect_to_orchestrator.return_value = False
-        with patch(f"{_INFRA}.ensure_networks"):
-            self.svc.ensure_running()
-        self.attach.assert_not_called()
-
    def test_is_healthy_delegates_to_the_orchestrator(self) -> None:
        self.orch.is_healthy.return_value = True
        self.assertTrue(self.svc.is_healthy())
@@ -151,31 +151,6 @@ class TestReprovisionGateway(unittest.TestCase):
                self.c.reprovision_gateway("b1", "key")


-class TestUpdateAgentSecret(unittest.TestCase):
-    def setUp(self) -> None:
-        self.c = OrchestratorClient("http://orch:8080")
-
-    def test_success_posts_single_secret(self) -> None:
-        with patch(_URLOPEN, return_value=_resp(200, {"updated": True})) as opened:
-            self.assertTrue(self.c.update_agent_secret("b1", "A", "fresh", "key"))
-        request = opened.call_args.args[0]
-        self.assertEqual("POST", request.get_method())
-        self.assertTrue(request.full_url.endswith("/bottles/b1/secret"))
-        self.assertEqual(
-            {"name": "A", "value": "fresh", "env_var_secret": "key"},
-            json.loads(request.data),
-        )
-
-    def test_unknown_bottle_is_false(self) -> None:
-        with patch(_URLOPEN, side_effect=_http_error(404)):
-            self.assertFalse(self.c.update_agent_secret("b1", "A", "v", "key"))
-
-    def test_other_status_raises(self) -> None:
-        with patch(_URLOPEN, side_effect=_http_error(400)):
-            with self.assertRaises(OrchestratorClientError):
-                self.c.update_agent_secret("b1", "A", "v", "key")
-
-
 class TestHealthAndPolicy(unittest.TestCase):
    def setUp(self) -> None:
        self.c = OrchestratorClient("http://orch:8080")
@@ -86,12 +86,9 @@ class TestDockerGateway(unittest.TestCase):
            return _proc()

        with patch(_RUN_DOCKER, side_effect=fake):
-            cold_booted = self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
+            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        self.assertEqual([], [c for c in calls if c[:2] == ["docker", "run"]])
        self.assertEqual([], [c for c in calls if c[:2] == ["docker", "rm"]])
-        # A healthy current gateway left untouched is not a cold boot — the
-        # bring-up flow skips the running-bottle reconcile (PRD 0081).
-        self.assertFalse(cold_booted)

    def test_ensure_running_recreates_when_image_is_stale(self) -> None:
        # Running, but the container was built from an OLD image → recreate so
@@ -109,12 +106,9 @@ class TestDockerGateway(unittest.TestCase):
            return _proc()

        with patch(_RUN_DOCKER, side_effect=fake):
-            cold_booted = self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
+            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        self.assertEqual(1, len([c for c in calls if c[:2] == ["docker", "run"]]))
        self.assertTrue(any(c[:2] == ["docker", "rm"] for c in calls))
-        # A recreated container minted/mounted its CA afresh — a cold boot that
-        # triggers the running-bottle reconcile (PRD 0081).
-        self.assertTrue(cold_booted)

    def test_ensure_running_starts_the_singleton_when_absent(self) -> None:
        calls: list[list[str]] = []
@@ -124,8 +118,7 @@ class TestDockerGateway(unittest.TestCase):
            return _proc(stdout="") if argv[:2] == ["docker", "ps"] else _proc()

        with patch(_RUN_DOCKER, side_effect=fake):
-            cold_booted = self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
-        self.assertTrue(cold_booted)  # started fresh → reconcile running bottles
+            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)

        runs = [c for c in calls if c[:2] == ["docker", "run"]]
        self.assertEqual(1, len(runs))
@@ -199,20 +199,6 @@ class TestAgentSecrets(unittest.TestCase):
        got = self.store.get_agent_secrets("bottle-1")
        self.assertEqual({"K": "new", "K2": "v2"}, got)

-    def test_store_agent_secret_upserts_one_key_leaving_others(self) -> None:
-        self.store.store_agent_secrets("bottle-1", {"K": "v1", "K2": "v2"})
-        self.store.store_agent_secret("bottle-1", "K", "v1-new")  # update existing
-        self.store.store_agent_secret("bottle-1", "K3", "v3")     # insert new
-        self.assertEqual(
-            {"K": "v1-new", "K2": "v2", "K3": "v3"},
-            self.store.get_agent_secrets("bottle-1"),
-        )
-
-    def test_store_agent_secret_does_not_duplicate_rows(self) -> None:
-        self.store.store_agent_secret("bottle-1", "K", "v1")
-        self.store.store_agent_secret("bottle-1", "K", "v2")
-        self.assertEqual({"K": "v2"}, self.store.get_agent_secrets("bottle-1"))
-
    def test_delete_removes_secrets(self) -> None:
        self.store.store_agent_secrets("bottle-1", {"K": "v"})
        self.store.delete_agent_secrets("bottle-1")
@@ -125,47 +125,6 @@ class TestDispatch(unittest.TestCase):
            {"EGRESS_TOKEN_0": "upstream-secret"}, self.orch.tokens_for(bottle_id),
        )

-    def test_update_agent_secret_in_place(self) -> None:
-        key = base64.urlsafe_b64encode(b"unit-test-key").rstrip(b"=").decode()
-        status, payload = dispatch(
-            self.orch, "POST", "/bottles", _body({
-                "source_ip": "10.243.0.21",
-                "tokens": {"A": "old", "B": "keep"},
-                "env_var_secret": key,
-            }),
-        )
-        self.assertEqual(201, status)
-        bottle_id = payload["bottle_id"]
-        assert isinstance(bottle_id, str)
-        status, response = dispatch(
-            self.orch, "POST", f"/bottles/{bottle_id}/secret",
-            _body({"name": "A", "value": "fresh", "env_var_secret": key}),
-        )
-        self.assertEqual((200, {"updated": True}), (status, response))
-        self.assertEqual({"A": "fresh", "B": "keep"}, self.orch.tokens_for(bottle_id))
-
-    def test_update_agent_secret_validates_and_unknown_bottle(self) -> None:
-        status, _ = dispatch(self.orch, "POST", "/bottles/b1/secret", b"not-json")
-        self.assertEqual(400, status)
-        status, _ = dispatch(
-            self.orch, "POST", "/bottles/b1/secret", _body({"name": "A"}),
-        )
-        self.assertEqual(400, status)  # value + env_var_secret missing
-        status, _ = dispatch(
-            self.orch, "POST", "/bottles/ghost/secret",
-            _body({"name": "A", "value": "v", "env_var_secret": "k"}),
-        )
-        self.assertEqual(404, status)
-
-    def test_update_agent_secret_is_cli_only(self) -> None:
-        # A data-plane (`gateway`) caller must not set a bottle's tokens.
-        status, _ = dispatch(
-            self.orch, "POST", "/bottles/b1/secret",
-            _body({"name": "A", "value": "v", "env_var_secret": "k"}),
-            role=ROLE_GATEWAY,
-        )
-        self.assertEqual(403, status)
-
    def test_reprovision_validates_request_and_missing_rows(self) -> None:
        status, _ = dispatch(
            self.orch, "POST", "/bottles/b1/reprovision_gateway", b"not-json",
@@ -103,25 +103,6 @@ class TestOrchestrator(unittest.TestCase):
        self.assertTrue(self.orch.reprovision_from_secret(rec.bottle_id, key))
        self.assertEqual({"EGRESS_TOKEN_0": "secret"}, self.orch.tokens_for(rec.bottle_id))

-    def test_update_agent_secret_refreshes_one_token_in_place(self) -> None:
-        key = new_env_var_secret()
-        rec = self.orch.launch_bottle(
-            "10.243.0.20", tokens={"A": "old-a", "B": "keep-b"}, env_var_secret=key,
-        )
-        self.assertTrue(self.orch.update_agent_secret(rec.bottle_id, "A", "new-a", key))
-        # In memory: A refreshed, B left untouched.
-        self.assertEqual({"A": "new-a", "B": "keep-b"}, self.orch.tokens_for(rec.bottle_id))
-        # At rest: the whole set stays decryptable with the SAME env_var_secret,
-        # so a later reprovision restores the refreshed value (not the launch one).
-        self.orch._tokens.clear()
-        self.assertTrue(self.orch.reprovision_from_secret(rec.bottle_id, key))
-        self.assertEqual({"A": "new-a", "B": "keep-b"}, self.orch.tokens_for(rec.bottle_id))
-
-    def test_update_agent_secret_unknown_bottle_is_false(self) -> None:
-        self.assertFalse(
-            self.orch.update_agent_secret("ghost", "A", "v", new_env_var_secret())
-        )
-
    def test_reprovision_rejects_missing_rows_and_wrong_key(self) -> None:
        self.assertFalse(self.orch.reprovision_from_secret("missing", new_env_var_secret()))
        key = "AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE"
Author	SHA1	Message	Date
didericis	9bf2961d13	fix: name an absolute path in the sudo re-run hint test / unit (push) Successful in 57s Details test / image-input-builds (push) Successful in 1m2s Details Update Quality Badges / update-badges (push) Successful in 1m8s Details test / integration-docker (push) Successful in 58s Details test / coverage (push) Successful in 15s Details lint / lint (push) Failing after 10m9s Details The firecracker setup message told users to run sudo bot-bottle backend setup --backend=firecracker which fails for exactly the users who followed the documented install. sudo replaces PATH with sudoers' secure_path — /usr/local/sbin:/usr/local/bin: /usr/sbin:/usr/bin:/sbin:/bin on Debian and Ubuntu — which deliberately excludes user-writable directories. Both supported install paths land in one: pipx uses ~/.local/bin, and install.sh's venv fallback symlinks there. So the hint works for anyone who installed system-wide and breaks with "command not found" for everyone else, which is how it survives a read-through. Add bot_bottle/invocation.py: self_path() resolves the running entry point to an absolute path, and sudo_command() builds the copy-pasteable form. The one sudo recommendation in the tree now uses it. Non-sudo hints keep the bare `bot-bottle`, which is correct — the user reached them by running it. self_path() falls back to the bare name when argv[0] cannot be resolved (a `python -m` style invocation), because a slightly wrong hint beats a traceback raised while reporting some unrelated problem. Tested behaviourally rather than by scanning source: the first version of the test grepped the module and failed on the comment explaining why the bare form is wrong. It now drives _setup_systemd() as non-root and asserts what is actually printed. Verified the guard bites by restoring the bare form and watching it fail. Not verified end to end: this message only prints on the systemd path, so it is unreachable on macOS, where the rest of this work was tested.	2026-07-27 12:35:51 -04:00
didericis	6385752040	feat: get Alpine and NixOS cells to green Completes the matrix: all five distros now pass both `test` and `test-ready`, validated end-to-end on the delphi KVM host (QEMU 11.0.2). Alpine — the generic_ cloud image silently ignores a NoCloud seed, so cloud-init never ran. Fixes, all confirmed by booting: - switch to the nocloud_ image variant (built for a local seed), - start sshd from a runcmd (OpenRC doesn't auto-start it after key injection), - give the account a throwaway password — Alpine's non-PAM sshd refuses pubkey auth for a cloud-init-locked ('!') account, unlike the UsePAM=yes distros, - install sudo via cloud-init packages: (the minimal image has none, so the test-ready `sudo apk add` prereq failed). Also add instance-id to the NoCloud meta-data, which the nocloud image requires. NixOS — publishes no downloadable cloud qcow2 (Hydra builds AMIs), so the harness now BUILDS one with nixos-generators (new scripts/linux-install-test-nixos.nix: cloud-init for the key, flakes enabled, deliberately no python/git/pipx). ensure_base_image branches to build_nixos_image for the nixos distro. The test-ready prereq uses `nix profile install` pinned to nixpkgs/nixos-24.11, because the guest's default unstable registry builds pipx from source and its test suite currently fails to build. Validation: 10/10 cells pass, clean teardown, no leaked VMs or run dirs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	496608fc25	fix: correct harness defects found running the matrix First full run on the KVM host surfaced three harness bugs and two stale image URLs; all fixed here. Harness bugs: - Liveness probe used `bot-bottle --version`, which the CLI does not implement (unknown args die non-zero) — so every SUCCESSFUL install was misreported as "no runnable entry point". Switched to `bot-bottle --help`, which exits 0 before any DB/migration/network work. - cmd_down never removed serial.log, so its rmdir failed and every run left an orphan scratch dir behind. Added serial.log to the cleanup. - The teardown trap was armed AFTER cmd_up, but cmd_up's wait_for_ssh can fail with QEMU already running (a guest that never opens SSH) — leaking the VM. Arm the trap before cmd_up. Stale image URLs: - Fedora 41 is EOL and 404s; bumped to Fedora 44 (44-1.7). - Alpine bumped to 3.21.7; its cloud images ship a .sha512 only (this verifier is sha256), so SUM_URL is now empty (skip) with a note. Validated on delphi (QEMU 11.0.2): ubuntu/fedora/arch pass BOTH test and test-ready. Alpine (cloud-init seed not applied → no SSH) and NixOS (no upstream cloud qcow2) remain blocked on image provisioning, documented in the research note as follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	218f29cb05	refactor: adopt macOS test/test-ready convention for variants Replaces the BB_TEST_PREREQS env toggle with the two subcommands the macOS harness established, so the two repos read the same: test A bare host — prerequisites NOT set up (the default state of a stock cloud image). Exercises install.sh's own prerequisite-guard logic. SOUND (PASS) when install.sh either installs cleanly or declines with one of its own recognized, actionable errors; a crash or unrecognized failure fails. test-ready Prerequisites satisfied — the harness installs python3/git/pipx first, then runs install.sh, which must actually land: entry point runnable, doctor reporting a usable python and config. Structure now mirrors scripts/macos-install-test.sh: a shared _test_cycle driving up -> (prereqs) -> run -> verdict -> down, thin cmd_test / cmd_test_ready wrappers setting _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL, a standalone `prereqs` subcommand, and a _test_teardown that prints the PASS/FAIL claim. The doctor check is now the macOS-style classifier rather than a bare exit-code read: a Traceback is an install defect (fail), a missing `ok: python:` / `ok: config:` line is a fail, and a not-ready backend is reported, not fatal (BB_TEST_REQUIRE_BACKEND=1 makes it fatal for a nested-virt host). This is where Linux diverges from macOS test-ready — a plain VM has no nested KVM for Firecracker and the harness doesn't provision the Docker backend, so backend readiness is never the Linux criterion. test-all now runs every distro × {test, test-ready}. Research note updated. Validated with `bash -n` and `shellcheck`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	1518f73de5	feat: add bare-host install variant to the Linux harness Splits the Linux clean-install harness into two variants, selected with BB_TEST_PREREQS: with (default) — install python3 + git + pipx first, then install.sh. The ready-host happy path; PASS = install.sh exits 0 and leaves a runnable bot-bottle entry point. without — run install.sh on the BARE cloud image. Exercises install.sh's own prerequisite-guard logic (python gate, git-for-git-specs gate, pipx/pip PEP-668 handling). PASS = install.sh either fully succeeds OR declines with one of its own recognized, actionable prerequisite errors; a crash or unrecognized failure is a FAIL. cmd_run now captures install.sh's exit code + full output (install.rc / install.log) instead of aborting on non-zero, so the verdict step applies the variant's criterion. The old assert_installed is replaced by a quiet entry_point_runnable helper plus classify_outcome, which matches the bare-host declines against the exact die() messages install.sh prints. doctor still runs for visibility, but only when an entry point exists. test-all now runs the full matrix -- every distro x both variants -- each cell in its own throwaway VM, and can be pinned to one variant via BB_TEST_PREREQS. Research note updated to document both variants and their pass criteria. Validated with `bash -n` and `shellcheck`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	9d82535390	feat: add Linux clean-install test harness Adds scripts/linux-install-test.sh, a throwaway-VM harness that exercises install.sh the way a brand-new user would across a Linux distro matrix (Ubuntu, Fedora, Arch, Alpine, NixOS), plus the research note docs/research/testing-clean-install-on-linux.md that motivates the approach. This is the Linux counterpart to the macOS clean-install harness. On Linux the boundary of choice flips from a throwaway user account to a disposable KVM VM: it's a genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and a single harness can swap distro cloud images to cover the several package-management regimes Linux fragments into. The host already needs KVM for the Firecracker backend, so a per-run, copy-on-write VM (qemu-img create -b base) is cheap here -- the equivalent of `docker run --rm`, for a whole machine. Per run the harness caches one read-only base image, boots a throwaway overlay via QEMU/KVM with user-mode networking (no root, no bridge), injects an ephemeral SSH key + passwordless login through a cloud-init seed ISO, installs the distro's prerequisites (python3 + git + pipx), pipes THIS checkout's install.sh into the guest exactly as `curl ... \| sh` would, and asserts the CLI installed cleanly. The overlay is deleted on teardown, so even the OS-level prerequisites are wiped -- unlike a throwaway user, the reset is total. Subcommands mirror the macOS harness (up/run/status/down/test) plus test-all (the matrix) and ssh (an interactive guest shell). test arms an EXIT/INT/TERM trap the moment the VM exists, so a failure or Ctrl-C still tears it down. Scope is installer correctness, not runtime: there is no nested KVM/Docker in the VM, so `bot-bottle doctor` correctly reports every backend not-ready and exits non-zero by design. The pass criterion is therefore install.sh exiting 0, the bot-bottle entry point being present on the fresh user's PATH, and `bot-bottle --version` running -- not a green doctor. doctor's output is still printed so a real installer regression (broken shim, import error) stays visible. Validated with `bash -n` and `shellcheck`. Runtime is host-only (needs /dev/kvm, qemu, cloud-localds), so like the macOS harness it isn't exercised by the Linux PR CI. The cloud-image URLs in the DISTRO table are the one place to bump when a distro cuts a newer build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00