bot-bottle/docs/prds/0081-reprovision-gateway-state-on-bringup.md

PRD 0081: Reprovision gateway-dependent state on gateway bring-up

• Status: Active
• Author: claude
• Created: 2026-07-26
• Issue: #510, #512

Summary

When the gateway is (re)built, reconcile every already-running bottle against the fresh gateway instead of persisting the gateway's state. On a gateway cold boot, the gateway reconciles all live bottles in one flow: replace each agent's trusted CA with the freshly-minted gateway CA, re-provision each bottle's git-gate repos + creds onto the gateway, and restore each bottle's egress tokens. One mechanism across all three services; the CA rotates for free on every bring-up.

Problem

A gateway rebuild/restart silently breaks every already-running bottle:

• CA (#510). The gateway's mitmproxy mints a new CA on a fresh rootfs; agents still trust the old one, so egress fails TLS verification (SSL certificate verification failed).
• git-gate (#512). Per-bottle bare repos (/git/<id>) + deploy creds (/git-gate/creds/<id>) live in the gateway's ephemeral rootfs; a rebuild wipes them and the agent 404s on fetch/push.

Both are the same root cause: per-bottle gateway-dependent state is provisioned once, at bottle launch, and nothing restores it for already-running bottles when the gateway comes back. The existing launch-time reprovision (reprovision_bottles) restores only egress tokens, and only as a side effect of the next launch.

Persisting the state (a host bind-mount / a per-VM data volume per service) was prototyped and rejected: it differs per service (a volume for the CA, another for git-gate, reprovision for tokens), it pins the CA static forever (no rotation), and it adds volume surface that docker volume prune / a wiped cache can silently destroy (the original #450 failure mode).

Goals / Success Criteria

• A gateway (re)boot restores all running bottles' gateway-dependent state with no manual step and no relaunch — the agent's next egress / fetch / push just works.
• One reconcile flow covering CA, git-gate, and egress tokens, rather than a different mechanism per service.
• The CA rotates on every gateway bring-up (no long-lived CA), distributed to running agents by the same reconcile.
• Reconcile failures are loud, never hidden: if a running bottle can't be re-attached to the fresh gateway, the bring-up fails and names the bottle rather than leaving it silently unable to reach the gateway (a bottle whose egress just starts failing TLS is worse than a loud failure). Every bottle is attempted first and the failures are raised together, so one bring-up surfaces the full blast radius. (Reversed from an earlier "tolerate per-bottle failures" draft after #519 review.)
• All backends (Firecracker, docker, macOS) reconcile through the same contract — an abstract method on the backend base class, so a new backend cannot forget to implement it and none drifts onto a bespoke mechanism.

Non-goals

• Deliberate mid-session CA rotation without a gateway restart — rotate_ca stays for that operator action.
• Changing source-IP attribution, /resolve, or the plane split (#469).

Design

The contract — attach_bottled_agents_to_gateway(), a concrete flow on the backend base. Reconciling running bottles against the current gateway is a backend responsibility (only the backend can enumerate its agents and reach them — firecracker over SSH, docker/macOS over exec/cp). To keep every backend reconciling identically, the control flow lives in a concrete attach_bottled_agents_to_gateway() on BottleBackend (backend/base.py) and each backend overrides only three primitives:

1. _gateway_attach_resources() — gather what a bottle needs to attach (the gateway CA now; egress secrets + git-gate state as this grows).
2. _running_bottles() — enumerate the live bottles as backend handles.
3. _attach_bottle_to_gateway(bottle, resources) — push the resources into one bottle, raising InfraLaunchError (naming it) on failure.

The base gathers resources once, attempts every bottle, and — fail hard, never skip — raises an aggregate InfraLaunchError if any failed, so a backend can't drift onto a bespoke loop or a silent skip. (Encoding shared behaviour in the base backend, with subclasses overriding primitives rather than control flow, is ADR 0006.) The host calls it whenever the gateway is (re)brought up. It reprovisions all registered bottles' gateway-dependent state: CA, git-gate, and egress tokens.

Trigger — the gateway bring-up path. The host calls attach_bottled_agents_to_gateway() only when the gateway was actually (re)brought up — the cold-boot branch of the infra bring-up (e.g. FirecrackerInfraService.ensure_running after it boots a fresh pair), never on an adopt of a healthy, current gateway (state intact). So it fires exactly when the gateway was (re)booted — including orchestrator restarts, since the pair boots together — and there is no bare-restart path that bypasses bring-up.

Per-backend implementation. Each backend's attach_bottled_agents_to_gateway enumerates its live bottles and, for each, reconciles the three services against the current gateway. The firecracker implementation, once the gateway VM is up and its CA is available:

1. Map each live bottle's guest IP → bottle_id from the orchestrator registry (list_bottles).
2. Install the current shared git-gate hooks (git_gate_render_hook / git_gate_render_access_hook) into the fresh gateway once — rendered from code, never from a bottle's possibly-stale state dir.
3. For each live agent VM (enumerated from its run dir):
   • CA: SSH the current gateway CA into the agent's trust store and run update-ca-certificates (unconditional replace — there is one gateway, so no fingerprint match is needed).
   • git-gate: rebuild the bottle's upstreams from its persisted git-gate state dir (deploy key, known_hosts, upstream URL) and re-init its bare repos
     • per-repo creds under /git/<bottle_id>.
   • egress token: read the agent's ENV_VAR_SECRET and feed reprovision_bottles, restoring the orchestrator's in-memory tokens.
4. If any per-bottle step fails, the reconcile attempts the rest and then raises an aggregate naming every bottle that failed — bring-up fails loudly rather than leaving a bottle silently unable to reach the gateway.

Retire the persistence prototype. No CA data volume, no git-gate data volume (the abandoned PRs #511 / #513). The launch-time _reprovision_running_bottles call folds into this bring-up reconcile, so egress tokens are restored on the same cold-boot trigger (an adopt needs no restore — the orchestrator never restarted).

CA rotation. Because the gateway rootfs is ephemeral, every cold boot mints a fresh CA; reconcile is what distributes it, so a routine gateway rebuild doubles as a CA rotation with zero extra machinery.

Open questions

• Docker's existing host-bind-mounted CA (host_gateway_ca_dir): once docker's attach_bottled_agents_to_gateway pushes the CA to running agents on bring-up, the bind-mount is redundant — drop it (so docker rotates like firecracker) or keep it as belt-and-suspenders? Leaning drop, for one behaviour across backends.