bot-bottle/docs/prds/0083-reprovision-gateway-state-on-bringup.md

PRD 0083: Reprovision gateway-dependent state on gateway bring-up

• Status: Active
• Author: claude
• Created: 2026-07-26
• Issue: #510, #512

Summary

When the gateway is (re)built, reconcile every already-running bottle against the fresh gateway instead of persisting the gateway's state. On a gateway cold boot, the gateway reconciles all live bottles in one flow: replace each agent's trusted CA with the freshly-minted gateway CA, re-provision each bottle's git-gate repos + creds onto the gateway, and restore each bottle's egress tokens. One mechanism across all three services; the CA rotates for free on every bring-up.

Problem

A gateway rebuild/restart silently breaks every already-running bottle:

• CA (#510). The gateway's mitmproxy mints a new CA on a fresh rootfs; agents still trust the old one, so egress fails TLS verification (SSL certificate verification failed).
• git-gate (#512). Per-bottle bare repos (/git/<id>) + deploy creds (/git-gate/creds/<id>) live in the gateway's ephemeral rootfs; a rebuild wipes them and the agent 404s on fetch/push.

Both are the same root cause: per-bottle gateway-dependent state is provisioned once, at bottle launch, and nothing restores it for already-running bottles when the gateway comes back. The existing launch-time reprovision (reprovision_bottles) restores only egress tokens, and only as a side effect of the next launch.

Persisting the state (a host bind-mount / a per-VM data volume per service) was prototyped and rejected: it differs per service (a volume for the CA, another for git-gate, reprovision for tokens), it pins the CA static forever (no rotation), and it adds volume surface that docker volume prune / a wiped cache can silently destroy (the original #450 failure mode).

Goals / Success Criteria

• A gateway (re)boot restores all running bottles' gateway-dependent state with no manual step and no relaunch — the agent's next egress / fetch / push just works.
• One reconcile flow covering CA, git-gate, and egress tokens, rather than a different mechanism per service.
• The CA rotates on every gateway bring-up (no long-lived CA), distributed to running agents by the same reconcile.
• Per-bottle failures are tolerated: one unreachable or malformed bottle does not block the others or the gateway coming up.
• All backends (Firecracker, docker, macOS) reconcile through the same contract — an abstract method on the backend base class, so a new backend cannot forget to implement it and none drifts onto a bespoke mechanism.

Non-goals

• Deliberate mid-session CA rotation without a gateway restart — rotate_ca stays for that operator action.
• Changing source-IP attribution, /resolve, or the plane split (#469).

Design

The contract — attach_bottled_agents_to_gateway() on the backend ABC. Reconciling running bottles against the current gateway is a backend responsibility (only the backend can enumerate its agents and reach them — firecracker over SSH, docker/macOS over exec/cp), so it is an @abc.abstractmethod on BottleBackend (backend/base.py). Every backend implements it; the host calls it whenever the gateway is (re)brought up. This is what makes the fix cross-backend by construction rather than a per-backend follow-up. It reprovisions all registered bottles' gateway-dependent state: CA, git-gate, and egress tokens.

Trigger — the gateway bring-up path. The host calls attach_bottled_agents_to_gateway() only when the gateway was actually (re)brought up — the cold-boot branch of the infra bring-up (e.g. FirecrackerInfraService.ensure_running after it boots a fresh pair), never on an adopt of a healthy, current gateway (state intact). So it fires exactly when the gateway was (re)booted — including orchestrator restarts, since the pair boots together — and there is no bare-restart path that bypasses bring-up.

Per-backend implementation. Each backend's attach_bottled_agents_to_gateway enumerates its live bottles and, for each, reconciles the three services against the current gateway. The firecracker implementation, once the gateway VM is up and its CA is available:

1. Map each live bottle's guest IP → bottle_id from the orchestrator registry (list_bottles).
2. Install the current shared git-gate hooks (git_gate_render_hook / git_gate_render_access_hook) into the fresh gateway once — rendered from code, never from a bottle's possibly-stale state dir.
3. For each live agent VM (enumerated from its run dir):
   • CA: SSH the current gateway CA into the agent's trust store and run update-ca-certificates (unconditional replace — there is one gateway, so no fingerprint match is needed).
   • git-gate: rebuild the bottle's upstreams from its persisted git-gate state dir (deploy key, known_hosts, upstream URL) and re-init its bare repos
     • per-repo creds under /git/<bottle_id>.
   • egress token: read the agent's ENV_VAR_SECRET and feed reprovision_bottles, restoring the orchestrator's in-memory tokens.
4. Every per-bottle step is wrapped so one failure is logged and skipped.

Retire the persistence prototype. No CA data volume, no git-gate data volume (the abandoned PRs #511 / #513). The launch-time _reprovision_running_bottles call folds into this bring-up reconcile, so egress tokens are restored on the same cold-boot trigger (an adopt needs no restore — the orchestrator never restarted).

CA rotation. Because the gateway rootfs is ephemeral, every cold boot mints a fresh CA; reconcile is what distributes it, so a routine gateway rebuild doubles as a CA rotation with zero extra machinery.

Open questions

• Docker's existing host-bind-mounted CA (host_gateway_ca_dir): once docker's attach_bottled_agents_to_gateway pushes the CA to running agents on bring-up, the bind-mount is redundant — drop it (so docker rotates like firecracker) or keep it as belt-and-suspenders? Leaning drop, for one behaviour across backends.