bot-bottle/docs/prds/prd-new-secret-provider-encrypted-env.md

PRD prd-new: Encrypted at-rest egress secrets (SecretProvider, interim slice)

• Status: Draft
• Author: didericis
• Created: 2026-07-21
• Issue: #355

Summary

An interim step toward the generic SecretProvider (#355) that stops short of per-request minting. Today the orchestrator holds each bottle's egress auth tokens in process memory only, so any infra-container recreation silently strips every already-running bottle of its upstream credentials. This PRD makes those secrets survive a gateway restart by persisting them encrypted, under a key that is not itself sitting next to the ciphertext.

The end state in #355 — short-lived, scoped credentials minted per request — removes the need to store anything durable at all. That is a larger change gated on per-upstream minting support. This slice buys back restart-survivability now without regressing to plaintext secrets at rest.

Problem

Orchestrator._tokens (bot_bottle/orchestrator/service.py:74-79) is a plain in-memory dict, deliberately never written to the registry DB:

Held in memory only — never written to the registry DB — so the gateway can inject each bottle's upstream credential without secrets at rest. Lost on restart (re-launch re-registers them); the future SecretProvider (#355) replaces this with per-request minting.

The registry itself is durable (SQLite on a container-only volume), and so is the gateway CA since #450 / 2cd44cf7. The tokens are now the only piece of gateway state that does not survive a restart, which makes the failure mode both silent and confusing.

Observed failure

Checking out a branch that touches bot_bottle/**/*.py changes source_hash() (bot_bottle/orchestrator/lifecycle.py:88-99). MacosInfraService._source_current() (bot_bottle/backend/macos_container/infra.py:159-169) sees the mismatch and ensure_running() force-removes and recreates the infra container (infra.py:198-208). The registry rows survive on the DB volume; the CA survives on its host bind-mount; _tokens comes back empty.

Every already-running bottle then fails closed, mid-session, on its next outbound request:

• /resolve succeeds — the bottle is still active in orchestrator_bottles and its policy blob is served intact, including - host: "api.anthropic.com" with auth_scheme: Bearer / token_env: EGRESS_TOKEN_0.
• tokens_for() returns {}, so the resolved env overlay has no EGRESS_TOKEN_0.
• decide() (bot_bottle/egress_addon_core.py:644-652) blocks with egress: route for 'api.anthropic.com' declared auth but env var 'EGRESS_TOKEN_0' is unset — an 89-byte 403 on every request.

Confirmed live on the macOS backend on 2026-07-21: two bottles running since 20:29/20:30 were still registered active with valid policy after the 23:05 infra recreation, and both took 89-byte 403s from then on, while a bottle launched after the recreation egressed normally. The recovery today is to relaunch every affected bottle.

Note this is a re-attachment blocker distinct from #443/#445 and from #450 — the CA and the gateway address were both fine. It is specifically the credential wipe.

Goals / Success criteria

1. A bottle's egress auth tokens survive infra-container recreation: an already-running bottle keeps egressing across a gateway restart with no relaunch and no operator action.
2. Secrets are never at rest in plaintext, and never at rest next to a key that trivially decrypts them.
3. Compromise of the registry DB file alone does not yield usable upstream credentials.
4. The stored form is revocable and rotatable without relaunching bottles that are not affected.
5. When the retained bottle record reaches the lifecycle status removed, its stored secrets are destroyed. Status/event persistence and the removal transition land separately; this interim slice intentionally retains the ciphertext across today's teardown/reconcile calls until that lifecycle is available.
6. Migration is transparent: existing bottles keep working, no manifest changes required.

Non-goals

• Per-request minting of short-lived scoped credentials. That is the #355 end state; this PRD is explicitly the interim slice and should not foreclose it.
• Generalizing DeployKeyProvisioner into the full SecretProvider ABC, or the manifest-level { provider: <name> } reference surface.
• User-extensible provider discovery (~/.bot-bottle/contrib/<name>/).
• Changing the /resolve contract's shape (it already carries tokens).
• Fixing the trigger — source_hash churn on branch switch. Recreating infra is legitimate; it just must not cost running bottles their credentials. A separate guard that refuses recreation while bottles are active is complementary and out of scope here.

Design

TODO (didericis): the encryption flow goes here — key custody, where the key material lives relative to the ciphertext, the wrap/unwrap path at register and at /resolve, and what an attacker who holds only the DB (or only the host, or only the infra container) can recover.

Constraints the design has to satisfy, for reference while drafting:

• The gateway's PolicyResolver needs the cleartext at request time, on the data-plane path, so unwrap has to be cheap enough to sit in a per-flow /resolve (or be cached in memory after first unwrap).
• The infra container is recreated routinely and unattended. Anything requiring an interactive unlock on every recreation defeats the goal.
• The DB lives on a container-only volume that the host does not mount, so host-side and guest-side components see different filesystems — that asymmetry is available as a place to split custody.
• The agent must never be able to reach the key material. It is a separate container with no control-plane token, which is the existing boundary.

Open questions

• Where does the unwrap key live, and what recreates/re-derives it when the infra container is rebuilt?
• Is the cleartext cached in memory after first unwrap, or unwrapped per request? (Latency vs. exposure window.)
• What is the rotation story — re-wrap in place, or force re-registration?
• Does this land behind a flag, or replace _tokens outright?