bot-bottle/docs/prds/prd-new-secret-provider-encrypted-env.md

# PRD prd-new: Encrypted at-rest egress secrets (SecretProvider, interim slice)

- **Status:** Draft
- **Author:** didericis
- **Created:** 2026-07-21
- **Issue:** #355

## Summary

An interim step toward the generic `SecretProvider` (#355) that stops short
of per-request minting. Today the orchestrator holds each bottle's egress
auth tokens **in process memory only**, so any infra-container recreation
silently strips every already-running bottle of its upstream credentials.
This PRD makes those secrets survive a gateway restart by persisting them
**encrypted**, under a key that is not itself sitting next to the
ciphertext.

The end state in #355 — short-lived, scoped credentials minted per request
— removes the need to store anything durable at all. That is a larger
change gated on per-upstream minting support. This slice buys back
restart-survivability now without regressing to plaintext secrets at rest.

## Problem

`Orchestrator._tokens` (`bot_bottle/orchestrator/service.py:74-79`) is a
plain in-memory dict, deliberately never written to the registry DB:

> Held **in memory only** — never written to the registry DB — so the
> gateway can inject each bottle's upstream credential without secrets at
> rest. Lost on restart (re-launch re-registers them); the future
> SecretProvider (#355) replaces this with per-request minting.

The registry itself *is* durable (SQLite on a container-only volume), and
so is the gateway CA since #450 / `2cd44cf7`. The tokens are now the only
piece of gateway state that does not survive a restart, which makes the
failure mode both silent and confusing.

### Observed failure

Checking out a branch that touches `bot_bottle/**/*.py` changes
`source_hash()` (`bot_bottle/orchestrator/lifecycle.py:88-99`).
`MacosInfraService._source_current()`
(`bot_bottle/backend/macos_container/infra.py:159-169`) sees the mismatch
and `ensure_running()` force-removes and recreates the infra container
(`infra.py:198-208`). The registry rows survive on the DB volume; the CA
survives on its host bind-mount; `_tokens` comes back empty.

Every already-running bottle then fails closed, mid-session, on its next
outbound request:

- `/resolve` succeeds — the bottle is still `active` in
  `orchestrator_bottles` and its policy blob is served intact, including
  `- host: "api.anthropic.com"` with `auth_scheme: Bearer` /
  `token_env: EGRESS_TOKEN_0`.
- `tokens_for()` returns `{}`, so the resolved env overlay has no
  `EGRESS_TOKEN_0`.
- `decide()` (`bot_bottle/egress_addon_core.py:644-652`) blocks with
  `egress: route for 'api.anthropic.com' declared auth but env var
  'EGRESS_TOKEN_0' is unset` — an 89-byte `403` on every request.

Confirmed live on the macOS backend on 2026-07-21: two bottles running
since 20:29/20:30 were still registered `active` with valid policy after
the 23:05 infra recreation, and both took 89-byte `403`s from then on,
while a bottle launched *after* the recreation egressed normally. The
recovery today is to relaunch every affected bottle.

Note this is a re-attachment blocker distinct from #443/#445 and from #450
— the CA and the gateway address were both fine. It is specifically the
credential wipe.

## Goals / Success criteria

1. A bottle's egress auth tokens survive infra-container recreation: an
   already-running bottle keeps egressing across a gateway restart with no
   relaunch and no operator action.
2. Secrets are **never** at rest in plaintext, and never at rest next to a
   key that trivially decrypts them.
3. Compromise of the registry DB file alone does not yield usable
   upstream credentials.
4. The stored form is revocable and rotatable without relaunching bottles
   that are not affected.
5. When the retained bottle record reaches the lifecycle status `removed`,
   its stored secrets are destroyed. Status/event persistence and the removal
   transition land separately; this interim slice intentionally retains the
   ciphertext across today's teardown/reconcile calls until that lifecycle is
   available.
6. Migration is transparent: existing bottles keep working, no manifest
   changes required.

## Non-goals

- **Per-request minting** of short-lived scoped credentials. That is the
  #355 end state; this PRD is explicitly the interim slice and should not
  foreclose it.
- Generalizing `DeployKeyProvisioner` into the full `SecretProvider` ABC,
  or the manifest-level `{ provider: <name> }` reference surface.
- User-extensible provider discovery (`~/.bot-bottle/contrib/<name>/`).
- Changing the `/resolve` contract's shape (it already carries `tokens`).
- Fixing the *trigger* — `source_hash` churn on branch switch. Recreating
  infra is legitimate; it just must not cost running bottles their
  credentials. A separate guard that refuses recreation while bottles are
  active is complementary and out of scope here.

## Design

> **TODO (didericis):** the encryption flow goes here — key custody, where
> the key material lives relative to the ciphertext, the wrap/unwrap path
> at register and at `/resolve`, and what an attacker who holds only the
> DB (or only the host, or only the infra container) can recover.

Constraints the design has to satisfy, for reference while drafting:

- The gateway's `PolicyResolver` needs the cleartext at request time, on
  the data-plane path, so unwrap has to be cheap enough to sit in a
  per-flow `/resolve` (or be cached in memory after first unwrap).
- The infra container is recreated routinely and unattended. Anything
  requiring an interactive unlock on every recreation defeats the goal.
- The DB lives on a container-only volume that the host does not mount, so
  host-side and guest-side components see different filesystems — that
  asymmetry is available as a place to split custody.
- The agent must never be able to reach the key material. It is a separate
  container with no control-plane token, which is the existing boundary.

## Open questions

- Where does the unwrap key live, and what recreates/re-derives it when the
  infra container is rebuilt?
- Is the cleartext cached in memory after first unwrap, or unwrapped per
  request? (Latency vs. exposure window.)
- What is the rotation story — re-wrap in place, or force re-registration?
- Does this land behind a flag, or replace `_tokens` outright?