From 0f98d75eff6e905d3676b059e494a302b23ae2a7 Mon Sep 17 00:00:00 2001 From: didericis Date: Tue, 21 Jul 2026 19:22:24 -0400 Subject: [PATCH] docs(prd): draft encrypted at-rest egress secrets (SecretProvider interim) The orchestrator holds each bottle's egress auth tokens in process memory only, so recreating the infra container strips every already-running bottle of its upstream credentials. The registry row and the gateway CA both survive; the tokens do not, so /resolve serves an intact policy with an empty token map and the addon fails closed on `token_env unset`. Drafts the interim slice of #355: persist the tokens encrypted so they survive a restart, without regressing to plaintext at rest and without foreclosing the per-request minting end state. Design section is left for the author to fill in. --- .../prd-new-secret-provider-encrypted-env.md | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 docs/prds/prd-new-secret-provider-encrypted-env.md diff --git a/docs/prds/prd-new-secret-provider-encrypted-env.md b/docs/prds/prd-new-secret-provider-encrypted-env.md new file mode 100644 index 0000000..8ca5f9a --- /dev/null +++ b/docs/prds/prd-new-secret-provider-encrypted-env.md @@ -0,0 +1,128 @@ +# PRD prd-new: Encrypted at-rest egress secrets (SecretProvider, interim slice) + +- **Status:** Draft +- **Author:** didericis +- **Created:** 2026-07-21 +- **Issue:** #355 + +## Summary + +An interim step toward the generic `SecretProvider` (#355) that stops short +of per-request minting. Today the orchestrator holds each bottle's egress +auth tokens **in process memory only**, so any infra-container recreation +silently strips every already-running bottle of its upstream credentials. +This PRD makes those secrets survive a gateway restart by persisting them +**encrypted**, under a key that is not itself sitting next to the +ciphertext. + +The end state in #355 — short-lived, scoped credentials minted per request +— removes the need to store anything durable at all. That is a larger +change gated on per-upstream minting support. This slice buys back +restart-survivability now without regressing to plaintext secrets at rest. + +## Problem + +`Orchestrator._tokens` (`bot_bottle/orchestrator/service.py:74-79`) is a +plain in-memory dict, deliberately never written to the registry DB: + +> Held **in memory only** — never written to the registry DB — so the +> gateway can inject each bottle's upstream credential without secrets at +> rest. Lost on restart (re-launch re-registers them); the future +> SecretProvider (#355) replaces this with per-request minting. + +The registry itself *is* durable (SQLite on a container-only volume), and +so is the gateway CA since #450 / `2cd44cf7`. The tokens are now the only +piece of gateway state that does not survive a restart, which makes the +failure mode both silent and confusing. + +### Observed failure + +Checking out a branch that touches `bot_bottle/**/*.py` changes +`source_hash()` (`bot_bottle/orchestrator/lifecycle.py:88-99`). +`MacosInfraService._source_current()` +(`bot_bottle/backend/macos_container/infra.py:159-169`) sees the mismatch +and `ensure_running()` force-removes and recreates the infra container +(`infra.py:198-208`). The registry rows survive on the DB volume; the CA +survives on its host bind-mount; `_tokens` comes back empty. + +Every already-running bottle then fails closed, mid-session, on its next +outbound request: + +- `/resolve` succeeds — the bottle is still `active` in + `orchestrator_bottles` and its policy blob is served intact, including + `- host: "api.anthropic.com"` with `auth_scheme: Bearer` / + `token_env: EGRESS_TOKEN_0`. +- `tokens_for()` returns `{}`, so the resolved env overlay has no + `EGRESS_TOKEN_0`. +- `decide()` (`bot_bottle/egress_addon_core.py:644-652`) blocks with + `egress: route for 'api.anthropic.com' declared auth but env var + 'EGRESS_TOKEN_0' is unset` — an 89-byte `403` on every request. + +Confirmed live on the macOS backend on 2026-07-21: two bottles running +since 20:29/20:30 were still registered `active` with valid policy after +the 23:05 infra recreation, and both took 89-byte `403`s from then on, +while a bottle launched *after* the recreation egressed normally. The +recovery today is to relaunch every affected bottle. + +Note this is a re-attachment blocker distinct from #443/#445 and from #450 +— the CA and the gateway address were both fine. It is specifically the +credential wipe. + +## Goals / Success criteria + +1. A bottle's egress auth tokens survive infra-container recreation: an + already-running bottle keeps egressing across a gateway restart with no + relaunch and no operator action. +2. Secrets are **never** at rest in plaintext, and never at rest next to a + key that trivially decrypts them. +3. Compromise of the registry DB file alone does not yield usable + upstream credentials. +4. The stored form is revocable and rotatable without relaunching bottles + that are not affected. +5. Reap/teardown destroys a bottle's stored secrets along with its + registry row (no ciphertext outliving its bottle). +6. Migration is transparent: existing bottles keep working, no manifest + changes required. + +## Non-goals + +- **Per-request minting** of short-lived scoped credentials. That is the + #355 end state; this PRD is explicitly the interim slice and should not + foreclose it. +- Generalizing `DeployKeyProvisioner` into the full `SecretProvider` ABC, + or the manifest-level `{ provider: }` reference surface. +- User-extensible provider discovery (`~/.bot-bottle/contrib//`). +- Changing the `/resolve` contract's shape (it already carries `tokens`). +- Fixing the *trigger* — `source_hash` churn on branch switch. Recreating + infra is legitimate; it just must not cost running bottles their + credentials. A separate guard that refuses recreation while bottles are + active is complementary and out of scope here. + +## Design + +> **TODO (didericis):** the encryption flow goes here — key custody, where +> the key material lives relative to the ciphertext, the wrap/unwrap path +> at register and at `/resolve`, and what an attacker who holds only the +> DB (or only the host, or only the infra container) can recover. + +Constraints the design has to satisfy, for reference while drafting: + +- The gateway's `PolicyResolver` needs the cleartext at request time, on + the data-plane path, so unwrap has to be cheap enough to sit in a + per-flow `/resolve` (or be cached in memory after first unwrap). +- The infra container is recreated routinely and unattended. Anything + requiring an interactive unlock on every recreation defeats the goal. +- The DB lives on a container-only volume that the host does not mount, so + host-side and guest-side components see different filesystems — that + asymmetry is available as a place to split custody. +- The agent must never be able to reach the key material. It is a separate + container with no control-plane token, which is the existing boundary. + +## Open questions + +- Where does the unwrap key live, and what recreates/re-derives it when the + infra container is rebuilt? +- Is the cleartext cached in memory after first unwrap, or unwrapped per + request? (Latency vs. exposure window.) +- What is the rotation story — re-wrap in place, or force re-registration? +- Does this land behind a flag, or replace `_tokens` outright?