didericis
46e595ffb1
tracker-policy-pr / check-pr (pull_request) Successful in 11s
The orchestrator holds each bottle's egress auth tokens in process memory only, so recreating the infra container strips every already-running bottle of its upstream credentials. The registry row and the gateway CA both survive; the tokens do not, so /resolve serves an intact policy with an empty token map and the addon fails closed on `token_env unset`. Drafts the interim slice of #355: persist the tokens encrypted so they survive a restart, without regressing to plaintext at rest and without foreclosing the per-request minting end state. Design section is left for the author to fill in.
129 lines
6.0 KiB
Markdown
129 lines
6.0 KiB
Markdown
# PRD prd-new: Encrypted at-rest egress secrets (SecretProvider, interim slice)
|
|
|
|
- **Status:** Draft
|
|
- **Author:** didericis
|
|
- **Created:** 2026-07-21
|
|
- **Issue:** #355
|
|
|
|
## Summary
|
|
|
|
An interim step toward the generic `SecretProvider` (#355) that stops short
|
|
of per-request minting. Today the orchestrator holds each bottle's egress
|
|
auth tokens **in process memory only**, so any infra-container recreation
|
|
silently strips every already-running bottle of its upstream credentials.
|
|
This PRD makes those secrets survive a gateway restart by persisting them
|
|
**encrypted**, under a key that is not itself sitting next to the
|
|
ciphertext.
|
|
|
|
The end state in #355 — short-lived, scoped credentials minted per request
|
|
— removes the need to store anything durable at all. That is a larger
|
|
change gated on per-upstream minting support. This slice buys back
|
|
restart-survivability now without regressing to plaintext secrets at rest.
|
|
|
|
## Problem
|
|
|
|
`Orchestrator._tokens` (`bot_bottle/orchestrator/service.py:74-79`) is a
|
|
plain in-memory dict, deliberately never written to the registry DB:
|
|
|
|
> Held **in memory only** — never written to the registry DB — so the
|
|
> gateway can inject each bottle's upstream credential without secrets at
|
|
> rest. Lost on restart (re-launch re-registers them); the future
|
|
> SecretProvider (#355) replaces this with per-request minting.
|
|
|
|
The registry itself *is* durable (SQLite on a container-only volume), and
|
|
so is the gateway CA since #450 / `2cd44cf7`. The tokens are now the only
|
|
piece of gateway state that does not survive a restart, which makes the
|
|
failure mode both silent and confusing.
|
|
|
|
### Observed failure
|
|
|
|
Checking out a branch that touches `bot_bottle/**/*.py` changes
|
|
`source_hash()` (`bot_bottle/orchestrator/lifecycle.py:88-99`).
|
|
`MacosInfraService._source_current()`
|
|
(`bot_bottle/backend/macos_container/infra.py:159-169`) sees the mismatch
|
|
and `ensure_running()` force-removes and recreates the infra container
|
|
(`infra.py:198-208`). The registry rows survive on the DB volume; the CA
|
|
survives on its host bind-mount; `_tokens` comes back empty.
|
|
|
|
Every already-running bottle then fails closed, mid-session, on its next
|
|
outbound request:
|
|
|
|
- `/resolve` succeeds — the bottle is still `active` in
|
|
`orchestrator_bottles` and its policy blob is served intact, including
|
|
`- host: "api.anthropic.com"` with `auth_scheme: Bearer` /
|
|
`token_env: EGRESS_TOKEN_0`.
|
|
- `tokens_for()` returns `{}`, so the resolved env overlay has no
|
|
`EGRESS_TOKEN_0`.
|
|
- `decide()` (`bot_bottle/egress_addon_core.py:644-652`) blocks with
|
|
`egress: route for 'api.anthropic.com' declared auth but env var
|
|
'EGRESS_TOKEN_0' is unset` — an 89-byte `403` on every request.
|
|
|
|
Confirmed live on the macOS backend on 2026-07-21: two bottles running
|
|
since 20:29/20:30 were still registered `active` with valid policy after
|
|
the 23:05 infra recreation, and both took 89-byte `403`s from then on,
|
|
while a bottle launched *after* the recreation egressed normally. The
|
|
recovery today is to relaunch every affected bottle.
|
|
|
|
Note this is a re-attachment blocker distinct from #443/#445 and from #450
|
|
— the CA and the gateway address were both fine. It is specifically the
|
|
credential wipe.
|
|
|
|
## Goals / Success criteria
|
|
|
|
1. A bottle's egress auth tokens survive infra-container recreation: an
|
|
already-running bottle keeps egressing across a gateway restart with no
|
|
relaunch and no operator action.
|
|
2. Secrets are **never** at rest in plaintext, and never at rest next to a
|
|
key that trivially decrypts them.
|
|
3. Compromise of the registry DB file alone does not yield usable
|
|
upstream credentials.
|
|
4. The stored form is revocable and rotatable without relaunching bottles
|
|
that are not affected.
|
|
5. Reap/teardown destroys a bottle's stored secrets along with its
|
|
registry row (no ciphertext outliving its bottle).
|
|
6. Migration is transparent: existing bottles keep working, no manifest
|
|
changes required.
|
|
|
|
## Non-goals
|
|
|
|
- **Per-request minting** of short-lived scoped credentials. That is the
|
|
#355 end state; this PRD is explicitly the interim slice and should not
|
|
foreclose it.
|
|
- Generalizing `DeployKeyProvisioner` into the full `SecretProvider` ABC,
|
|
or the manifest-level `{ provider: <name> }` reference surface.
|
|
- User-extensible provider discovery (`~/.bot-bottle/contrib/<name>/`).
|
|
- Changing the `/resolve` contract's shape (it already carries `tokens`).
|
|
- Fixing the *trigger* — `source_hash` churn on branch switch. Recreating
|
|
infra is legitimate; it just must not cost running bottles their
|
|
credentials. A separate guard that refuses recreation while bottles are
|
|
active is complementary and out of scope here.
|
|
|
|
## Design
|
|
|
|
> **TODO (didericis):** the encryption flow goes here — key custody, where
|
|
> the key material lives relative to the ciphertext, the wrap/unwrap path
|
|
> at register and at `/resolve`, and what an attacker who holds only the
|
|
> DB (or only the host, or only the infra container) can recover.
|
|
|
|
Constraints the design has to satisfy, for reference while drafting:
|
|
|
|
- The gateway's `PolicyResolver` needs the cleartext at request time, on
|
|
the data-plane path, so unwrap has to be cheap enough to sit in a
|
|
per-flow `/resolve` (or be cached in memory after first unwrap).
|
|
- The infra container is recreated routinely and unattended. Anything
|
|
requiring an interactive unlock on every recreation defeats the goal.
|
|
- The DB lives on a container-only volume that the host does not mount, so
|
|
host-side and guest-side components see different filesystems — that
|
|
asymmetry is available as a place to split custody.
|
|
- The agent must never be able to reach the key material. It is a separate
|
|
container with no control-plane token, which is the existing boundary.
|
|
|
|
## Open questions
|
|
|
|
- Where does the unwrap key live, and what recreates/re-derives it when the
|
|
infra container is rebuilt?
|
|
- Is the cleartext cached in memory after first unwrap, or unwrapped per
|
|
request? (Latency vs. exposure window.)
|
|
- What is the rotation story — re-wrap in place, or force re-registration?
|
|
- Does this land behind a flag, or replace `_tokens` outright?
|