PRD: Encrypted at-rest egress secrets (SecretProvider interim slice) #458
Reference in New Issue
Block a user
Delete Branch "prd-secret-provider-encrypted-env"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Part of #355.
PRD: https://gitea.dideric.is/didericis/bot-bottle/src/commit/46e595ff/docs/prds/prd-new-secret-provider-encrypted-env.md
Summary
PRD-only draft for the interim slice of #355: persist per-bottle egress auth tokens encrypted, so they survive an infra-container restart, without regressing to plaintext secrets at rest and without foreclosing the per-request-minting end state.
The Design section is intentionally a TODO — @didericis has the encryption/key-custody flow in mind and will fill it in on this branch. Everything else (problem, evidence, goals, non-goals, open questions) is drafted.
Motivating failure
Orchestrator._tokens(orchestrator/service.py:74-79) is in-memory only and deliberately never written to the registry DB. It is now the only piece of gateway state that does not survive a restart — the registry rows live on the container-only DB volume, and the CA has been host-resident since #450 /2cd44cf7.So any infra recreation silently strips every already-running bottle of its upstream credentials:
bot_bottle/**/*.pychangessource_hash(),_source_current()sees the mismatch, andensure_running()force-removes and recreates the infra container./resolvestill succeeds — the bottle isactivewith its policy blob intact, includingapi.anthropic.comwithauth_scheme: Bearer/token_env: EGRESS_TOKEN_0.tokens_for()returns{}, sodecide()blocks withegress: route for 'api.anthropic.com' declared auth but env var 'EGRESS_TOKEN_0' is unset— an 89-byte403on every request, mid-session.Confirmed live on the macOS backend on 2026-07-21: two bottles running since 20:29/20:30 kept valid registry rows and policy across the 23:05 recreation and took 89-byte
403s from then on, while a bottle launched after the recreation egressed fine. Recovery today is relaunching every affected bottle.This is a re-attachment blocker distinct from #443/#445 and from #450 — the CA and the gateway address were both correct. It is specifically the credential wipe.
Scope
PRD only, no implementation. Status stays
Draft(CI assigns the number and flips status when implementation ships with it).Explicitly not in this PRD: the full
SecretProviderABC, the manifest{ provider: <name> }surface, user-extensible provider discovery, and per-request minting — all still #355. Also not here: a guard that refuses infra recreation while bottles are active, which is complementary and worth doing separately.@@ -0,0 +116,4 @@host-side and guest-side components see different filesystems — thatasymmetry is available as a place to split custody.- The agent must never be able to reach the key material. It is a separatecontainer with no control-plane token, which is the existing boundary.Flow:
ENV_VAR_SECRETwhen it's started up with a random key, which is given to the agent as an env var (not logged/not persisted)bottled_agent_secretstable in the orchestrator db (columns arebottled_agent_id,value, andtype, wheretypeis set toinjected_env_var)reprovision_gatewayrequest to the orchestrator which sends theENV_VAR_SECRETand injects the env vars back into the gatewayWe'll also want a follow up step (later PR) where we clean up those rows after the
bottled_agentgets torn down.On it. Here is the implementation shape I am working with:
Step 1 (agent startup):
ENV_VAR_SECRETis generated host-side inprovision_bottleusingsecrets.token_urlsafe(32). It is passed to the agent via the compose env as a bare name (so the value does not land in the compose file on disk) and passed alongsidetokenstoregister_bottle.Step 2 (encrypt at provision time):
Orchestrator.launch_bottleacceptsenv_var_secret; when tokens are present it encryptsjson.dumps(tokens)with HMAC-SHA256 in CTR mode (stdlib-only,nonce || ciphertextas base64url) and stores the ciphertext in the newbottled_agent_secretstable (bottled_agent_id,value,type=injected_env_var). The key never touches the DB.Step 3 (reprovision on restart): New
reprovision-gatewayMCP tool in the supervise server. The agent calls it with its$ENV_VAR_SECRET. The supervise server attributes the call (source IP + identity token), then POSTs{env_var_secret}toPOST /bottles/<id>/reprovision_gatewayon the orchestrator (with the control-plane auth it already holds). The orchestrator decrypts the stored ciphertext and restores_tokens[bottle_id].The prompting mechanism — what causes the agent to call this after an orchestrator restart — I am leaving as a follow-up. The tool is available; AGENTS.md or the system prompt can document when to call it.
One question for you: should
POST /bottles/<id>/reprovision_gatewayrequire the control-plane token (trusted-caller only, always reached via the supervise server), or should it eventually be callable directly by agents over an unauthenticated gateway-proxied path? Keeping it control-plane-only is simpler and matches the existing trust model; I will go that way unless you say otherwise.@didericis-claude instead of a json, I wanted each individual key to be it's own row in the table. Forgot to add a
keycolumn indicating that. That makes it clearer what env vars were provided (which should mirror what's in the manifest), and it's good to have the labels visible for audit purposes.RE the prompting mechanism, by "prompt" I don't literally mean we need claude to call
/reprovision_gateway, I mean there should be some way we can trigger the running agent that has the env var to call the api. A dumb idea would be to start up a small process in the agent container that polls a file and then fires the event when we exec in there and write to it, but there's probably a better way to do that. And we should do that in this pr/not as a follow up. We do NOT want an mcp server for reprovisioning/that's unecessary.Got it on both points.
Schema: Will add a
keycolumn so each env var is its own row —(bottled_agent_id, key, value, type). Env var names stay plaintext for the audit trail; only the values are encrypted.Triggering reprovision: How about this — during provisioning we write a one-shot shell script into the agent container at a fixed path (e.g.,
/usr/local/bin/bb-reprovision). The script reads$ENV_VAR_SECRETfrom its own env and POSTs to a new plain HTTP (non-MCP) endpointPOST /reprovisionon the supervise server at$BOT_BOTTLE_SUPERVISE_URL, with the identity token as a header for attribution. The supervise server decrypts and calls the orchestrator.No background process, no polling, no MCP. The host-side reattach logic (or a new
reprovision-bottleCLI command) just runsdocker exec <container> /usr/local/bin/bb-reprovisionfor each bottle that lost its tokens. One exec, done.Does that work for you, or do you want the exec avoided entirely (e.g., trigger from a Unix socket or named pipe the container already has open)?
@didericis-claude does that exec have access to the env var? Was thinking we wouldn't be able to get the env var from
exec. If we can then we don't need the endpoint, we can simply callexec, read the env var into a variable, and then reprovision the gateway during gateway reboot.Yes —
docker execruns inside the container and inherits its full environment, sodocker exec <container> printenv ENV_VAR_SECRETworks fine.And you are right: that means no endpoint in the container and no script to provision. The host-side reattachment path (triggered on infra restart or
ensure_running) can do the whole thing itself:docker exec <container> printenv ENV_VAR_SECRETPOST /bottles/<id>/reprovision_gatewayon the orchestrator control plane (trusted caller, control-plane token) with the result_tokensClean — no sidecar, no supervise endpoint, no script. The supervise server stays out of it entirely. I will wire this into
OrchestratorService.ensure_running(or a dedicatedreprovision_allstep that runs after the infra container is healthy) so it happens automatically on every gateway reboot.Done — pushed as
01cee05.What landed:
bot_bottle/orchestrator/secret_store.py—new_env_var_secret(),encrypt_value(),decrypt_value()(HMAC-SHA256 CTR, stdlib-only)bottled_agent_secretstable (migrations v4+v5):(bottled_agent_id, key, value, type)— env-var name plaintext, value encryptedOrchestrator.launch_bottleacceptsenv_var_secretand writes one row per tokenPOST /bottles/<id>/reprovision_gatewayon the control planeENV_VAR_SECRETinjected into agent container via compose subprocess env (bare name — value never hits the compose file on disk)_reprovision_running_bottlesruns insidelaunch_consolidatedafterensure_running: for each registered bottle alive on the gateway network,docker exec … printenv ENV_VAR_SECRET+ reprovision call. No sidecar, no polling.All 111 existing unit tests pass.
350b1def0dto3b6f68d26dFound three merge-blocking issues (CI failures intentionally ignored):
[P0] Restart recovery does not work on the default macOS backend. The generated key is returned in
LaunchContext, but the macOS launch path never injectsctx.env_var_secretinto the container or its exec environment, and only the Docker launcher invokes_reprovision_running_bottles. After the host launcher exits, the sole key is gone; an infra recreation therefore still leaves existing macOS bottles without credentials—the exact motivating failure for this PR. The Firecracker backend has the same incomplete plumbing. Relevant locations:bot_bottle/backend/macos_container/consolidated_launch.py:149-156,bot_bottle/backend/docker/consolidated_launch.py:91-169, andbot_bottle/backend/firecracker/consolidated_launch.py:72-84.[P1] Stored secrets are never removed with their bottle.
teardown_bottle,reconcile, and failed-launch rollback remove the registry record and in-memory tokens but never call the newly addeddelete_agent_secrets. Since the secrets table has no cascading foreign key, ciphertext rows survive indefinitely, violating success criterion 5 and accumulating orphaned credentials. Seebot_bottle/orchestrator/service.py:117-155.[P1] The encryption format provides no authentication. The stored blob is only
nonce || CTR ciphertext; there is no MAC/tag. Database corruption or modification therefore changes decrypted credentials undetectably. A wrong key is also accepted whenever the resulting bytes happen to be valid UTF-8, after whichreprovision_from_secretreports success and installs garbage tokens. Please use authenticated encryption, or derive separate encryption/MAC keys and verify an encrypt-then-MAC tag before decrypting. Seebot_bottle/orchestrator/secret_store.py:53-91.The focused unit tests pass, but they do not exercise the missing backend integration or lifecycle cleanup.
@didericis-codex
finding 1: fix it for both macos and firecracker backends
finding 2: we actually want
teardown_bottleandreconcileto maintain records of agents, and simply update an events log with statuses. See conversation here. Cleaning up secrets was explicitly called out in here as a later step. Logic should be to remove the secrets once the bottled agent gets a status ofremoved. But we're ok to defer that.finding 3: too complicated, do not implement this
Acknowledged.
Finding 1 remains blocking: this should be completed for both macOS-container and Firecracker, including durable key delivery and an automatic backend-appropriate reprovision path after infra restart.
I withdraw finding 2 as a blocker given the intended retained-agent/event-log lifecycle and the explicit decision to defer deletion until an agent reaches
removed. The current PRD success criterion saying teardown/reap destroys stored secrets should be adjusted or marked as follow-up so the shipped behavior and requirements do not contradict each other.Agreed that ciphertext authentication is unnecessary complexity for this interim threat model. The requirement is confidentiality when the registry DB is disclosed without the per-bottle key; database corruption or an incorrect recovery key simply yields unusable upstream credentials and fails closed. Keep the existing
nonce || ciphertextformat without an authentication tag.I’ll re-review the updated head once those changes are pushed.
Re-reviewed at
74f79cd6. The prior backend-recovery blocker is resolved for Docker, macOS-container, and Firecracker; the deferred secret-deletion criterion is now aligned with the intended retained-record lifecycle. Focused recovery/orchestrator tests pass, and I found no new actionable issues.74f79cd690to26d0f5e3b2