bot-bottle/docs/prds/0012-stuck-agent-recovery-flow.md

PRD 0012: Stuck-agent recovery flow

• Status: Draft
• Author: didericis
• Created: 2026-05-24

Summary

When an agent running inside a claude-bottle container gets blocked, it invokes one of three MCP tool calls — cred-proxy-block, pipelock-block, or capability-block — passing a proposed config change (modified routes.json, modified pipelock allowlist, or modified agent Dockerfile) plus text describing why the change is justified. The supervisor sees the proposal in a host-side TUI, approves / modifies / rejects it, and the corresponding remediation runs: SIGHUP-reload cred-proxy with the new routes; restart pipelock with the new allowlist; rebuild the bottle from the new Dockerfile on the same branch. The agent's tool call blocks until the operator acts. The supervisor never opens a live channel into a running bottle; all signal flow goes through a per-bottle MCP sidecar on the existing internal network.

Problem

Running parallel agents in isolated bottles makes it cheap to spin up work in parallel, but expensive to recover when an agent gets stuck. Today, if a bottle is missing a permission or a tool the agent needs to make progress, the only options are to kill the container and start over (losing work) or open a live channel into the bottle to fix it in place (breaking the sandbox property that makes bottles trustworthy in the first place). The user feels this directly whenever a parallel run blocks on something the manifest didn't anticipate.

Goals / Success Criteria

A real stuck agent recovers end-to-end in each of the three categories: a cred-proxy block is fixed by the operator approving the agent's proposed routes.json, SIGHUP-reloading cred-proxy, and the tool returning "approved, retry now"; a pipelock block is fixed by the operator approving the proposed allowlist, restarting pipelock, and the tool returning "approved, retry now"; a capability block triggers a bottle rebuild from the proposed Dockerfile, with the replacement agent picking up on the same branch. All three complete without anyone running docker attach or opening any live channel into the original container.

Non-goals

• Live attach or in-place mutation of running containers. The whole design exists to avoid this.
• Agent-to-agent communication. Re-stated from the project's existing non-goals; the recovery flow is human→agent only.
• Auditing or forensic replay of agent runs. Git/forge history is the audit log; this PRD does not add a separate run log.
• Reducing time-to-unstuck below some target. Faster than kill-and-restart is implicit, but no specific SLO is in scope.

Scope

In scope

• A per-bottle MCP sidecar on the bottle's internal network that exposes three tools to the agent: cred-proxy-block, pipelock-block, capability-block. Each tool blocks until the operator acts.
• Tool input schemas: each tool takes the full proposed file (a new routes.json, a new pipelock allowlist, or a new Dockerfile, respectively) plus a justification text field. Tool output: {status: "approved"|"modified"|"rejected", notes: "..."}.
• A read-only mount at /etc/claude-bottle/current-config/ in the agent container that exposes the current routes.json, pipelock allowlist, and Dockerfile, so the agent can read them before composing a proposed change.
• SIGHUP-based hot reload of routes.json on cred-proxy, so an approved cred-proxy block proposal takes effect without restarting the proxy or dropping in-flight calls.
• A clean restart path on pipelock that picks up a new allowlist on container restart; v1 does not ship SIGHUP reload for pipelock (see Open questions).
• A TUI dashboard that lists running bottles and pending tool-call proposals. For each pending proposal: shows the diff (current → proposed) and the agent's justification. Operator actions: approve, modify-then-approve (edit the proposed diff before applying), reject with reason. Also exposes proactive operator-initiated edits: routes edit <bottle>, pipelock edit <bottle> — useful when the operator wants to change config without waiting for an agent prompt.
• Host-side audit logs at ~/.claude-bottle/audit/cred-proxy-<slug>.log and ~/.claude-bottle/audit/pipelock-<slug>.log. Each entry: timestamp, diff before/after, the agent's justification text (if the edit came from a tool call), and the operator's action (approve / modify / reject) with notes. Records config shape, never secret values.
• A rebuild path for the capability block case where the agent Dockerfile must change. Orchestrator tears down the bottle, applies the approved Dockerfile, and starts a replacement bottle on the same branch.
• A state-preservation helper for the rebuild path: working tree push is mandatory; transcript / reasoning context is best-effort.

Out of scope

• A tool-denial hook that auto-detects "stuck" without the agent's involvement. Deferred to a follow-up; v1 is opt-in via the agent calling one of the three MCP tools.
• A web dashboard. TUI only in v1.
• Live channel into running containers (see Non-goals).
• Agent-to-agent communication (see Non-goals).
• Auditing / forensic replay (see Non-goals).

Proposed Design

Stuck categories

Three named categories, each with its own MCP tool. Ordered by remediation cost:

• cred-proxy block. Tool: cred-proxy-block. The agent's request was refused by cred-proxy — missing route, expired token, wrong scope. The agent reads the current routes.json from /etc/claude-bottle/current-config/, composes a modified version, and calls the tool with {routes: <new file>, justification: "..."}. The operator reviews the diff in the TUI; on approval, the supervisor writes the new routes.json and cred-proxy SIGHUP-reloads. In-flight connections are not dropped. The tool returns {status: "approved", notes: "..."} and the agent retries.
• pipelock block. Tool: pipelock-block. The agent's outbound request was refused by pipelock — host not in the allowlist, protocol not permitted. The agent reads the current allowlist, composes a modified version, and calls the tool with {allowlist: <new file>, justification: "..."}. On approval, the supervisor writes the new allowlist and restarts pipelock; in-flight outbound calls may drop and rely on retry. The tool returns the same approve/reject shape.
• capability block. Tool: capability-block. The bottle is missing a tool, skill, permission, or env var the agent needs — something that lives in the agent Dockerfile rather than in routes or the pipelock allowlist. The agent reads the current Dockerfile, composes a modified version, and calls the tool with {dockerfile: <new file>, justification: "..."}. On approval, the rebuild orchestrator tears down the bottle, builds from the new Dockerfile, and starts a replacement bottle on the same branch via the state-preservation helper. Because the current agent is about to be replaced, the tool's return is best-effort — the replacement agent inherits the approval record via the preserved transcript.

The three tools are dispatched by name, so the operator's TUI knows which remediation engine to wire to which proposal. The agent must choose the right tool for what failed: a 403 from a credentialed request is a cred-proxy-block; a connection refused at the egress is a pipelock-block; a "command not found" or missing-skill error is a capability-block.

New services / components

• MCP sidecar. New per-bottle container on the bottle's internal network. Exposes the three tools (cred-proxy-block, pipelock-block, capability-block) to the agent. On a tool call: validates the proposed file syntactically (valid JSON for routes.json, parseable Dockerfile, etc.), persists the proposal to a host-mounted queue, and holds the tool-call connection open until the supervisor acts. On a response from the supervisor, returns {status, notes} to the agent. Whether this lives as its own container or as a mode of cred-proxy is an Open question; v1 plan is its own container to keep cred-proxy focused on credentials.
• cred-proxy-block MCP tool. Input: {routes: <full file>, justification: <text>}. Output: {status: "approved"|"modified"|"rejected", notes: <text>}. On approval, the supervisor writes the new routes.json and SIGHUPs cred-proxy.
• pipelock-block MCP tool. Input: {allowlist: <full file>, justification: <text>}. Output: same approve/reject shape. On approval, the supervisor writes the new allowlist and restarts pipelock.
• capability-block MCP tool. Input: {dockerfile: <full file>, justification: <text>}. Output: best-effort approval acknowledgement (the agent is about to be torn down). On approval, the rebuild orchestrator runs.
• Read-only current-config mount. /etc/claude-bottle/current-config/ in the agent container exposes routes.json, the pipelock allowlist, and the agent Dockerfile from the host. Read-only — the agent proposes changes via the tool call, never by writing the file directly.
• cred-proxy SIGHUP reload. New behavior on the existing process: SIGHUP re-reads routes.json without dropping connections or breaking in-flight calls. ~30 lines added to the server. Used by the cred-proxy block path.
• pipelock restart path. v1 uses restart, not SIGHUP: the supervisor writes the new allowlist and restarts the pipelock container. Agent's in-flight outbound calls drop and rely on retry. Used by the pipelock block path.
• TUI dashboard. A claude-bottle dashboard (or similarly named) command. Lists running bottles and pending tool-call proposals. For each proposal: shows current vs. proposed diff and the agent's justification; operator actions are approve, modify-then-approve, or reject-with-reason. Also exposes proactive operator-initiated edits: routes edit <bottle> and pipelock edit <bottle> — useful when the operator wants to change config without waiting for an agent prompt. Targets stdlib only; a TUI library is added only if the experience truly demands it.
• Config-edit audit logs. ~/.claude-bottle/audit/cred-proxy-<slug>.log and ~/.claude-bottle/audit/pipelock-<slug>.log. Every edit appends: timestamp, diff before/after, the agent's justification (if the edit came from a tool call), and the operator's action with notes. Records config shape, never the secret values themselves.
• Rebuild orchestrator (capability-block path). Used when the Dockerfile must change. On approval, tears down the existing bottle, builds from the new Dockerfile, snapshots state via the state-preservation helper, and starts a fresh bottle on the same branch.
• State-preservation helper. Mandatory: ensures the working tree is pushed before teardown. Best-effort: carries forward the agent's transcript / reasoning context — including the approved capability-block proposal — into the replacement container so the new agent starts warm rather than cold.

Existing code touched

• cred-proxy (PRD 0010) — gains SIGHUP reload of routes.json. The supervise-notify responsibility moves out of cred-proxy and into the new MCP sidecar, keeping cred-proxy focused on credentials.
• pipelock — gains a clean restart path that picks up the new allowlist on container restart. No code changes likely needed if pipelock already reads its config on startup; the orchestration is supervisor-side.
• cli.py — gains the dashboard subcommand (approve/modify/reject pending tool-call proposals; routes edit and pipelock edit for proactive operator-initiated changes) and the rebuild path.
• Bottle lifecycle scripts — extended to launch the MCP sidecar alongside the other sidecars; mount the read-only current-config directory into the agent container; orchestrate teardown + rebuild with state hand-off.
• Bottle manifest schema — may need to record the originating manifest version / change history per agent run, so the dashboard can show "what changed" rather than "what is."

Data model changes

• A per-bottle pending-proposal queue: the MCP sidecar holds the agent's open tool-call connection; the queue holds the proposal payload (id, bottle slug, tool name, current file hash, proposed file content, agent's justification text, arrival timestamp) the TUI needs to render the ask.
• Per-bottle config audit log files at ~/.claude-bottle/audit/cred-proxy-<slug>.log and ~/.claude-bottle/audit/pipelock-<slug>.log, append-only. Each entry includes the agent's justification and the operator's action.
• A per-agent-run record sufficient to map a running bottle back to its PR / branch, so the rebuild orchestrator knows where to post the comment and which branch to resume on.

External dependencies

• The Gitea API / tea CLI is already in the toolbox (the project is on Gitea); no new auth surface beyond what the orchestrator already needs to read/post on PRs.
• A TUI library is a maybe — only if stdlib can't carry the dashboard experience. Default to no new dependency.
• An MCP server library / framework. The MCP sidecar implements the MCP protocol so the agent can call the three tools natively. Pick the lightest option that lets the sidecar advertise three tools with structured input/output schemas; do not adopt a heavier MCP framework than the three tools justify.

Open questions

• Text-only vs. structured tools. Earlier drafts of this PRD used a text-only protocol (/supervise/notify returning {text}); this revision uses three structured MCP tools that carry the agent's proposed file. Structured wins on: richer triage signal (operator sees the diff up front, not just a description of it), cleaner audit (the agent's proposed shape is captured alongside the operator's action), and the agent does diff-authoring work the operator would otherwise have to do. Structured costs: larger wire surface, the agent has to know the file formats (routes.json schema, Dockerfile syntax, pipelock allowlist format), miscategorization is possible (e.g. a 403 the agent reads as a cred-proxy-block might actually be a pipelock issue at a different layer). Text-only wins on: smallest possible protocol, no schema burden on the agent, easy to extend (every new category is just another reason in prose). Text-only costs: operator does all the diff authoring, audit log loses the agent's proposed shape, no opportunity for the agent's understanding of the fix to be inspected. Worth re-litigating if the MCP sidecar grows complex relative to the value it produces.
• MCP sidecar placement. v1 plan is its own container. Alternative is folding the supervise plane back into cred-proxy as a second tool surface. Own container keeps separation clean; folded saves one sidecar per bottle. Worth deciding once the sidecar's actual line count is known.
• capability-block return semantics. The current agent is torn down on approval, so the tool's return value never reaches it. Options: (a) fire-and-forget, the tool returns immediately with "queued" and the agent halts; (b) block the tool, let the rebuild orchestrator's teardown kill the connection, replacement agent gets the approval record via state-preservation; (c) the tool blocks, returns "approved" right before teardown, the agent has milliseconds to log it. (b) seems cleanest but is worth confirming during implementation.
• SIGHUP race window. An agent that retries within msec of the SIGHUP may hit old routes once before the reload completes, fail, and retry against the new routes. Assumption is that normal HTTP retry semantics absorb this; worth confirming under real usage rather than designing around it preemptively.
• SIGHUP reload for pipelock. v1 ships restart-based reload, which drops in-flight outbound calls. Should pipelock gain SIGHUP support so pipelock block is as cheap as cred-proxy block? Depends on how often the operator edits the allowlist mid-task and how disruptive a pipelock bounce actually is.
• Multiple pending proposals from the same bottle. If the agent calls a second tool before the first is answered, what does the queue do — replace, append, or refuse? Append feels safest; replace is wrong (loses context); refuse forces the agent to handle a new error mode. Also: can different tools from the same bottle be pending simultaneously (e.g. a cred-proxy-block and a pipelock-block in flight at once)?
• Proposal validation strictness. The MCP sidecar validates the proposed file syntactically. Should it also do a deeper check — e.g. does the proposed routes.json introduce a route the operator already rejected this session? Probably no for v1; the operator is the gate.
• Best-effort transcript preservation on the rebuild path. Mount the agent's state directory, snapshot on teardown, remount in the replacement? How much fidelity is "good enough" for the new agent to pick up?
• Tool-denial auto-detection. Should v1 also ship a denial hook that auto-invokes one of the three tools without the agent's reasoning step, or strictly the agent-initiated form? Currently deferred; agent-initiated is safer (the agent has the most context about why it needed the call that was denied).
• Bottle → PR/branch mapping. Recorded at bottle-spawn time, derived from the working tree, or specified in the manifest?
• How does the flow handle one-off exceptions to gitlock / pipelock denials — e.g. a commit that includes docs with intentionally-bogus tokens that the secret scanner correctly flags? The shape (agent blocked → tool call → operator decides → result) is the same, but the resolution differs: a per-operation override or a scoped allowlist entry, not a routes edit, allowlist edit, or Dockerfile change. Is this a fourth tool (exception-block?), or does it fold into pipelock-block with a scoped one-shot allowlist entry? Either way, the approval must be auditable so a future reader can see what was waived and why. See docs/research/git-gate-commit-approval.md for a survey of gitleaks's native allowlist primitives and a recommendation.

References

• PRD 0010 — cred-proxy (gains SIGHUP reload of routes.json; the supervise plane lives in a separate MCP sidecar).
• CLAUDE.md — project non-goal on agent-to-agent communication; this PRD stays on the human→agent side of that line.