bot-bottle/docs/prds/0012-stuck-agent-recovery-flow.md

PRD 0012: Stuck-agent recovery flow

• Status: Active
• Author: didericis
• Created: 2026-05-24

Summary

When an agent running inside a bot-bottle container gets blocked, it invokes one of three MCP tool calls — cred-proxy-block, pipelock-block, or capability-block — passing a proposed config change (modified routes.json, modified pipelock allowlist, or modified agent Dockerfile) plus text describing why the change is justified. The supervisor sees the proposal in a host-side TUI, approves / modifies / rejects it, and the corresponding remediation runs: SIGHUP-reload cred-proxy with the new routes; restart pipelock with the new allowlist; rebuild the bottle from the new Dockerfile on the same branch. The agent's tool call blocks until the operator acts. The supervisor never opens a live channel into a running bottle; all signal flow goes through a per-bottle MCP sidecar on the existing internal network.

This PRD is the overview. Implementation is split across four follow-on PRDs (0013–0016); see Implementation chunks below.

Problem

Running parallel agents in isolated bottles makes it cheap to spin up work in parallel, but expensive to recover when an agent gets stuck. Today, if a bottle is missing a permission or a tool the agent needs to make progress, the only options are to kill the container and start over (losing work) or open a live channel into the bottle to fix it in place (breaking the sandbox property that makes bottles trustworthy in the first place). The user feels this directly whenever a parallel run blocks on something the manifest didn't anticipate.

Goals / Success Criteria

A real stuck agent recovers end-to-end in each of the three categories: a cred-proxy block is fixed by the operator approving the agent's proposed routes.json, SIGHUP-reloading cred-proxy, and the tool returning "approved, retry now"; a pipelock block is fixed by the operator approving the proposed allowlist, restarting pipelock, and the tool returning "approved, retry now"; a capability block triggers a bottle rebuild from the proposed Dockerfile, with the replacement agent picking up on the same branch. All three complete without anyone running docker attach or opening any live channel into the original container.

Non-goals

• Live attach or in-place mutation of running containers. The whole design exists to avoid this.
• Agent-to-agent communication. Re-stated from the project's existing non-goals; the recovery flow is human→agent only.
• Auditing or forensic replay of agent runs. Git/Gitea history is the audit log; this PRD does not add a separate run log.
• Reducing time-to-unstuck below some target. Faster than kill-and-restart is implicit, but no specific SLO is in scope.

Stuck categories

Three named categories, each with its own MCP tool. Ordered by remediation cost:

• cred-proxy block. Tool: cred-proxy-block. The agent's request was refused by cred-proxy — missing route, expired token, wrong scope. The agent reads the current routes.json from /etc/bot-bottle/current-config/, composes a modified version, and calls the tool with {routes: <new file>, justification: "..."}. The operator reviews the diff in the TUI; on approval, the supervisor writes the new routes.json and cred-proxy SIGHUP-reloads. In-flight connections are not dropped. The tool returns {status: "approved", notes: "..."} and the agent retries. Implementation: PRD 0014.
• pipelock block. Tool: pipelock-block. The agent's outbound request was refused by pipelock — host not in the allowlist, protocol not permitted. The agent reads the current allowlist, composes a modified version, and calls the tool with {allowlist: <new file>, justification: "..."}. On approval, the supervisor writes the new allowlist and restarts pipelock; in-flight outbound calls may drop and rely on retry. The tool returns the same approve/reject shape. Implementation: PRD 0015.
• capability block. Tool: capability-block. The bottle is missing a tool, skill, permission, or env var the agent needs — something that lives in the agent Dockerfile rather than in routes or the pipelock allowlist. The agent reads the current Dockerfile, composes a modified version, and calls the tool with {dockerfile: <new file>, justification: "..."}. On approval, the rebuild orchestrator tears down the bottle, builds from the new Dockerfile, and starts a replacement bottle on the same branch via the state-preservation helper. Because the current agent is about to be replaced, the tool's return is best-effort — the replacement agent inherits the approval record via the preserved transcript. Implementation: PRD 0016.

The three tools are dispatched by name, so the operator's TUI knows which remediation engine to wire to which proposal. The agent must choose the right tool for what failed: a 403 from a credentialed request is a cred-proxy-block; a connection refused at the egress is a pipelock-block; a "command not found" or missing-skill error is a capability-block.

Why the MCP server is a sidecar, not in-container

The MCP server could in principle run inside the agent container. It doesn't, for reasons that are individually soft but together argue for the sidecar shape.

The security boundary is the operator approving in the TUI plus the host-side config write, neither of which depends on where the MCP server runs. Even if a compromised agent forged a proposal end-to-end, a human still has to approve before cred-proxy, pipelock, or the Dockerfile change. So in-container isn't a hard violation of the bottle wall.

It's still the wrong placement for five reasons:

1. Pattern consistency. cred-proxy and pipelock are sidecars because boundary services don't live in the agent's process space. An in-container MCP server would be the first exception, and the next reader has to ask why this one is different.
2. Audit metadata trust. Sidecar-generated timestamps and bottle metadata are trusted; in-container metadata is whatever the agent wrote. Same outcome (the operator is still the gate) but weaker provenance.
3. Connection lifecycle. A sidecar holds the tool-call connection independently of the agent process — agent crash or restart doesn't orphan a pending operator response.
4. Future enforcement headroom. If the MCP server ever needs to enforce something (rate limits, dedup, schema-strict rejection), it has to be a trusted process. Building it in-container now means re-architecting later.
5. Pipelock cleanliness. Sidecar-on-internal-network is the same egress shape pipelock already permits. In-container would need a loopback exception in the allowlist.

Implementation chunks

• PRD 0013 — Supervise plane foundation. MCP sidecar shell, three tool definitions, proposal queue, read-only current-config mount, minimal TUI, audit log format. After 0013, an operator can see proposals and approve/reject them but no remediation actually runs (the approval handlers are no-ops).
• PRD 0014 — cred-proxy block remediation. cred-proxy SIGHUP reload, host-side write on approval, routes edit <bottle> TUI verb, cred-proxy audit log filled in. First end-to-end useful category.
• PRD 0015 — pipelock block remediation. pipelock restart wiring, host-side write on approval, pipelock edit <bottle> TUI verb, pipelock audit log filled in. Same shape as 0014 for a different sidecar.
• PRD 0016 — capability block remediation. Rebuild orchestrator, state-preservation helper, capability-block end-to-end wiring, bottle-lifecycle changes for orchestrated teardown + rebuild. Heaviest chunk, lands last.

0013 is a hard prerequisite for 0014–0016. The other three can in principle ship in any order, but the recommended sequence is cheapest-blast-radius first (0014 → 0015 → 0016) so cheaper wins land while the rebuild path is being designed.

Open questions

• Text-only vs. structured tools. An earlier draft of this PRD used a text-only protocol (/supervise/notify returning {text}); this revision uses three structured MCP tools that carry the agent's proposed file. Structured wins on: richer triage signal (operator sees the diff up front, not just a description of it), cleaner audit (the agent's proposed shape is captured alongside the operator's action), and the agent does diff-authoring work the operator would otherwise have to do. Structured costs: larger wire surface, the agent has to know the file formats (routes.json schema, Dockerfile syntax, pipelock allowlist format), miscategorization is possible (e.g. a 403 the agent reads as a cred-proxy-block might actually be a pipelock issue at a different layer). Text-only wins on: smallest possible protocol, no schema burden on the agent, easy to extend (every new category is just another reason in prose). Text-only costs: operator does all the diff authoring, audit log loses the agent's proposed shape, no opportunity for the agent's understanding of the fix to be inspected. Worth re-litigating if the MCP sidecar grows complex relative to the value it produces.
• Tool-denial auto-detection. Should v1 also ship a denial hook that auto-invokes one of the three tools without the agent's reasoning step, or strictly the agent-initiated form? Currently deferred; agent-initiated is safer (the agent has the most context about why it needed the call that was denied).

References

• PRD 0010 — cred-proxy (gains SIGHUP reload of routes.json in 0014).
• PRD 0013 — supervise plane foundation.
• PRD 0014 — cred-proxy block remediation.
• PRD 0015 — pipelock block remediation.
• PRD 0016 — capability block remediation.
• AGENTS.md — project non-goal on agent-to-agent communication; this PRD stays on the human→agent side of that line.