bot-bottle/docs/prds/0012-stuck-agent-recovery-flow.md

# PRD 0012: Stuck-agent recovery flow

- **Status:** Draft
- **Author:** didericis
- **Created:** 2026-05-24

## Summary

When an agent running inside a claude-bottle container gets blocked, it signals via the per-bottle cred-proxy sidecar's `/supervise/notify` endpoint. The supervisor sees the message in a host-side TUI and responds with one of four shapes: a text hint (no infrastructure change, the agent continues); a cred-proxy routes edit (SIGHUP-reload of cred-proxy, agent retries); a pipelock allowlist edit (restart pipelock, agent retries); or an approved manifest diff that triggers a full rebuild of the bottle on the same branch. These map to three categories of stuck — **cred-proxy block**, **pipelock block**, and **capability gap** — described below. The supervisor never opens a live channel into a running bottle; all signal flow goes through the existing internal-network endpoint that cred-proxy already terminates.

## Problem

Running parallel agents in isolated bottles makes it cheap to spin up work in parallel, but expensive to recover when an agent gets stuck. Today, if a bottle is missing a permission or a tool the agent needs to make progress, the only options are to kill the container and start over (losing work) or open a live channel into the bottle to fix it in place (breaking the sandbox property that makes bottles trustworthy in the first place). The user feels this directly whenever a parallel run blocks on something the manifest didn't anticipate.

## Goals / Success Criteria

A real stuck agent recovers end-to-end in each of the three categories: a **cred-proxy block** is fixed by a `routes edit` + SIGHUP and a "retry now" reply without restarting anything; a **pipelock block** is fixed by an allowlist edit + pipelock restart and a "retry now" reply; a **capability gap** triggers a manifest-diff approval and a bottle rebuild that picks up on the same branch. All three complete without anyone running `docker attach` or opening any live channel into the original container.

## Non-goals

- Live attach or in-place mutation of running containers. The whole design exists to avoid this.
- Agent-to-agent communication. Re-stated from the project's existing non-goals; the recovery flow is human→agent only.
- Auditing or forensic replay of agent runs. Git/forge history is the audit log; this PRD does not add a separate run log.
- Reducing time-to-unstuck below some target. Faster than kill-and-restart is implicit, but no specific SLO is in scope.

## Scope

### In scope

- A `/stuck` slash command the agent invokes when blocked. POSTs free-text to cred-proxy's `/supervise/notify` and blocks awaiting a text reply.
- A `/supervise/notify` endpoint on cred-proxy that persists the agent's message host-side and holds the agent's connection open until the supervisor responds. Wire protocol is text-only: request is the agent's message; response is `{text: "..."}`.
- SIGHUP-based hot reload of `routes.json` on cred-proxy, so the supervisor can change the agent's credential surface without restarting the proxy or dropping in-flight calls.
- A TUI dashboard that lists running bottles and pending stuck-notifications. Two verbs:
  - `r <id> <text>` — reply to a pending stuck message (delivers text to the waiting agent).
  - `routes edit <bottle>` — open the bottle's `routes.json` in `$EDITOR`, SIGHUP cred-proxy on save. Not gated on a pending message; the supervisor can edit routes anytime.
- Equivalent support for pipelock: a `pipelock edit <bottle>` TUI verb that opens pipelock's allowlist in `$EDITOR` and restarts pipelock on save. (v1 uses restart, not SIGHUP — see Open questions.)
- Host-side audit logs at `~/.claude-bottle/audit/cred-proxy-<slug>.log` and `~/.claude-bottle/audit/pipelock-<slug>.log` that record every config edit: timestamp, diff before/after, the operator's reply text if the edit was tied to a `/stuck` reply. Records config shape, never secret values.
- A rebuild path for the **capability gap** case where the bottle's *manifest* (not just routes or pipelock allowlist) must change. Orchestrator tears down the bottle, applies the approved manifest diff, and starts a replacement bottle on the same branch.
- A state-preservation helper for the rebuild path: working tree push is mandatory; transcript / reasoning context is best-effort.

### Out of scope

- A tool-denial hook that auto-detects "stuck" without the agent's involvement. Deferred to a follow-up; v1 is opt-in via the slash command.
- A web dashboard. TUI only in v1.
- Live channel into running containers (see Non-goals).
- Agent-to-agent communication (see Non-goals).
- Auditing / forensic replay (see Non-goals).

## Proposed Design

### Stuck categories

Three named categories, ordered by remediation cost:

- **cred-proxy block.** The agent's request was refused by cred-proxy — missing route, expired token, wrong scope. The bottle is otherwise healthy. *Remediation:* operator runs `routes edit <bottle>`, edits `routes.json`, saves. cred-proxy SIGHUP-reloads; in-flight connections are not dropped. Operator replies to the `/stuck` message with a "retry now" hint. The agent retries against the (now-reloaded) cred-proxy and proceeds.
- **pipelock block.** The agent's outbound request was refused by pipelock — host not in the allowlist, protocol not permitted, etc. The bottle is otherwise healthy, but the egress perimeter is wrong. *Remediation:* operator runs `pipelock edit <bottle>`, edits the allowlist, saves. pipelock restarts; the agent's in-flight outbound calls may drop and need retry. Operator replies to the `/stuck` message with a "retry now" hint. (v1 uses restart; SIGHUP reload for pipelock is an Open question.)
- **capability gap.** The bottle is missing something the agent needs that lives in the manifest itself — a tool, a skill, a permission grant, an env var. Routes and pipelock are correct; the agent container just doesn't have the capability. *Remediation:* operator approves a manifest diff in the TUI. The rebuild orchestrator tears down the bottle, applies the diff, and starts a replacement bottle on the same branch via the state-preservation helper. The replacement agent picks up where the original was, now with the missing capability.

The wire protocol does not change between categories: the agent POSTs free text to `/supervise/notify` and receives `{text: "..."}`. The category is the operator's mental model for triage, not a field on the request. The agent does not need to know which category its message will fall into.

### New services / components

- **`/stuck` slash command.** Shipped as a skill mounted into bottles. POSTs the agent's free-text message to cred-proxy's `/supervise/notify` and blocks awaiting a text reply. Reply text is handed back to the agent verbatim — the agent doesn't need to know whether the supervisor edited routes, opened an editor, or did anything else before composing the reply.
- **cred-proxy `/supervise/notify` endpoint.** Receives the agent's message, persists it to a host-mounted queue, and holds the agent's connection open until the supervisor responds. The wire protocol is text-only in both directions; the supervisor's side-effects (routes edit, manifest diff, no-op) are invisible to the agent.
- **cred-proxy SIGHUP reload.** New behavior on the existing process: SIGHUP re-reads `routes.json` without dropping connections or breaking in-flight calls. ~30 lines added to the server. Used by the **cred-proxy block** category.
- **pipelock edit + restart.** v1 ships restart-based reload for pipelock: on `pipelock edit <bottle>` save, the supervisor writes the new allowlist and restarts the pipelock container. The agent's in-flight outbound calls drop and rely on retry. Used by the **pipelock block** category.
- **TUI dashboard.** A `claude-bottle dashboard` (or similarly named) command. Lists running bottles, surfaces pending stuck-notifications, exposes the `r <id> <text>`, `routes edit <bottle>`, and `pipelock edit <bottle>` verbs, and (for the **capability gap** category) shows proposed manifest diffs with approve/reject input. Targets stdlib only; a TUI library is added only if the experience truly demands it.
- **Config-edit audit logs.** `~/.claude-bottle/audit/cred-proxy-<slug>.log` and `~/.claude-bottle/audit/pipelock-<slug>.log`. Every edit appends: timestamp, diff before/after, operator's reply text if tied to a `/stuck` reply. Records what the bottle's credential surface and egress perimeter looked like at time T without storing secret values.
- **Rebuild orchestrator (capability-gap path).** Used when the manifest itself must change, not just routes or the pipelock allowlist. On approval, tears down the existing bottle, applies the approved manifest diff, snapshots state via the state-preservation helper, and starts a fresh bottle on the same branch.
- **State-preservation helper.** Mandatory: ensures the working tree is pushed before teardown. Best-effort: carries forward the agent's transcript / reasoning context into the replacement container so the new agent starts warm rather than cold.

### Existing code touched

- **cred-proxy** (PRD 0010) — gains the `/supervise/notify` endpoint, the host-mounted notification queue, and SIGHUP reload of `routes.json`.
- **pipelock** — gains a clean restart path that picks up the new allowlist on container restart. No code changes likely needed if pipelock already reads its config on startup; the orchestration is supervisor-side.
- **`cli.py`** — gains the dashboard subcommand (with `r`, `routes edit`, and `pipelock edit` verbs) and the rebuild path.
- **Bottle lifecycle scripts** — extended for orchestrated teardown + rebuild with state hand-off, distinct from a fresh-spawn.
- **Bottle manifest schema** — may need to record the originating manifest version / change history per agent run, so the dashboard can show "what changed" rather than "what is."

### Data model changes

- A per-bottle pending-notification queue: cred-proxy holds the agent's open connection; the queue holds the metadata (id, bottle slug, message body, arrival timestamp) the TUI needs to render the ask.
- Per-bottle config audit log files at `~/.claude-bottle/audit/cred-proxy-<slug>.log` and `~/.claude-bottle/audit/pipelock-<slug>.log`, append-only.
- A per-agent-run record sufficient to map a running bottle back to its PR / branch, so the rebuild orchestrator knows where to post the comment and which branch to resume on.

### External dependencies

- The Gitea API / `tea` CLI is already in the toolbox (the project is on Gitea); no new auth surface beyond what the orchestrator already needs to read/post on PRs.
- A TUI library is a *maybe* — only if stdlib can't carry the dashboard experience. Default to no new dependency.

## Open questions

- SIGHUP race window. An agent that retries within msec of the SIGHUP may hit old routes once before the reload completes, fail, and retry against the new routes. Assumption is that normal HTTP retry semantics absorb this; worth confirming under real usage rather than designing around it preemptively.
- SIGHUP reload for pipelock. v1 ships restart-based reload, which drops in-flight outbound calls. Should pipelock gain SIGHUP support so **pipelock block** is as cheap as **cred-proxy block**? Depends on how often the operator edits the allowlist mid-task and how disruptive a pipelock bounce actually is.
- Multiple pending notifications from the same bottle. If the agent calls `/stuck` again before the prior message is answered, what does the queue do — replace, append, or refuse? Append feels safest; replace is wrong (loses context); refuse forces the agent to handle a new error mode.
- Verb naming under load. `r <id> <text>` optimizes for muscle memory mid-incident; `reply <id> <text>` reads better cold. Worth picking once and committing.
- Best-effort transcript preservation on the rebuild path. Mount the agent's state directory, snapshot on teardown, remount in the replacement? How much fidelity is "good enough" for the new agent to pick up?
- Tool-denial auto-detection. Should v1 also ship a tool-denial hook that auto-invokes `/stuck` without the agent's involvement, or strictly the agent-initiated form? Currently deferred; line worth confirming during implementation.
- Rejection semantics on the rebuild path. Does the agent receive a `/stuck` reply explaining the rejection, or does the bottle just stay torn down?
- Bottle → PR/branch mapping. Recorded at bottle-spawn time, derived from the working tree, or specified in the manifest?
- How does the flow handle one-off exceptions to gitlock / pipelock denials — e.g. a commit that includes docs with intentionally-bogus tokens that the secret scanner correctly flags? The shape (agent blocked → `/stuck` → operator decides → reply) is the same, but the *resolution* differs: a per-operation override or a scoped allowlist entry, not a routes edit or a manifest change. Does the operator express the exception by commit SHA, by content hash, or by a narrow allowlist rule? Either way, the approval must be auditable so a future reader can see what was waived and why. See `docs/research/git-gate-commit-approval.md` for a survey of gitleaks's native allowlist primitives and a recommendation.

## References

- PRD 0010 — cred-proxy (the endpoint extended to carry stuck-requests).
- `CLAUDE.md` — project non-goal on agent-to-agent communication; this PRD stays on the human→agent side of that line.