bot-bottle/docs/prds/0082-authoritative-failure-boundaries.md

# PRD 0082: Authoritative failure boundaries

- **Status:** Draft
- **Author:** codex
- **Created:** 2026-07-27
- **Issue:** #444

## Summary

Make every security- or lifecycle-sensitive snapshot distinguish authoritative
empty state from unavailable state, and make every destructive or
resource-consuming boundary revalidate the assumptions it acts on. This
finishes the focused quality work begun under #444 without broad rewrites:
cleanup cannot act on stale identities, policy introspection cannot publish a
fabricated empty policy, gateway servers bound untrusted work, and daemon
shutdown does not emit uncaught background-thread failures.

## Problem

Several paths are individually fail-closed but compose into unsafe or
misleading behavior:

1. `cleanup` prepares a plan, waits indefinitely for operator confirmation,
   then kills stored PIDs and removes stored paths without checking that those
   identities still describe the same orphan. A PID may be reused or a run
   directory may become active during the prompt.
2. The supervisor reuses egress's deny-all fallback for
   `list-egress-routes`. Deny-all is correct for enforcement, but presenting it
   as a successful empty route table can cause a later replace-all proposal to
   discard live routes.
3. The supervisor and Git HTTP services accept bounded declared body sizes but
   use blocking reads and unbounded request threads. An untrusted bottle can
   exhaust the shared gateway with slow or parallel requests.
4. macOS cleanup enumerates containers and networks independently and treats a
   failed query as an empty class, so a partial snapshot can still become a
   destructive plan.
5. Gateway log-pump threads race stream closure during shutdown and emit
   uncaught exceptions even when shutdown otherwise succeeds.
6. Firecracker discovers VMs through whitespace-split `pgrep -a` output.
   A configured cache path containing spaces can hide a live VM from the
   snapshot and make its run directory appear orphaned.
7. Docker cleanup asks compose for its project snapshot in best-effort mode.
   A transient query failure can therefore become an empty stopped-project
   set and authorize deletion of associated state directories.
8. Firecracker artifact downloads and registry publication have no network
   deadline, so an unresponsive registry can hold setup or release work
   indefinitely.
9. Authenticated secret blobs select the unauthenticated legacy decoder when
   their in-band version prefix is changed, allowing storage tampering to
   bypass tag verification.
10. Cleanup executes the entire post-confirmation snapshot rather than the
    intersection with what the operator saw, and mutation failures are not
    reflected in the command result.

These are one design problem: state used to authorize deletion, replacement,
or resource allocation must be authoritative at the point of use.

## Goals / Success Criteria

- Cleanup never signals a PID or recursively deletes a path solely because it
  appeared in a pre-confirmation snapshot.
- Firecracker cleanup proves immediately before action that a PID is still the
  same Firecracker process and that a run directory is still orphaned.
- Firecracker process discovery reads NUL-delimited argv from `/proc`; paths
  are never reconstructed from whitespace-delimited process listings.
- All backend cleanup discovery primitives raise a typed enumeration error on
  operational failure. No backend may independently continue from a partial
  snapshot.
- Shared cleanup control flow lives in the backend layer; concrete backends
  override resource-specific discovery and validation primitives rather than
  each implementing a bespoke failure policy.
- `list-egress-routes` returns an MCP error when attribution or policy
  resolution is unavailable. A genuine, authoritatively resolved empty policy
  remains a successful empty list.
- Supervisor and Git HTTP request bodies have total read deadlines, and each
  service bounds concurrent request work. Limits apply to authenticated
  callers because bottles themselves are untrusted.
- Gateway child-output pumping treats expected stream closure during shutdown
  as completion while preserving diagnostics for unexpected failures.
- Artifact pull, existence-check, and publication requests use explicit
  network deadlines.
- Persisted secrets accept only the authenticated format. The schema migration
  intentionally clears legacy rows; local agents are reprovisioned rather
  than retaining a ciphertext-controlled downgrade path.
- Cleanup executes only resources present in both the displayed and current
  authoritative plans, attempts every approved mutation, and returns failure
  when any mutation does not complete.
- Unit tests cover PID/path reuse, partial backend enumeration, transient
  policy resolution failure, slow bodies, concurrency saturation, and stream
  closure races.

## Non-goals

- Further decomposition solely to reduce module line counts.
- Replacing gateway stdlib HTTP services with a web framework.
- Changing egress matching, DLP decisions, proposal semantics, or backend
  launch behavior beyond the synchronization required for safe cleanup.
- Making cleanup silently skip uncertain resources. Uncertainty is an
  operator-visible failure.

## Design

### Shared backend control flow

Follow the backend architecture rule used by gateway attachment: shared
behavior lives above concrete backends; subclasses provide primitives, not
control flow.

Cleanup remains previewable, but confirmation authorizes a *new authoritative
evaluation*, not blind execution of the displayed object. The shared flow:

1. asks each available backend for a preview;
2. displays the union and asks for confirmation;
3. refreshes each non-empty backend plan;
4. validates destructive identities immediately before action;
5. aborts loudly if the refreshed plan or any identity cannot be proven safe.

Backend-specific primitives define how to identify a resource. Firecracker
uses process start identity plus canonical config/run paths; container
backends use authoritative CLI queries and stable resource names/labels.

Container engines expose destructive name-based commands without a portable
compare-and-delete operation. Cleanup therefore refreshes after confirmation
and requires every discovery query to succeed, minimizing but not claiming to
eliminate the final name-reuse race. A future engine-specific stable-ID
primitive may close that residual window without moving control flow back
into each backend.

### Enforcement state versus introspection state

Egress enforcement retains its deny-all fallback because uncertainty must not
grant network access. Supervisor introspection uses a strict resolver path:
unattributed callers and resolver failures become typed MCP errors, while a
successfully resolved policy containing zero routes returns `routes: []`.

### Gateway resource boundaries

Both stdlib servers set a per-connection body deadline before reading and use a
bounded request executor or semaphore. Saturated capacity fails quickly with a
service-unavailable response. Existing size caps remain independent:
supervisor proposals retain the 1 MiB cap and Git pack requests retain their
larger protocol-appropriate cap.

### Shutdown diagnostics

The gateway output pump catches only stream-closure exceptions expected after
the supervisor closes child pipes. Other I/O failures remain visible and are
reported through the supervisor's normal diagnostic channel.

## Implementation chunks

1. Existing fail-closed security and backend enumeration fixes.
2. FastAPI orchestrator transport and bounded control-plane bodies.
3. Egress request-policy and outbound-DLP pipeline extraction.
4. Supervisor MCP dispatch extraction.
5. Shared cleanup refresh/revalidation plus authoritative macOS discovery.
6. Strict supervisor introspection and bounded supervisor/Git HTTP work.
7. Gateway shutdown log-pump closure handling.
8. Lossless Firecracker process identities, authoritative Docker cleanup
   queries, and bounded Firecracker artifact transfers.
9. Mandatory authenticated secret storage, shared cleanup-plan intersection
   and mutation accounting, and contained Git backend process failures.

## Open questions

None.