bot-bottle/docs/prds/0082-authoritative-failure-boundaries.md

PRD 0082: Authoritative failure boundaries

• Status: Draft
• Author: codex
• Created: 2026-07-27
• Issue: #444

Summary

Make every security- or lifecycle-sensitive snapshot distinguish authoritative empty state from unavailable state, and make every destructive or resource-consuming boundary revalidate the assumptions it acts on. This finishes the focused quality work begun under #444 without broad rewrites: cleanup cannot act on stale identities, policy introspection cannot publish a fabricated empty policy, gateway servers bound untrusted work, and daemon shutdown does not emit uncaught background-thread failures. Shared control-plane storage and gateway credential provisioning also enforce their filesystem security contract before sensitive data is written.

Problem

Several paths are individually fail-closed but compose into unsafe or misleading behavior:

1. cleanup prepares a plan, waits indefinitely for operator confirmation, then kills stored PIDs and removes stored paths without checking that those identities still describe the same orphan. A PID may be reused or a run directory may become active during the prompt.
2. The supervisor reuses egress's deny-all fallback for list-egress-routes. Deny-all is correct for enforcement, but presenting it as a successful empty route table can cause a later replace-all proposal to discard live routes.
3. The supervisor and Git HTTP services accept bounded declared body sizes but use blocking reads and unbounded request threads. An untrusted bottle can exhaust the shared gateway with slow or parallel requests.
4. macOS cleanup enumerates containers and networks independently and treats a failed query as an empty class, so a partial snapshot can still become a destructive plan.
5. Gateway log-pump threads race stream closure during shutdown and emit uncaught exceptions even when shutdown otherwise succeeds.
6. Firecracker discovers VMs through whitespace-split pgrep -a output. A configured cache path containing spaces can hide a live VM from the snapshot and make its run directory appear orphaned.
7. Docker cleanup asks compose for its project snapshot in best-effort mode. A transient query failure can therefore become an empty stopped-project set and authorize deletion of associated state directories.
8. Firecracker artifact downloads and registry publication have no network deadline, so an unresponsive registry can hold setup or release work indefinitely.
9. Authenticated secret blobs select the unauthenticated legacy decoder when their in-band version prefix is changed, allowing storage tampering to bypass tag verification.
10. Cleanup executes the entire post-confirmation snapshot rather than the intersection with what the operator saw, and mutation failures are not reflected in the command result.
11. Git smart-HTTP can retain sixteen 100 MiB request bodies concurrently, cleanup mutations have no subprocess deadline, and Firecracker signalling failures bypass shared mutation accounting.
12. SQLite creates the shared control-plane database before its mode is restricted, then suppresses permission-repair failures. Gateway transports also differ in whether copied deploy-key modes are preserved.

These are one design problem: state used to authorize deletion, replacement, or resource allocation must be authoritative at the point of use.

Goals / Success Criteria

• Cleanup never signals a PID or recursively deletes a path solely because it appeared in a pre-confirmation snapshot.
• Firecracker cleanup proves immediately before action that a PID is still the same Firecracker process and that a run directory is still orphaned.
• Firecracker process discovery reads NUL-delimited argv from /proc; paths are never reconstructed from whitespace-delimited process listings.
• All backend cleanup discovery primitives raise a typed enumeration error on operational failure. No backend may independently continue from a partial snapshot.
• Shared cleanup control flow lives in the backend layer; concrete backends override resource-specific discovery and validation primitives rather than each implementing a bespoke failure policy.
• list-egress-routes returns an MCP error when attribution or policy resolution is unavailable. A genuine, authoritatively resolved empty policy remains a successful empty list.
• Supervisor and Git HTTP request bodies have total read deadlines, and each service bounds concurrent request work. Limits apply to authenticated callers because bottles themselves are untrusted.
• Gateway child-output pumping treats expected stream closure during shutdown as completion while preserving diagnostics for unexpected failures.
• Artifact pull, existence-check, and publication requests use explicit network deadlines.
• Persisted secrets accept only the authenticated format. The schema migration intentionally clears legacy rows; local agents are reprovisioned rather than retaining a ciphertext-controlled downgrade path.
• Cleanup executes only resources present in both the displayed and current authoritative plans, attempts every approved mutation, and returns failure when any mutation does not complete.
• Git request bodies spool to disk behind a separate heavy-work semaphore; cleanup commands have configurable deadlines; Firecracker signalling failures aggregate while identity-verification uncertainty still aborts.
• The shared database directory and file are private before SQLite writes any control-plane state; an inability to enforce those modes aborts startup.
• Gateway credential directories and files receive explicit private modes inside the gateway, independent of Docker, Apple Container, or SSH copy semantics.
• Unit tests cover PID/path reuse, partial backend enumeration, transient policy resolution failure, slow bodies, concurrency saturation, and stream closure races.

Non-goals

• Further decomposition solely to reduce module line counts.
• Replacing gateway stdlib HTTP services with a web framework.
• Changing egress matching, DLP decisions, proposal semantics, or backend launch behavior beyond the synchronization required for safe cleanup.
• Making cleanup silently skip uncertain resources. Uncertainty is an operator-visible failure.

Design

Shared backend control flow

Follow the backend architecture rule used by gateway attachment: shared behavior lives above concrete backends; subclasses provide primitives, not control flow.

Cleanup remains previewable, but confirmation authorizes a new authoritative evaluation, not blind execution of the displayed object. The shared flow:

1. asks each available backend for a preview;
2. displays the union and asks for confirmation;
3. refreshes each non-empty backend plan;
4. validates destructive identities immediately before action;
5. aborts loudly if the refreshed plan or any identity cannot be proven safe.

Backend-specific primitives define how to identify a resource. Firecracker uses process start identity plus canonical config/run paths; container backends use authoritative CLI queries and stable resource names/labels.

Container engines expose destructive name-based commands without a portable compare-and-delete operation. Cleanup therefore refreshes after confirmation and requires every discovery query to succeed, minimizing but not claiming to eliminate the final name-reuse race. A future engine-specific stable-ID primitive may close that residual window without moving control flow back into each backend.

Enforcement state versus introspection state

Egress enforcement retains its deny-all fallback because uncertainty must not grant network access. Supervisor introspection uses a strict resolver path: unattributed callers and resolver failures become typed MCP errors, while a successfully resolved policy containing zero routes returns routes: [].

Gateway resource boundaries

Both stdlib servers set a per-connection body deadline before reading and use a bounded request executor or semaphore. Saturated capacity fails quickly with a service-unavailable response. Existing size caps remain independent: supervisor proposals retain the 1 MiB cap and Git pack requests retain their larger protocol-appropriate cap.

Shutdown diagnostics

The gateway output pump catches only stream-closure exceptions expected after the supervisor closes child pipes. Other I/O failures remain visible and are reported through the supervisor's normal diagnostic channel.

Shared filesystem security

The common SQLite store owns database creation for every backend. It creates the parent directory and an empty database with private modes before opening SQLite, repairs existing modes, verifies the resulting state, and propagates every enforcement failure. Backend launchers do not duplicate this policy.

The backend-neutral gateway provisioner likewise applies directory and file modes after transport copies complete. This avoids relying on copy behavior that differs among Docker, Apple Container, and Firecracker's SSH transport.

Implementation chunks

1. Existing fail-closed security and backend enumeration fixes.
2. FastAPI orchestrator transport and bounded control-plane bodies.
3. Egress request-policy and outbound-DLP pipeline extraction.
4. Supervisor MCP dispatch extraction.
5. Shared cleanup refresh/revalidation plus authoritative macOS discovery.
6. Strict supervisor introspection and bounded supervisor/Git HTTP work.
7. Gateway shutdown log-pump closure handling.
8. Lossless Firecracker process identities, authoritative Docker cleanup queries, and bounded Firecracker artifact transfers.
9. Mandatory authenticated secret storage, shared cleanup-plan intersection and mutation accounting, and contained Git backend process failures.
10. Disk-spooled and separately bounded Git bodies, cleanup command deadlines, and classified Firecracker signalling failures.
11. Fail-closed shared database creation and backend-neutral gateway credential permissions.

Open questions

None.