docs/prds/prd-new-consolidate-docker-infra-backend.md

@@ -0,0 +1,155 @@
+# PRD prd-new: Consolidate infra backend for Docker
+
+- **Status:** Draft
+- **Author:** Claude
+- **Created:** 2026-07-20
+- **Issue:** #431
+
+## Summary
+
+The Docker backend runs two containers — `bot-bottle-orch-gateway` (gateway
+data plane) and `bot-bottle-orchestrator` (control plane) — where the
+macOS and Firecracker backends already run a single combined infra
+unit. This PRD collapses Docker to the same model: one `bot-bottle-infra`
+container running both processes under the `gateway_init` supervise tree, a
+restructured `Dockerfile.infra` as the shared gateway+orchestrator base,
+and a handful of extracted shared utilities (CA cert polling, teardown
+sequence, launch skeleton) that are currently duplicated across all three
+`consolidated_launch.py` files.
+
+## Goals / success criteria
+
+- Docker backend starts exactly one infra container instead of two.
+- `Dockerfile.infra` is the shared base image (gateway + orchestrator, no
+  buildah); the Firecracker image layers buildah on top of it.
+- The orchestrator process runs under the `gateway_init` supervise tree
+  inside the combined container (one PID-1, one restart/health surface).
+- CA cert polling, the teardown sequence, and the shared launch skeleton
+  (ensure-infra → register → provision → return context) live in a single
+  shared module; all three backends import from it.
+- No functional change to macOS or Firecracker launch paths.
+
+## Non-goals
+
+- Changing per-bottle isolation — agents stay one-VM/container-each.
+- Consolidating transport implementations (`DockerGatewayTransport`,
+  `AppleGatewayTransport`, `SshGatewayTransport`) — these are already the
+  right abstraction boundary.
+- macOS DHCP-inversion of registration order — irreducible backend
+  difference, stays as-is.
+- Any changes to the orchestrator RPC protocol or the attribution model.
+
+## Design
+
+### Dockerfile restructuring
+
+**Current shape:**
+
+- `Dockerfile.gateway` — data plane (mitmproxy, gitleaks, git, openssh,
+  supervise daemons)
+- `Dockerfile.orchestrator` — control plane (python:3.12-slim + bot_bottle
+  package; stdlib-only, no third-party deps)
+- `Dockerfile.infra` — Firecracker only: `FROM bot-bottle-gateway` +
+  buildah + `COPY --from bot-bottle-orchestrator`
+
+**New shape:**
+
+- `Dockerfile.gateway` — unchanged
+- `Dockerfile.orchestrator` — unchanged (single definition of orchestrator
+  content; both Docker infra and Firecracker infra `COPY --from` it)
+- `Dockerfile.infra` — **shared base**: `FROM bot-bottle-gateway` + `COPY
+  --from bot-bottle-orchestrator` (no buildah — Docker infra image)
+- `Dockerfile.infra.fc` — Firecracker only: `FROM bot-bottle-infra` +
+  buildah/crun/netavark/aardvark-dns (layered on the shared base, same net
+  result as today)
+
+The comment in `Dockerfile.infra` that says "the docker backend keeps
+orchestrator + gateway as separate images; this combined image exists only
+for the Firecracker single-VM cut" is removed.
+
+### Orchestrator in the supervise tree
+
+`gateway_init` already supervises the data-plane daemons (egress, git-http,
+supervise-MCP). The orchestrator control plane is added as another supervised
+process: `python3 -m bot_bottle.orchestrator --host 0.0.0.0 --port <port>
+--broker stub`.
+
+The orchestrator source is bind-mounted (`/app` → repo root, as today) so
+dev live-reload still works. `source_hash`-based container recreation in
+`OrchestratorService.ensure_running` continues to apply — a code change
+recreates the combined infra container, which bounces both gateway and
+orchestrator. This is acceptable: the docker backend is a dev/legacy target
+where in-flight egress connections across a code deploy are not a hard
+requirement.
+
+### `OrchestratorService` changes
+
+`OrchestratorService` currently starts two containers in sequence: gateway
+first (`DockerGateway.ensure_running`), then orchestrator. After this PRD:
+
+- Single `docker run` of `bot-bottle-infra:latest`
+- Container name: `bot-bottle-infra` (replaces `bot-bottle-orch-gateway` +
+  `bot-bottle-orchestrator`)
+- Published ports: `127.0.0.1:{port}:{port}` for the control plane (same as
+  today)
+- Bind mounts: repo root + host root (same as today)
+- `DockerGateway` becomes an implementation detail of `OrchestratorService`
+  rather than a separately started container; the gateway image name
+  (`GATEWAY_IMAGE`) is no longer referenced at runtime, only at build time
+  for the `Dockerfile.infra` base
+
+The `_gateway()` / `ensure_running` two-step in `OrchestratorService` is
+replaced by a single `_run_infra_container()`.
+
+### Shared backend utilities
+
+Three items are duplicated across
+`backend/docker/consolidated_launch.py`,
+`backend/macos_container/consolidated_launch.py`, and
+`backend/firecracker/consolidated_launch.py`:
+
+1. **CA cert polling loop** — `deadline = time.monotonic() + timeout; while
+   ...: try fetch CA; sleep` — extracted to
+   `backend/consolidated_util.py:poll_ca_cert(transport, *, timeout)`.
+
+2. **Teardown sequence** — `OrchestratorClient(url).teardown_bottle(id)` +
+   `deprovision_git_gate(transport, id)` — extracted to
+   `backend/consolidated_util.py:teardown_consolidated(url, transport,
+   bottle_id)`.
+
+3. **Launch skeleton** — all three follow: ensure-infra → allocate/register
+   → provision git-gate → fetch CA cert → return launch context. The macOS
+   inversion (agent starts before registration, source IP from DHCP) is the
+   only deviation. Extract a shared `_provision_bottle(transport, bottle_id,
+   plan, orchestrator_url)` helper covering the register → provision →
+   return-token steps; the backends keep their own `launch_consolidated`
+   wrappers for the before/after (infra-ensure + agent-start + IP
+   allocation), calling the shared helper.
+
+The new `backend/consolidated_util.py` module holds only backend-neutral,
+transport-agnostic logic. All three backends import from it.
+
+## Implementation chunks
+
+1. **(this PR)** Dockerfile restructuring: rename current `Dockerfile.infra`
+   content to `Dockerfile.infra.fc`; write new `Dockerfile.infra` as
+   gateway+orchestrator base. Update Firecracker image-build references from
+   `Dockerfile.infra` → `Dockerfile.infra.fc`.
+
+2. Add orchestrator process to `gateway_init` supervise tree.
+
+3. Collapse `OrchestratorService` to a single-container start; rename
+   container from `bot-bottle-orch-gateway`/`bot-bottle-orchestrator` →
+   `bot-bottle-infra`; update image name constant.
+
+4. Extract `backend/consolidated_util.py` with `poll_ca_cert`,
+   `teardown_consolidated`, and `_provision_bottle`; update all three
+   `consolidated_launch.py` files to import from it.
+
+5. Update tests that reference the old container names or two-container
+   startup sequence.
+
+## Open questions
+
+None — the supervise-tree approach and shared Dockerfile layering were
+confirmed in issue #431.