bot-bottle/docs/prds/0024-consolidate-sidecar-bundle.md

# PRD 0024: Consolidate per-bottle sidecars into a single bundle

- **Status:** Draft
- **Author:** didericis
- **Created:** 2026-05-26

## Summary

Replace the four per-bottle sidecar containers in the Docker
backend (pipelock, egress, git-gate, supervise) with a single
container image — `bot-bottle-sidecars` — that runs all four
daemons under a small stdlib-Python init supervisor. Same
per-bottle lifetime, same scope, fewer containers per bottle,
one Dockerfile to maintain instead of three. Outcome: the
docker backend's compose file goes from five services
(`agent`, `pipelock`, `egress`, `git-gate`, `supervise`) to
two (`agent`, `sidecars`); the smolmachines backend defined in
PRD 0023 reuses the same image as its sole sidecar container.

## Problem

The four sidecars are tightly coupled in lifetime and scope:

- All four start when a bottle starts and stop when it stops.
  There is no scenario where one runs without the others.
- `egress` is `pipelock`'s upstream over the internal network —
  nothing on the agent side ever addresses egress directly. Its
  separateness today is a docker-compose-ism: one Dockerfile per
  service was the easiest way to ship the chunk-by-chunk
  rollouts of PRDs 0001, 0008, 0013, and 0017.
- `git-gate` and `supervise` run their own daemons but with the
  same "started + stopped with the bottle" lifecycle.

Three concrete costs of keeping them split:

1. **Compose-file surface area.** Five `services:` entries per
   bottle. The renderer in `backend/docker/compose.py` has to
   know each one's image, env, healthcheck, port-mapping,
   dependency wiring (`depends_on`), and CA / config bind mounts.
   That's a lot of moving parts for what is really one logical
   sidecar.
2. **Cold start latency.** Docker creates and starts four
   containers in dependency order even for a trivial agent run.
   Each container costs ~50-100ms of compose orchestration even
   when the image is cached.
3. **Cross-backend duplication.** PRD 0023's smolmachines
   backend would otherwise need its own four-process supervisor
   on the host side. A shared bundle image collapses both
   backends onto the same sidecar primitive.

This PRD is also the prerequisite for chunk 3 of PRD 0023.

## Goals / Success Criteria

The feature works when all of the following are observable:

- `cli.py start <agent>` on the Docker backend produces a
  compose project with exactly two services (`agent`,
  `sidecars`) and three published agent-facing ports
  (HTTPS_PROXY, git-gate, supervise) on the `sidecars`
  container.
- All existing integration tests pass with no behavior change
  visible to the agent. The four daemons inside the bundle
  speak the same protocols on the same well-known in-container
  ports as before; only the container hostname changes.
- The sandbox-escape suite from PRD 0022 stays green.
- `docker logs bot-bottle-sidecars-<slug>` shows interleaved
  output from all four daemons, prefixed by the supervisor with
  the daemon name. Each daemon's exit propagates through the
  supervisor to the container's exit code.
- Sending SIGTERM to the bundle container (the docker stop path)
  shuts down all four daemons cleanly within the existing
  compose stop-grace timeout (10s).

The feature is **done** when all of the following ship:

- A new `Dockerfile.sidecars` (multi-stage) that:
  - Copies the `pipelock` binary from the upstream pipelock
    image (currently `ghcr.io/luckypipewrench/pipelock` pinned
    by digest in `bot_bottle/backend/docker/pipelock.py`).
  - Copies the `gitleaks` binary from `zricethezav/gitleaks`
    (currently pinned by digest in `Dockerfile.git-gate`).
  - Installs `mitmdump` (via `pip install mitmproxy==<pinned>`).
  - Installs the system deps `git-daemon` + `openssh-client`
    that git-gate needs.
  - Copies the existing addon + server Python from
    `bot_bottle/egress_addon.py`, `egress_addon_core.py`,
    `yaml_subset.py`, `supervise.py`, `supervise_server.py`.
  - Drops in a new `bot_bottle/sidecar_init.py` (stdlib
    Python) as the container's `ENTRYPOINT`.
- A new `bot_bottle/sidecar_init.py` — a small Python init
  supervisor that:
  - Reads which daemons to run from env (defaults: all four).
  - Spawns each as a `subprocess.Popen` with prefixed
    line-buffered output.
  - Catches `SIGTERM` / `SIGINT`, propagates to each child,
    `waitpid()`s with a per-child grace deadline, escalates to
    `SIGKILL` past the deadline.
  - Exits with code 0 only if every child exited 0; otherwise
    exits 1. (Or: any-child-died → tear down the rest and exit
    that child's code — see open question 2.)
- `bot_bottle/backend/docker/compose.py` renderer updated to
  emit one `sidecars` service in place of the four. The four
  in-container ports (8888 / 9099 / 9418 / 9100, today) all
  land on the same container; the agent-facing ports
  (HTTPS_PROXY, git-gate-SSH, supervise-MCP) are published as
  before, just from one container instead of three.
- `bot_bottle/backend/docker/{pipelock,egress,git_gate,supervise}.py`
  collapsed: the platform-neutral pieces stay
  (`PipelockProxy`, `Egress`, `GitGate`, `Supervise` ABCs and
  their plans), the docker-specific subclasses lose their
  per-container start/stop / image-build / healthcheck logic
  and gain shared bundle-aware helpers. Container name helpers
  (`pipelock_container_name(slug)` etc.) become a single
  `sidecar_bundle_container_name(slug)`.
- `Dockerfile.egress`, `Dockerfile.git-gate`, and
  `Dockerfile.supervise` deleted. The bundle is the only image.
- Tests:
  - Unit: the compose renderer emits exactly two services and
    one sidecars service has all three published ports.
  - Unit: the sidecar-init supervisor propagates SIGTERM and
    returns nonzero when a child crashes.
  - Integration: existing PRD 0001 / 0008 / 0013 / 0017
    integration tests run against the bundle and pass.
  - Integration: PRD 0022 sandbox-escape suite stays green.
- `AGENTS.md` updated to describe the bundle and the
  daemons-inside layout.

## Non-goals

- **No protocol changes between sidecars.** pipelock still
  speaks the same HTTPS-proxy protocol on the same port; egress
  is still pipelock's upstream; git-gate still listens on
  git-daemon's port; supervise still serves the same MCP HTTP
  endpoint. Only the container they run in changes.
- **No config-schema changes.** `pipelock.yaml`,
  `routes.yaml`, the git-gate access-hook, and the supervise
  queue path all stay where they are; the bundle just bind-mounts
  them at the same in-container paths as before.
- **No host-bind-mount surgery.** Each daemon's existing bind
  mounts (per-bottle CA paths, the supervise queue dir, the
  git-gate creds dir) remain. The bundle aggregates them onto
  one container.
- **No supervisord / s6 / runit.** A 50-line stdlib Python init
  is the supervisor. Adding a new init system for this is more
  weight than the problem deserves and conflicts with the
  project's stdlib-first ethos.
- **No selective daemon disable surfaced to the manifest.** The
  init understands "skip git-gate / supervise when the bottle
  doesn't use them" via env vars set by the compose renderer,
  but operators don't get a manifest knob — the existing
  `bottle.git` / `bottle.supervise` flags continue to drive it.
- **No agent-image changes.** The agent container (PRD 0023's
  microVM in the smolmachines case) is unaffected; this PRD is
  strictly about consolidating the sidecar chain.

## Scope

### In scope

- New `Dockerfile.sidecars` (multi-stage) bringing pipelock,
  mitmproxy, gitleaks, git-daemon, openssh-client, and the
  project's addon + server Python into one image.
- New `bot_bottle/sidecar_init.py` supervising the four
  daemons.
- `backend/docker/compose.py` renderer collapse (five services
  → two).
- `backend/docker/{pipelock,egress,git_gate,supervise}.py`
  reshape: keep the abstract `Plan` / proxy classes; remove
  per-container lifecycle code that compose-up no longer needs.
- Image name and tag pinning (env var override + default; see
  open question 3).
- Test updates: unit and integration tests that probe the
  four-container shape get rewritten against the one-container
  shape.
- README + AGENTS.md doc updates.

### Out of scope

- The smolmachines backend itself (PRD 0023). This PRD just
  produces the image; PRD 0023 consumes it.
- Per-daemon resource limits (CPU / memory caps) on the bundle.
  Today nothing in the project sets them; consolidation
  doesn't change that.
- Healthcheck redesign. The agent's `depends_on:
  service_healthy` against the bundle covers all four daemons;
  defining a single bundle-level healthcheck that aggregates
  the per-daemon readiness is open question 4.
- Multi-arch image builds (arm64 + amd64). The current
  per-sidecar images are amd64-only or whatever their bases
  ship; we keep that posture.

## Proposed Design

### Bundle image

`Dockerfile.sidecars` is a four-stage multi-stage build, one
stage per source binary, plus a final stage that assembles them:

```dockerfile
# Stage 1: pull pipelock binary
FROM ghcr.io/luckypipewrench/pipelock@sha256:<pinned> AS pipelock-src
# pipelock binary is at /usr/local/bin/pipelock in this image.

# Stage 2: pull gitleaks binary
FROM zricethezav/gitleaks@sha256:<pinned> AS gitleaks-src
# gitleaks binary is at /usr/bin/gitleaks in this image.

# Stage 3: mitmproxy base (already has Python + mitmdump installed)
FROM mitmproxy/mitmproxy:11.1.3 AS final
USER root

# System deps for the git-gate daemon side
RUN apt-get update \
 && apt-get install -y --no-install-recommends \
      git git-daemon-run openssh-client ca-certificates \
 && rm -rf /var/lib/apt/lists/*

# Drop in the project's Python addon + server code
COPY bot_bottle/egress_addon_core.py /app/egress_addon_core.py
COPY bot_bottle/egress_addon.py      /app/egress_addon.py
COPY bot_bottle/yaml_subset.py       /app/yaml_subset.py
COPY bot_bottle/supervise.py         /app/supervise.py
COPY bot_bottle/supervise_server.py  /app/supervise_server.py
COPY bot_bottle/sidecar_init.py      /app/sidecar_init.py

# Pull the standalone binaries into the final stage
COPY --from=pipelock-src /usr/local/bin/pipelock /usr/local/bin/pipelock
COPY --from=gitleaks-src /usr/bin/gitleaks       /usr/bin/gitleaks

# Layout the bundle uses at runtime — preserved verbatim from the
# four previous Dockerfiles so existing docker-cp paths still work.
RUN mkdir -p \
      /etc/pipelock \
      /etc/egress \
      /etc/git-gate \
      /git-gate/creds \
      /git \
      /run/supervise/queue \
      /home/mitmproxy/.mitmproxy

EXPOSE 8888 9099 9418 9100

ENTRYPOINT ["python3", "/app/sidecar_init.py"]
```

The final stage starts from the mitmproxy image because
mitmproxy has the heaviest install footprint (Python + mitmdump
+ deps); copying the other two binaries in is cheaper than the
reverse. Pinning each base by digest is unchanged from the
existing Dockerfiles.

### Init supervisor

`bot_bottle/sidecar_init.py` (sketch — actual code lands as
part of implementation):

```python
"""Per-bottle sidecar supervisor.

Spawns the configured daemons, forwards SIGTERM/SIGINT, exits
with the first non-zero child code (or 0 if every child exited
cleanly during normal shutdown)."""

DAEMONS = [
    ("egress",    ["sh", "-c", EGRESS_ENTRYPOINT_SH]),
    ("pipelock",  ["/usr/local/bin/pipelock", "run",
                   "--config", "/etc/pipelock/pipelock.yaml"]),
    ("git-gate",  ["/git-gate-entrypoint.sh"]),
    ("supervise", ["python3", "/app/supervise_server.py"]),
]

# Order matters only for first-launch race-window reasons:
# egress starts first so pipelock's upstream connect succeeds
# during pipelock startup. git-gate and supervise are
# independent.
```

The env-driven daemon subset is the same handshake as today's
compose renderer: bottles without `git` skip git-gate, bottles
with `supervise: false` skip supervise.

### Compose renderer collapse

`bottle_plan_to_compose` emits one `sidecars` service in place
of the four. The service inherits the union of the four's
existing bind mounts; environment variables get prefixed by
daemon name where they clash (none clash today, but the renderer
becomes the central place to enforce that). Container hostname
becomes `sidecars` (or `bot-bottle-sidecars-<slug>` for the
externally-visible name). The agent service's HTTPS_PROXY and
git-gate URL move from per-sidecar hostnames to the single
`sidecars` hostname:

```yaml
# Before (sketch — five services)
services:
  agent:
    environment:
      HTTPS_PROXY: "http://pipelock:8888"
      GIT_GATE_URL: "git://git-gate:9418/repo"
      MCP_SUPERVISE_URL: "http://supervise:9100"
  pipelock: { image: ghcr.io/luckypipewrench/pipelock:... }
  egress:   { image: bot-bottle-egress:latest }
  git-gate: { image: bot-bottle-git-gate:latest }
  supervise:{ image: bot-bottle-supervise:latest }

# After (two services)
services:
  agent:
    environment:
      HTTPS_PROXY: "http://sidecars:8888"
      GIT_GATE_URL: "git://sidecars:9418/repo"
      MCP_SUPERVISE_URL: "http://sidecars:9100"
  sidecars:
    image: bot-bottle-sidecars:<pinned>
    # union of the four prior services' volumes / env / ports
```

`depends_on` collapses: the agent depends on `sidecars` only.

### Backend Python collapse

The four `bot_bottle/backend/docker/<sidecar>.py` files keep
their platform-neutral abstractions (proxy/plan classes) but
shed the docker-container-lifecycle code that compose-up
already owns. Container-name helpers consolidate:

```python
# was:
def pipelock_container_name(slug): ...
def egress_container_name(slug): ...
def git_gate_container_name(slug): ...
def supervise_container_name(slug): ...

# becomes:
def sidecar_bundle_container_name(slug: str) -> str:
    return f"bot-bottle-sidecars-{slug}"
```

Per-daemon "is the container up?" helpers used by orphan
cleanup converge on a single check against the bundle name.

### External dependencies

None new. The bundle build pulls the same upstream images we
already pull; the consolidation is a packaging change.

### Migration

This PRD's change is large but mechanical. A pre-merge dry-run:

1. Land the bundle image build (`Dockerfile.sidecars` +
   `sidecar_init.py`) without changing the renderer.
   Confirm `docker build -f Dockerfile.sidecars .` succeeds
   and the resulting container runs all four daemons.
2. Switch the renderer to emit the two-service shape behind an
   env-var feature flag (e.g.
   `BOT_BOTTLE_SIDECAR_BUNDLE=1`).
3. Update integration tests in-place; flip the default once
   green; delete the flag and the old Dockerfiles in a
   follow-up commit on the same branch.

The compose-per-instance work in PRD 0018 already separated
sidecar lifecycle from agent lifecycle, so this PRD is
materially a renderer + image-build change — not a backend
rewrite.

## Sizing — into chunks

1. **Bundle image + init supervisor.** Write `Dockerfile.sidecars`
   and `sidecar_init.py`, ship them, add a unit test that
   builds the image in CI and asserts the four daemons start.
   No renderer change yet.
2. **Compose renderer collapse.** Update
   `bottle_plan_to_compose` to emit two services. Feature flag
   it via env var. Update unit tests to assert on both shapes
   (flag on vs off) during the migration window.
3. **Backend Python collapse.** Drop the vestigial per-container
   `.start()` / `.stop()` methods from `DockerPipelockProxy`,
   `DockerEgress`, `DockerGitGate`, `DockerSupervise` (and from
   the ABCs in `bot_bottle/{pipelock,egress,git_gate,supervise}.py`).
   These were already documented as vestigial in PRD 0018 ch3.
   Strip vestigial sidecar-instance parameters from
   `launch.launch()` and `prepare.resolve_plan()`. Delete the
   integration tests that exclusively exercised those methods
   (`test_pipelock_sidecar_smoke`, `test_supervise_sidecar`,
   `test_git_gate_sidecar`, `test_git_gate_mirror`). Skip
   `test_pipelock_apply` pending chunk 4 bringup rewrite.
   Orphan cleanup already uses a prefix scan and catches the
   bundle for free; no change needed. Dockerfile deletion is
   deferred to chunk 5 — until the flag flips, the legacy path
   still needs `Dockerfile.{egress,git-gate,supervise}` for
   compose `build:`.
4. **Integration test sweep.** Rewrite `test_pipelock_apply`'s
   bringup with direct `docker run` so the
   `apply_allowlist_change` hot-reload retains coverage. Add
   any bundle-specific integration smoke as needed. Confirm
   PRD 0022 stays green.
5. **Docs + flag removal.** Flip the default, remove the
   feature flag, delete `Dockerfile.{egress,git-gate,supervise}`,
   update README + AGENTS.md.

## Open questions

1. **Init failure semantics.** When one daemon crashes mid-run,
   the bundle does NOT tear down the survivors — the failure is
   logged, the surviving daemons keep running, and whatever the
   dead one served starts failing in a way the agent surfaces.
   The eventual design is restart-the-dead-daemon plus a
   notification to the supervise sidecar so the operator sees
   the event explicitly; chunk 1 ships only the "log and leave
   alone" half. Tear-down-the-bundle was considered and
   rejected: one sick daemon shouldn't take the bottle offline.
2. **Exit-code propagation.** When the supervisor finally exits
   (signal-driven shutdown, or every child having died on its
   own), the container exits with `max(child returncodes)` —
   the worst nonzero code wins. On graceful shutdown every child
   is signal-killed (negative returncode) so the max is 0; a
   crashed-before-signal daemon's nonzero code wins and reaches
   the operator on container exit.
3. **Image pin policy.** Pin `bot-bottle-sidecars` by tag
   (`:latest` rebuilt per-release) or by digest written into a
   `BOT_BOTTLE_SIDECAR_IMAGE` env var like the existing
   `BOT_BOTTLE_PIPELOCK_IMAGE`? Default to env-var override
   + a documented tag; digest pinning is an operator opt-in.
4. **Healthcheck aggregation.** Today each sidecar service has
   its own compose healthcheck and `agent.depends_on:
   service_healthy: { pipelock: true, ... }`. With one
   container, the bundle needs one healthcheck that returns
   ready iff all daemons are listening. Cheapest: TCP probe on
   pipelock's port + git-gate's port + supervise's port from
   inside the container, scripted into a small `/app/healthcheck.sh`.
   Resolve in chunk 1.
5. **Log interleaving + debuggability.** All four daemons'
   stdout/stderr merge into one container log. The init
   prefixes each line with the daemon name, but operators may
   want per-daemon log files for easier triage. Default: no
   per-daemon files in v1; revisit if debug-time pain shows up.
6. **Backwards compat for an installed-base test fixture.**
   Some integration tests synthesize compose files by hand and
   assert on per-sidecar container names. They'll need
   touching in chunk 4. List them up front in the chunk-4
   commit so the diff isn't a surprise.

## References

- `Dockerfile.egress`, `Dockerfile.git-gate`,
  `Dockerfile.supervise` — the three Dockerfiles this PRD
  collapses into `Dockerfile.sidecars`.
- `bot_bottle/backend/docker/compose.py` — the renderer this
  PRD slims down.
- `bot_bottle/backend/docker/pipelock.py` — current home of
  `PIPELOCK_IMAGE` and the pinned digest the bundle's first
  stage reuses.
- PRD 0017
  (`docs/prds/0017-egress-proxy-via-mitmproxy.md`) — defines
  egress's role as pipelock's upstream; this PRD relies on
  that being implementable over localhost just as easily as
  over the internal docker network.
- PRD 0018
  (`docs/prds/0018-compose-per-instance.md`) — the
  compose-per-instance refactor this PRD builds on. PRD 0018
  separated sidecar lifecycle from agent lifecycle, which is
  what makes a single-bundle compose service a renderer-only
  change instead of a backend rewrite.
- PRD 0022
  (`docs/prds/0022-sandbox-escape-integration-test.md`) — must
  remain green through the migration.
- PRD 0023
  (`docs/prds/0023-smolmachines-backend.md`) — the second
  consumer of this bundle; depends on this PRD's image being
  available before its chunk 3.