bot-bottle/docs/prds/0024-consolidate-sidecar-bundle.md

PRD 0024: Consolidate per-bottle sidecars into a single bundle

• Status: Active
• Author: didericis
• Created: 2026-05-26

Summary

Replace the four per-bottle sidecar containers in the Docker backend (pipelock, egress, git-gate, supervise) with a single container image — bot-bottle-sidecars — that runs all four daemons under a small stdlib-Python init supervisor. Same per-bottle lifetime, same scope, fewer containers per bottle, one Dockerfile to maintain instead of three. Outcome: the docker backend's compose file goes from five services (agent, pipelock, egress, git-gate, supervise) to two (agent, sidecars); the smolmachines backend defined in PRD 0023 reuses the same image as its sole sidecar container.

Problem

The four sidecars are tightly coupled in lifetime and scope:

• All four start when a bottle starts and stop when it stops. There is no scenario where one runs without the others.
• egress is pipelock's upstream over the internal network — nothing on the agent side ever addresses egress directly. Its separateness today is a docker-compose-ism: one Dockerfile per service was the easiest way to ship the chunk-by-chunk rollouts of PRDs 0001, 0008, 0013, and 0017.
• git-gate and supervise run their own daemons but with the same "started + stopped with the bottle" lifecycle.

Three concrete costs of keeping them split:

1. Compose-file surface area. Five services: entries per bottle. The renderer in backend/docker/compose.py has to know each one's image, env, healthcheck, port-mapping, dependency wiring (depends_on), and CA / config bind mounts. That's a lot of moving parts for what is really one logical sidecar.
2. Cold start latency. Docker creates and starts four containers in dependency order even for a trivial agent run. Each container costs ~50-100ms of compose orchestration even when the image is cached.
3. Cross-backend duplication. PRD 0023's smolmachines backend would otherwise need its own four-process supervisor on the host side. A shared bundle image collapses both backends onto the same sidecar primitive.

This PRD is also the prerequisite for chunk 3 of PRD 0023.

Goals / Success Criteria

The feature works when all of the following are observable:

• cli.py start <agent> on the Docker backend produces a compose project with exactly two services (agent, sidecars) and three published agent-facing ports (HTTPS_PROXY, git-gate, supervise) on the sidecars container.
• All existing integration tests pass with no behavior change visible to the agent. The four daemons inside the bundle speak the same protocols on the same well-known in-container ports as before; only the container hostname changes.
• The sandbox-escape suite from PRD 0022 stays green.
• docker logs bot-bottle-sidecars-<slug> shows interleaved output from all four daemons, prefixed by the supervisor with the daemon name. Each daemon's exit propagates through the supervisor to the container's exit code.
• Sending SIGTERM to the bundle container (the docker stop path) shuts down all four daemons cleanly within the existing compose stop-grace timeout (10s).

The feature is done when all of the following ship:

• A new Dockerfile.sidecars (multi-stage) that:
  • Copies the pipelock binary from the upstream pipelock image (currently ghcr.io/luckypipewrench/pipelock pinned by digest in bot_bottle/backend/docker/pipelock.py).
  • Copies the gitleaks binary from zricethezav/gitleaks (currently pinned by digest in Dockerfile.git-gate).
  • Installs mitmdump (via pip install mitmproxy==<pinned>).
  • Installs the system deps git-daemon + openssh-client that git-gate needs.
  • Copies the existing addon + server Python from bot_bottle/egress_addon.py, egress_addon_core.py, yaml_subset.py, supervise.py, supervise_server.py.
  • Drops in a new bot_bottle/sidecar_init.py (stdlib Python) as the container's ENTRYPOINT.
• A new bot_bottle/sidecar_init.py — a small Python init supervisor that:
  • Reads which daemons to run from env (defaults: all four).
  • Spawns each as a subprocess.Popen with prefixed line-buffered output.
  • Catches SIGTERM / SIGINT, propagates to each child, waitpid()s with a per-child grace deadline, escalates to SIGKILL past the deadline.
  • Exits with code 0 only if every child exited 0; otherwise exits 1. (Or: any-child-died → tear down the rest and exit that child's code — see open question 2.)
• bot_bottle/backend/docker/compose.py renderer updated to emit one sidecars service in place of the four. The four in-container ports (8888 / 9099 / 9418 / 9100, today) all land on the same container; the agent-facing ports (HTTPS_PROXY, git-gate-SSH, supervise-MCP) are published as before, just from one container instead of three.
• bot_bottle/backend/docker/{pipelock,egress,git_gate,supervise}.py collapsed: the platform-neutral pieces stay (PipelockProxy, Egress, GitGate, Supervise ABCs and their plans), the docker-specific subclasses lose their per-container start/stop / image-build / healthcheck logic and gain shared bundle-aware helpers. Container name helpers (pipelock_container_name(slug) etc.) become a single sidecar_bundle_container_name(slug).
• Dockerfile.egress, Dockerfile.git-gate, and Dockerfile.supervise deleted. The bundle is the only image.
• Tests:
  • Unit: the compose renderer emits exactly two services and one sidecars service has all three published ports.
  • Unit: the sidecar-init supervisor propagates SIGTERM and returns nonzero when a child crashes.
  • Integration: existing PRD 0001 / 0008 / 0013 / 0017 integration tests run against the bundle and pass.
  • Integration: PRD 0022 sandbox-escape suite stays green.
• AGENTS.md updated to describe the bundle and the daemons-inside layout.

Non-goals

• No protocol changes between sidecars. pipelock still speaks the same HTTPS-proxy protocol on the same port; egress is still pipelock's upstream; git-gate still listens on git-daemon's port; supervise still serves the same MCP HTTP endpoint. Only the container they run in changes.
• No config-schema changes. pipelock.yaml, routes.yaml, the git-gate access-hook, and the supervise queue path all stay where they are; the bundle just bind-mounts them at the same in-container paths as before.
• No host-bind-mount surgery. Each daemon's existing bind mounts (per-bottle CA paths, the supervise queue dir, the git-gate creds dir) remain. The bundle aggregates them onto one container.
• No supervisord / s6 / runit. A 50-line stdlib Python init is the supervisor. Adding a new init system for this is more weight than the problem deserves and conflicts with the project's stdlib-first ethos.
• No selective daemon disable surfaced to the manifest. The init understands "skip git-gate / supervise when the bottle doesn't use them" via env vars set by the compose renderer, but operators don't get a manifest knob — the existing bottle.git / bottle.supervise flags continue to drive it.
• No agent-image changes. The agent container (PRD 0023's microVM in the smolmachines case) is unaffected; this PRD is strictly about consolidating the sidecar chain.

Scope

In scope

• New Dockerfile.sidecars (multi-stage) bringing pipelock, mitmproxy, gitleaks, git-daemon, openssh-client, and the project's addon + server Python into one image.
• New bot_bottle/sidecar_init.py supervising the four daemons.
• backend/docker/compose.py renderer collapse (five services → two).
• backend/docker/{pipelock,egress,git_gate,supervise}.py reshape: keep the abstract Plan / proxy classes; remove per-container lifecycle code that compose-up no longer needs.
• Image name and tag pinning (env var override + default; see open question 3).
• Test updates: unit and integration tests that probe the four-container shape get rewritten against the one-container shape.
• README + AGENTS.md doc updates.

Out of scope

• The smolmachines backend itself (PRD 0023). This PRD just produces the image; PRD 0023 consumes it.
• Per-daemon resource limits (CPU / memory caps) on the bundle. Today nothing in the project sets them; consolidation doesn't change that.
• Healthcheck redesign. The agent's depends_on: service_healthy against the bundle covers all four daemons; defining a single bundle-level healthcheck that aggregates the per-daemon readiness is open question 4.
• Multi-arch image builds (arm64 + amd64). The current per-sidecar images are amd64-only or whatever their bases ship; we keep that posture.

Proposed Design

Bundle image

Dockerfile.sidecars is a four-stage multi-stage build, one stage per source binary, plus a final stage that assembles them:

# Stage 1: pull pipelock binary
FROM ghcr.io/luckypipewrench/pipelock@sha256:<pinned> AS pipelock-src
# pipelock binary is at /usr/local/bin/pipelock in this image.

# Stage 2: pull gitleaks binary
FROM zricethezav/gitleaks@sha256:<pinned> AS gitleaks-src
# gitleaks binary is at /usr/bin/gitleaks in this image.

# Stage 3: mitmproxy base (already has Python + mitmdump installed)
FROM mitmproxy/mitmproxy:11.1.3 AS final
USER root

# System deps for the git-gate daemon side
RUN apt-get update \
 && apt-get install -y --no-install-recommends \
      git git-daemon-run openssh-client ca-certificates \
 && rm -rf /var/lib/apt/lists/*

# Drop in the project's Python addon + server code
COPY bot_bottle/egress_addon_core.py /app/egress_addon_core.py
COPY bot_bottle/egress_addon.py      /app/egress_addon.py
COPY bot_bottle/yaml_subset.py       /app/yaml_subset.py
COPY bot_bottle/supervise.py         /app/supervise.py
COPY bot_bottle/supervise_server.py  /app/supervise_server.py
COPY bot_bottle/sidecar_init.py      /app/sidecar_init.py

# Pull the standalone binaries into the final stage
COPY --from=pipelock-src /usr/local/bin/pipelock /usr/local/bin/pipelock
COPY --from=gitleaks-src /usr/bin/gitleaks       /usr/bin/gitleaks

# Layout the bundle uses at runtime — preserved verbatim from the
# four previous Dockerfiles so existing docker-cp paths still work.
RUN mkdir -p \
      /etc/pipelock \
      /etc/egress \
      /etc/git-gate \
      /git-gate/creds \
      /git \
      /run/supervise/queue \
      /home/mitmproxy/.mitmproxy

EXPOSE 8888 9099 9418 9100

ENTRYPOINT ["python3", "/app/sidecar_init.py"]

The final stage starts from the mitmproxy image because mitmproxy has the heaviest install footprint (Python + mitmdump

• deps); copying the other two binaries in is cheaper than the reverse. Pinning each base by digest is unchanged from the existing Dockerfiles.

Init supervisor

bot_bottle/sidecar_init.py (sketch — actual code lands as part of implementation):

"""Per-bottle sidecar supervisor.

Spawns the configured daemons, forwards SIGTERM/SIGINT, exits
with the first non-zero child code (or 0 if every child exited
cleanly during normal shutdown)."""

DAEMONS = [
    ("egress",    ["sh", "-c", EGRESS_ENTRYPOINT_SH]),
    ("pipelock",  ["/usr/local/bin/pipelock", "run",
                   "--config", "/etc/pipelock/pipelock.yaml"]),
    ("git-gate",  ["/git-gate-entrypoint.sh"]),
    ("supervise", ["python3", "/app/supervise_server.py"]),
]

# Order matters only for first-launch race-window reasons:
# egress starts first so pipelock's upstream connect succeeds
# during pipelock startup. git-gate and supervise are
# independent.

The env-driven daemon subset is the same handshake as today's compose renderer: bottles without git skip git-gate, bottles with supervise: false skip supervise.

Compose renderer collapse

bottle_plan_to_compose emits one sidecars service in place of the four. The service inherits the union of the four's existing bind mounts; environment variables get prefixed by daemon name where they clash (none clash today, but the renderer becomes the central place to enforce that). Container hostname becomes sidecars (or bot-bottle-sidecars-<slug> for the externally-visible name). The agent service's HTTPS_PROXY and git-gate URL move from per-sidecar hostnames to the single sidecars hostname:

# Before (sketch — five services)
services:
  agent:
    environment:
      HTTPS_PROXY: "http://pipelock:8888"
      GIT_GATE_URL: "git://git-gate:9418/repo"
      MCP_SUPERVISE_URL: "http://supervise:9100"
  pipelock: { image: ghcr.io/luckypipewrench/pipelock:... }
  egress:   { image: bot-bottle-egress:latest }
  git-gate: { image: bot-bottle-git-gate:latest }
  supervise:{ image: bot-bottle-supervise:latest }

# After (two services)
services:
  agent:
    environment:
      HTTPS_PROXY: "http://sidecars:8888"
      GIT_GATE_URL: "git://sidecars:9418/repo"
      MCP_SUPERVISE_URL: "http://sidecars:9100"
  sidecars:
    image: bot-bottle-sidecars:<pinned>
    # union of the four prior services' volumes / env / ports

depends_on collapses: the agent depends on sidecars only.

Backend Python collapse

The four bot_bottle/backend/docker/<sidecar>.py files keep their platform-neutral abstractions (proxy/plan classes) but shed the docker-container-lifecycle code that compose-up already owns. Container-name helpers consolidate:

# was:
def pipelock_container_name(slug): ...
def egress_container_name(slug): ...
def git_gate_container_name(slug): ...
def supervise_container_name(slug): ...

# becomes:
def sidecar_bundle_container_name(slug: str) -> str:
    return f"bot-bottle-sidecars-{slug}"

Per-daemon "is the container up?" helpers used by orphan cleanup converge on a single check against the bundle name.

External dependencies

None new. The bundle build pulls the same upstream images we already pull; the consolidation is a packaging change.

Migration

This PRD's change is large but mechanical. A pre-merge dry-run:

1. Land the bundle image build (Dockerfile.sidecars + sidecar_init.py) without changing the renderer. Confirm docker build -f Dockerfile.sidecars . succeeds and the resulting container runs all four daemons.
2. Switch the renderer to emit the two-service shape behind an env-var feature flag (e.g. BOT_BOTTLE_SIDECAR_BUNDLE=1).
3. Update integration tests in-place; flip the default once green; delete the flag and the old Dockerfiles in a follow-up commit on the same branch.

The compose-per-instance work in PRD 0018 already separated sidecar lifecycle from agent lifecycle, so this PRD is materially a renderer + image-build change — not a backend rewrite.

Sizing — into chunks

1. Bundle image + init supervisor. Write Dockerfile.sidecars and sidecar_init.py, ship them, add a unit test that builds the image in CI and asserts the four daemons start. No renderer change yet.
2. Compose renderer collapse. Update bottle_plan_to_compose to emit two services. Feature flag it via env var. Update unit tests to assert on both shapes (flag on vs off) during the migration window.
3. Backend Python collapse. Drop the vestigial per-container .start() / .stop() methods from DockerPipelockProxy, DockerEgress, DockerGitGate, DockerSupervise (and from the ABCs in bot_bottle/{pipelock,egress,git_gate,supervise}.py). These were already documented as vestigial in PRD 0018 ch3. Strip vestigial sidecar-instance parameters from launch.launch() and prepare.resolve_plan(). Delete the integration tests that exclusively exercised those methods (test_pipelock_sidecar_smoke, test_supervise_sidecar, test_git_gate_sidecar, test_git_gate_mirror). Skip test_pipelock_apply pending chunk 4 bringup rewrite. Orphan cleanup already uses a prefix scan and catches the bundle for free; no change needed. Dockerfile deletion is deferred to chunk 5 — until the flag flips, the legacy path still needs Dockerfile.{egress,git-gate,supervise} for compose build:.
4. Integration test sweep. Rewrite test_pipelock_apply's bringup with direct docker run so the apply_allowlist_change hot-reload retains coverage. Add any bundle-specific integration smoke as needed. Confirm PRD 0022 stays green.
5. Docs + flag removal. Flip the default, remove the feature flag, delete Dockerfile.{egress,git-gate,supervise}, update README + AGENTS.md.

Open questions

1. Init failure semantics. When one daemon crashes mid-run, the bundle does NOT tear down the survivors — the failure is logged, the surviving daemons keep running, and whatever the dead one served starts failing in a way the agent surfaces. The eventual design is restart-the-dead-daemon plus a notification to the supervise sidecar so the operator sees the event explicitly; chunk 1 ships only the "log and leave alone" half. Tear-down-the-bundle was considered and rejected: one sick daemon shouldn't take the bottle offline.
2. Exit-code propagation. When the supervisor finally exits (signal-driven shutdown, or every child having died on its own), the container exits with max(child returncodes) — the worst nonzero code wins. On graceful shutdown every child is signal-killed (negative returncode) so the max is 0; a crashed-before-signal daemon's nonzero code wins and reaches the operator on container exit.
3. Image pin policy. Pin bot-bottle-sidecars by tag (:latest rebuilt per-release) or by digest written into a BOT_BOTTLE_SIDECAR_IMAGE env var like the existing BOT_BOTTLE_PIPELOCK_IMAGE? Default to env-var override
   • a documented tag; digest pinning is an operator opt-in.
4. Healthcheck aggregation. Today each sidecar service has its own compose healthcheck and agent.depends_on: service_healthy: { pipelock: true, ... }. With one container, the bundle needs one healthcheck that returns ready iff all daemons are listening. Cheapest: TCP probe on pipelock's port + git-gate's port + supervise's port from inside the container, scripted into a small /app/healthcheck.sh. Resolve in chunk 1.
5. Log interleaving + debuggability. All four daemons' stdout/stderr merge into one container log. The init prefixes each line with the daemon name, but operators may want per-daemon log files for easier triage. Default: no per-daemon files in v1; revisit if debug-time pain shows up.
6. Backwards compat for an installed-base test fixture. Some integration tests synthesize compose files by hand and assert on per-sidecar container names. They'll need touching in chunk 4. List them up front in the chunk-4 commit so the diff isn't a surprise.

References

• Dockerfile.egress, Dockerfile.git-gate, Dockerfile.supervise — the three Dockerfiles this PRD collapses into Dockerfile.sidecars.
• bot_bottle/backend/docker/compose.py — the renderer this PRD slims down.
• bot_bottle/backend/docker/pipelock.py — current home of PIPELOCK_IMAGE and the pinned digest the bundle's first stage reuses.
• PRD 0017 (docs/prds/0017-egress-proxy-via-mitmproxy.md) — defines egress's role as pipelock's upstream; this PRD relies on that being implementable over localhost just as easily as over the internal docker network.
• PRD 0018 (docs/prds/0018-compose-per-instance.md) — the compose-per-instance refactor this PRD builds on. PRD 0018 separated sidecar lifecycle from agent lifecycle, which is what makes a single-bundle compose service a renderer-only change instead of a backend rewrite.
• PRD 0022 (docs/prds/0022-sandbox-escape-integration-test.md) — must remain green through the migration.
• PRD 0023 (docs/prds/0023-smolmachines-backend.md) — the second consumer of this bundle; depends on this PRD's image being available before its chunk 3.