Consolidate infra backend for docker #431

New Issue

2026-07-20T14:44:44-04:00

didericis commented

2026-07-20 14:44:44 -04:00

We originally separated the gateway from the orchestrator to lower the surface area of things to secure around the secrets, but we backed out of that due to overhead and complexity issues on both the macos container and the firecracker backend, and because we can't really afford either to be compromised the same way we can afford an issue in the agent/they both need to remain secure.

For consistency we should create an infra image and run a single infra container on the docker backend as well. Evaluate whether or not creating a shared docker imagefile also makes sense. Probably, but there's an argument for keeping it separate so it's easier to read and maintain/the orchestrator needs remain conceptually separate from the gateway needs.

We originally separated the gateway from the orchestrator to lower the surface area of things to secure around the secrets, but we backed out of that due to overhead and complexity issues on both the macos container and the firecracker backend, and because we can't really afford either to be compromised the same way we can afford an issue in the agent/they both need to remain secure. For consistency we should create an infra image and run a single infra container on the docker backend as well. Evaluate whether or not creating a shared docker imagefile also makes sense. Probably, but there's an argument for keeping it separate so it's easier to read and maintain/the orchestrator needs remain conceptually separate from the gateway needs.

gitea-actions bot added the Status/Needs Triage label 2026-07-20 14:44:52 -04:00

didericis added Kind/Enhancement and removed Status/Needs Triage labels 2026-07-20 14:45:15 -04:00

didericis-claude was assigned by didericis

2026-07-20 14:45:25 -04:00

didericis-claude commented

2026-07-20 14:49:22 -04:00

Looking at the current docker slice: OrchestratorService.ensure_running starts two containers — bot-bottle-orch-gateway from Dockerfile.gateway and bot-bottle-orchestrator from Dockerfile.orchestrator — and wires them together on the shared network. The firecracker backend has the consolidated pattern already (gateway + orchestrator content in one image, supervised by gateway_init). This would mirror that.

On the shared Dockerfile question: I lean yes, with a layered model:

Dockerfile.infra = gateway + orchestrator (docker infra, no buildah) — currently this is firecracker-only but the gateway/orchestrator combination is what both backends need
Firecracker extends it: FROM bot-bottle-infra + buildah/crun/netavark
Dockerfile.orchestrator stays as the single canonical orchestrator content definition (still referenced via COPY --from)

That makes the split explicit by layer rather than by filename, and firecracker gets its builder without duplicating the base.

One open question before I start: should the orchestrator join the gateway's gateway_init supervise tree (so a single entrypoint manages both), or run as a sibling CMD alongside it? The gateway_init path seems cleaner since restart/health logic is already there — but it changes how the source_hash live-reload detect would work (currently OrchestratorService recreates the container on a code change; with a combined container we'd recreate the whole thing, which also bounces the gateway and drops in-flight egress connections). That may be acceptable given this is a dev-only concern, but worth confirming.

If the supervise-tree approach is the right call, I'll proceed with: (1) restructure Dockerfile.infra as the shared gateway+orchestrator base, (2) add orchestrator to gateway_init supervision, (3) update OrchestratorService to start one bot-bottle-infra container, (4) update firecracker to FROM bot-bottle-infra + buildah.

Looking at the current docker slice: `OrchestratorService.ensure_running` starts two containers — `bot-bottle-orch-gateway` from `Dockerfile.gateway` and `bot-bottle-orchestrator` from `Dockerfile.orchestrator` — and wires them together on the shared network. The firecracker backend has the consolidated pattern already (gateway + orchestrator content in one image, supervised by `gateway_init`). This would mirror that. **On the shared Dockerfile question:** I lean yes, with a layered model: - `Dockerfile.infra` = gateway + orchestrator (docker infra, no buildah) — currently this is firecracker-only but the gateway/orchestrator combination is what both backends need - Firecracker extends it: `FROM bot-bottle-infra` + buildah/crun/netavark - `Dockerfile.orchestrator` stays as the single canonical orchestrator content definition (still referenced via `COPY --from`) That makes the split explicit by layer rather than by filename, and firecracker gets its builder without duplicating the base. **One open question before I start:** should the orchestrator join the gateway's `gateway_init` supervise tree (so a single entrypoint manages both), or run as a sibling CMD alongside it? The `gateway_init` path seems cleaner since restart/health logic is already there — but it changes how the `source_hash` live-reload detect would work (currently `OrchestratorService` recreates the container on a code change; with a combined container we'd recreate the whole thing, which also bounces the gateway and drops in-flight egress connections). That may be acceptable given this is a dev-only concern, but worth confirming. If the supervise-tree approach is the right call, I'll proceed with: (1) restructure `Dockerfile.infra` as the shared gateway+orchestrator base, (2) add orchestrator to `gateway_init` supervision, (3) update `OrchestratorService` to start one `bot-bottle-infra` container, (4) update firecracker to `FROM bot-bottle-infra` + buildah.

didericis commented

2026-07-20 14:54:03 -04:00

@didericis-claude supervise-tree is the right approach. Also look for opportunities to consolidate logic between macos container, firecracker, and docker backends, as they should all be using the same structure now.

didericis-claude commented

2026-07-20 14:56:35 -04:00

Got it. Mapping out what I see across the three backends before writing the PRD.

Dockerfile restructuring plan:

Dockerfile.infra becomes the shared base: FROM bot-bottle-gateway + COPY --from bot-bottle-orchestrator (no buildah, that's Firecracker-only)
Firecracker gets a new Dockerfile.infra.fc (or similar) that does FROM bot-bottle-infra + buildah/crun/netavark — same net result, just now layered from the shared base
Dockerfile.orchestrator stays as the canonical orchestrator content definition (nothing changes there)

Supervise-tree: orchestrator process added to gateway_init, OrchestratorService starts one bot-bottle-infra container instead of two.

Consolidation opportunities I see:

CA cert polling loop — identical code in all three consolidated_launch.py files. Move to a shared utility.
Teardown sequence — all three do OrchestratorClient.teardown_bottle + deprovision_git_gate(transport, bottle_id); could be a shared helper that accepts a transport.
consolidated_launch.py shape — all three follow ensure-infra → allocate/register → provision → return context; macOS inverts the allocation timing (DHCP, source IP known only after agent runs) but the rest is the same. This is probably worth extracting a shared skeleton, with the macOS DHCP inversion as the explicit override point.
Gateway provisioning transport — already abstracted (GatewayTransport protocol + provision_git_gate/deprovision_git_gate). No change needed there.

Irreducible differences I'll leave in place:

macOS inverted registration order (DHCP means no --ip; agent starts before registration)
Per-transport impls (docker exec/cp vs container exec/cp vs SSH)
Infra lifecycle mechanics (containers vs microVM)

Will scope the PRD around these four items and open it once drafted.

Got it. Mapping out what I see across the three backends before writing the PRD. **Dockerfile restructuring plan:** - `Dockerfile.infra` becomes the shared base: `FROM bot-bottle-gateway` + `COPY --from bot-bottle-orchestrator` (no buildah, that's Firecracker-only) - Firecracker gets a new `Dockerfile.infra.fc` (or similar) that does `FROM bot-bottle-infra` + buildah/crun/netavark — same net result, just now layered from the shared base - `Dockerfile.orchestrator` stays as the canonical orchestrator content definition (nothing changes there) **Supervise-tree:** orchestrator process added to `gateway_init`, `OrchestratorService` starts one `bot-bottle-infra` container instead of two. **Consolidation opportunities I see:** 1. **CA cert polling loop** — identical code in all three `consolidated_launch.py` files. Move to a shared utility. 2. **Teardown sequence** — all three do `OrchestratorClient.teardown_bottle` + `deprovision_git_gate(transport, bottle_id)`; could be a shared helper that accepts a transport. 3. **`consolidated_launch.py` shape** — all three follow ensure-infra → allocate/register → provision → return context; macOS inverts the allocation timing (DHCP, source IP known only after agent runs) but the rest is the same. This is probably worth extracting a shared skeleton, with the macOS DHCP inversion as the explicit override point. 4. **Gateway provisioning transport** — already abstracted (`GatewayTransport` protocol + `provision_git_gate`/`deprovision_git_gate`). No change needed there. **Irreducible differences** I'll leave in place: - macOS inverted registration order (DHCP means no `--ip`; agent starts before registration) - Per-transport impls (docker exec/cp vs container exec/cp vs SSH) - Infra lifecycle mechanics (containers vs microVM) Will scope the PRD around these four items and open it once drafted.

👍 1

didericis added the

Priority

High

2

label 2026-07-20 14:57:25 -04:00

didericis referenced this issue from a commit

2026-07-20 14:59:31 -04:00

docs(prd): consolidate infra backend for docker

didericis-claude referenced a pull request that will close this issue

2026-07-20 14:59:49 -04:00

PRD: Consolidate infra backend for Docker #432