Run Docker/Compose inside bottles without bypassing isolation #392
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: didericis/bot-bottle#392
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Problem
Agent tasks routinely involve containers —
docker compose upto verify a scaffolded stack, a throwaway database for a test run, building an image to check the Dockerfile works. Bottles today have no way to do any of that, so those tasks either fail or get done outside the bottle, which defeats the point.The core need: run containers inside a bottle, without weakening what a bottle is.
Status
Spiked in #449 (closed unmerged; branch
spike/rootless-docker-macosis pushed). Findings in this comment anddocs/research/rootless-docker-in-apple-container-spike.md.Short version:
CAP_SYS_ADMIN, which the kernel requires to write a multi-rangeuid_mapvianewuidmap. No packaging fix exists; the only way through is granting the bottleCAP_SYS_ADMIN, which is close to root in practical terms.docker compose up -d --waitpulls an image, starts a service, and serves from the workspace inside a registered bottle. The agent still typesdockeranddocker compose.So the mechanism question looks settled: guest-local rootless podman, exposing a Docker-compatible socket.
Hard non-goal
Do not mount the host Docker socket (
/var/run/docker.sock), on any backend.Open questions
Is
docker_access: truethe right interface — or an interface we want at all?The flag is hypothetical. It exists in the spike branch because the spike needed some switch, not because we decided bottles should have one. Arguments both ways:
/dev/fuseand/dev/net/tun(which the host must do as root, and which the majority of bottles that never touch containers shouldn't get).If we keep a flag,
docker_accessis the wrong name — it no longer implies Docker and grants no "access" to anything on the host. Something likenested_containersdescribes what it does.We'll experiment before committing to either.
Other open questions:
Authorizationand both need a client-fetched per-scope bearer token. quay.io, mcr.microsoft.com, registry.k8s.io, and unauthenticated internal registries work today. Options: a gateway that performs the registry token dance itself, a per-route opt-in to preserve client auth, or pre-seeding images at provision.rootinside a nested container is the agent user outside it. This is by design: the goal is to run project Docker dependencies inside an existing bottle, not to add another isolation layer. Should be stated wherever the feature is documented — it's for build/test use, not sandboxing untrusted code.Blocked on
0for everydocker runthrough the compat API, so agent scripts would see failures as successes.docs/research/egress-proxy-oom-on-large-downloads.md— a large pull OOM-kills the shared egress proxy and takes egress down for every bottle. Pre-existing, but pulling images makes it easy to hit.Backend scope
The working implementation is on
macos-container, not thedockerbackend this issue originally scoped itself to. Worth deciding whether the docker and firecracker backends need an equivalent or whether this is macOS-only for now.@didericis-codex reasses macos docker access story now (macos containers are live)
Reassessed now that
macos-containeris live.The socket mount is technically plausible now: Apple
containersupports host bind mounts, and its documented--sshbehavior is itself a host Unix-socket mount. We could add the equivalent of:However, I don't think that gives us a sound macOS Docker-access story:
.:/appare sent to Docker Desktop and resolved in the daemon's host context, not in the guest, so they will be missing or point at a stale/different host tree.docker compose up && curl localhost:...verification flows therefore still would not behave like native Docker access.So my recommendation is:
macos-container, rejectdocker_access: truewith a clear backend-specific error for now rather than silently providing misleading/boundary-breaking behavior.If guest-local DinD proves infeasible with Apple's current kernel/runtime, a narrowly scoped host-side Docker broker could be considered, but it would need path/context translation and strict lifecycle/API restrictions; raw Docker Desktop socket forwarding should not be the fallback.
@didericis-codex do a quick research spike here on running docker within macos containers/see if anyone has done it and made a guide yet
Quick research spike: yes, someone has now documented and tested real Docker Engine + Compose inside Apple
container, and the result is promising:container1.0.0. It uses a persistentcontainer machinebuilt from Ubuntu + systemd, installsdocker.ioanddocker-compose-v2, and runs the real daemon inside that VM. The author verifiesdocker run, published ports, Compose bridge networking, service-name DNS, and daemon restart.container system kernel set --recommended --force, which installs Apple's recommended Kata kernel with the needed netfilter support. This matches Apple's own Containerization documentation, which recommends the Kata kernel as a container-oriented kernel with the required configuration.docker-for-apple-containerproject translating a subset of Docker/Compose commands to Apple's host CLI. Interesting, but it is not a fit for an untrusted bottle: the shim needs access to the hostcontainerCLI/API, supports only a subset, and would let the agent orchestrate resources outside its guest.Verdict: guest-local Docker is demonstrably feasible, but the proven recipe uses
container machine, not an ordinarycontainer runcontainer. For bot-bottle that likely means a dedicateddocker_accesslaunch variant rather than a small argv flag:container run);The guide removes the main feasibility concern. The remaining work is a small live-Mac spike against bot-bottle's networking: create an ephemeral machine on/behind the shared gateway, verify proxied pulls/builds,
docker compose up, service DNS, guest-local bind mounts,curl localhost:<published-port>, and teardown. I would pursue that before either socket forwarding or a host broker.@didericis-codex before implementing that spike, evaluate the security considerations. Also weigh pros and cons of running
container machineby default (if it provides more capabilities without significant security boundary concerns and isn't significantly slower/bigger when spinning up images might we worth using by default)Agreed. Before touching the live-Mac spike, I’ll treat this as an architecture/security decision rather than only a Docker feasibility test.
I’ll compare the current
container runmodel with one ephemeralcontainer machineper bottle across:I’ll explicitly evaluate both designs: machines only for
docker_accessbottles, andcontainer machineas the default macOS backend. If the machine model preserves the security boundary and the measured startup/resource cost is modest, making it the default may be cleaner than maintaining two substantially different macOS launch paths. If it weakens attribution/egress guarantees or carries material per-bottle overhead, it should remain an opt-in variant.The next deliverable here will be that threat-model/tradeoff assessment, with measurements where possible; the implementation spike comes after it.
Security/architecture assessment completed. My conclusion is do not make
container machinethe default, and do not implement the Docker spike until we can prove host-enforced network confinement for machines.What changes—and what does not
Both ordinary Apple
container runworkloads and container machines already have a Virtualization.framework VM boundary, so moving to a machine does not obviously add a stronger host-isolation boundary. The gain is a fuller, persistent Linux environment:/sbin/init, systemd services, mutable persistent storage, and a kernel suitable for Docker networking.That fuller environment materially increases the guest privilege surface. An agent able to use a guest-local Docker daemon should be treated as root-equivalent inside its entire machine: it can launch privileged children, mount the machine filesystem, alter guest routing/firewall state, create raw sockets, and retain state in Docker images/volumes. This is acceptable only if the outer machine remains a disposable VM with no sensitive host mounts and host-enforced egress.
It is still much safer than forwarding Docker Desktop's socket: guest Docker authority ends at the machine boundary and guest paths/localhost have the expected semantics. It must not receive the host Docker socket or host
containercontrol API.Current blockers against bot-bottle's invariants
The current macOS backend launches an agent with:
--network <bot-bottle-host-only-network>;--cap-drop CAP_NET_RAW;bot-bottle-*containers and networks.Those controls are not available on the documented
container machine createinterface. Apple's current command reference exposes CPU, memory, kernel, nested virtualization, andhome-mount, but no custom network, DNS, mount list, capability drop, read-only rootfs, or labels. See Apple's machine command reference.Two defaults are unsafe for bot-bottle:
home-mountdefaults torw, exposing the operator's entire macOS home. We would have to require--home-mount none; read-only is still excessive because it exposes credentials and source.More importantly, without custom-network attachment, a machine cannot currently be placed on the backend's gateway-only network. Docker/root inside the machine can also use raw sockets and change its guest network state, so the present “drop NET_RAW, source IP is unforgeable” proof does not carry over. The identity token is useful defense in depth, but the architecture explicitly requires the source-IP invariant too; it is not a substitute for host-enforced confinement.
Putting the gateway inside the same machine would not solve this: an agent with Docker/root-equivalent control could disable or bypass it. A safe design needs the outer host/runtime to enforce that the machine has no route except to a separately trusted gateway.
Default-machine tradeoff
Benefits
localhostall live in the same guest.machine rmas stopping and deleting the machine, and the machine guide describes OCI-based machine images and init support.Costs and regressions
container list, notcontainer machine ls.Recommendation
container runas the macOS default. It has the smaller privilege surface and already satisfies bot-bottle's tested attribution/egress model.container machineas a future opt-in implementation fordocker_access, not a default, and gate even the live spike on one prerequisite: demonstrate a host-enforced host-only/custom-network path for machines (or an equivalent outer firewall design that the guest cannot change).home-mount=none, explicit CPU/RAM/disk limits, no nested virtualization, no host control sockets, a pinned netfilter-capable kernel, and one ephemeral machine per bottle.Measurements still required on the live Mac
I cannot produce honest startup/RSS/disk numbers from this Linux runner because Apple
containerrequires Apple Silicon/macOS. If the networking prerequisite is resolved, the spike should record, for bothcontainer runandcontainer machine, ten warm and cold samples of create-to-first-exec, peak and settled host memory, initial disk allocation, post-Docker-install disk, post-compose updisk, and deletion time/residual bytes.Decision for now: machines only remain worth exploring for explicit Docker access; they are not suitable as the default backend with the current CLI/security controls.
@didericis-codex Is there a way we could create a container machine just for running docker, and then attach the docker socket from just that machine to a running container? That would theoretically mean agent bottles can keep their current network boundaries and docker could stay container, correct? Only issue there is how to secure the network for the machine while also making it accessible to the container, which gets into the same network escape issues, so maybe that doesn't solve anything.
Yes—that is a materially better architecture than turning the agent itself into a machine, but it only solves part of the problem.
The shape would be:
The agent keeps its current
--networkandCAP_NET_RAWrestrictions. Possession of the Docker socket makes it root-equivalent only inside the disposable Docker machine, not inside the agent VM or on macOS. That preserves the important host boundary and avoids exposing Docker Desktop or the hostcontainerAPI.I would not literally mount the machine's Unix socket into the agent, though. These are separate VMs, and the documented machine interface has no socket-publishing equivalent. It would likely need an authenticated TCP Docker endpoint (mTLS, unique per bottle) reachable over a deliberately constructed network or through a narrow host-side relay.
Your concern is the remaining hard blocker: the Docker machine must itself be host-confined. Every Docker child can otherwise egress directly, bypassing bot-bottle's gateway and DLP. Because the agent controls the daemon, we must assume it can run privileged children and modify all networking inside that machine; confinement therefore has to be outside the machine. The current
container machineCLI still does not expose the custom-network attachment we use for agent bottles.There are also two semantic gaps:
localhost. We would need discovery and forwarding, or document remote-daemon semantics instead of pretending it is local Docker.So: yes, this preserves the existing agent boundary and is the best machine-based direction, but it is closer to a per-bottle remote Docker builder/service than ordinary socket sharing. It becomes viable if we can provide:
If Apple adds machine network selection—or we can enforce the equivalent safely at the host layer—I would spike this design before a machine-as-agent design. Until then, it runs into the same outer-network escape problem you identified.
@didericis-codex do a research spike on typical solutions when needing to manage docker containers from within another container. I'm guessing connecting to a remote docker instance is the typical solution. For context, main reason this is high priority is that at least for me, docker is an essential tool for most projects/bot-bottle is nerfed without them. This is already affecting my ability to do a proper containerized build of the platform repo. Alternatively could look into other container solutions we could run from within the macos container that might not have the same issues. But we want some equivalent to
docker composewe can run within the agent container without bypassing our secure boundaries.Understood, and that priority changes the weighting: this is not an occasional privileged escape hatch; Docker/Compose-equivalent project builds are a core bottle capability.
I’ll research the established patterns rather than assuming remote Docker is the answer. The comparison will cover:
docker compose uprequirement;The hard acceptance criterion will be: the agent can run a normal containerized project workflow—including Compose services, builds, service DNS, bind-mounted source, and localhost/published-port checks—without gaining access to macOS, other bottles, the host runtime, or an unfiltered egress path.
I’ll optimize for a useful near-term path, not only the architecturally ideal end state. If a per-bottle remote worker is the safest practical option, I’ll identify the smallest implementation that unblocks the platform build and clearly separate required security controls from later ergonomics.
Research spike complete. The main finding is that remote Docker is common, but rootless Docker inside the existing Apple-container VM is the best first spike for bot-bottle. It is the only option that plausibly preserves normal local Compose semantics and the current outer security boundary without introducing a second-machine filesystem/port translation layer.
Established patterns
1. Host socket sharing (“Docker-outside-of-Docker”)
Mounting
/var/run/docker.sockis the simplest and most common CI/development pattern. It has native CLI/Compose compatibility and shares the host cache.It is unacceptable here. Docker explicitly warns that remote/socket access can grant root access to the daemon host. An agent could start privileged containers, mount host paths, inspect other workloads, and alter bot-bottle infrastructure. An API-filtering proxy is not a reliable repair: enough Docker API to support arbitrary Compose files, builds, mounts, and privileged options is effectively arbitrary execution on the daemon host.
Verdict: reject for both Docker Desktop and any shared daemon.
2. Privileged Docker-in-Docker
The official
docker:dindimage runs an isolated daemon and gives correct local paths, ports, networks, and Compose behavior. However, the official recipe requires--privileged; Docker's documentation explicitly cautions that this grants full access to the containing environment. See the official Docker image documentation.Inside a dedicated VM this is less catastrophic than privileged DinD on a shared Linux host, but granting the agent rootful Docker requires the outer guest capabilities we intentionally removed. In particular, it undermines the current
CAP_NET_RAW/source-IP attribution proof and lets the agent reconfigure the guest network.Verdict: do not use in the current shared-network agent VM.
3. Per-bottle remote Docker worker
Docker natively supports remote daemons over SSH or mutually authenticated TLS, and Compose supports deployment to a remote Docker host. See Docker's daemon access guidance and remote Compose documentation.
This is the standard answer when the client must not control its own host. A disposable worker VM plus unique per-bottle mTLS credentials would safely scope “Docker root” to that VM.
The ergonomics are not transparent, though. Docker documents that bind mounts are resolved on the daemon host, not the client, and cannot mount client files. Published ports also terminate on the worker rather than the agent's localhost. See Docker bind-mount constraints. We would need workspace synchronization, path parity, port forwarding, worker network confinement, and orphan cleanup.
Verdict: sound fallback architecture, but larger and less compatible than it initially appears.
4. Rootless Docker inside the existing agent VM
Docker rootless mode runs both
dockerdand its children inside a user namespace as a non-root user. Its documented prerequisites arenewuidmap/newgidmapand subordinate UID/GID ranges; it does not require a rootful daemon. See Docker rootless mode.This is a strong fit for bot-bottle:
docker compose up, relative bind mounts, service DNS, andcurl localhost:<port>retain normal local semantics.nodeuser in the outer guest.CAP_NET_ADMIN,CAP_NET_RAW,--privileged, host mounts, or host runtime sockets.There are real limitations: privileged ports below 1024 need extra setup, and Docker says cgroup resource flags require cgroup v2 plus systemd delegation. We can initially reject privileged port mappings and rely on the outer bottle's CPU/RAM boundary even if per-inner-container limits are unavailable. See Docker's rootless limitations.
The unknown is Apple Container kernel/runtime compatibility: unprivileged user namespaces, subordinate-ID helpers, overlay/fuse-overlayfs, cgroup behavior, and rootless networking must be tested. The documented working rootful-Docker machine recipe used the recommended Kata kernel for netfilter. RootlessKit may need fewer outer privileges, but this is not proven on our current image/kernel.
Verdict: first implementation spike.
5. Rootless Podman + Compose
Podman is daemonless and designed for rootless operation. Root inside a child maps to the invoking unprivileged user, and Podman supports rootless bind mounts and user-mode networking. See Podman's rootless tutorial.
There are two Compose routes:
podman-compose, an independent Compose implementation focused on rootless/daemonless operation;This has similar kernel prerequisites to rootless Docker and potentially a smaller always-running surface, but compatibility is weaker:
podman-composeis not Docker Compose, and the Docker API layer is a compatibility subset. Since the requirement is “normal project Docker/Compose workflows,” changing engines introduces avoidable failures in real repositories.Verdict: second spike/fallback if rootless Docker fails for a specific technical reason.
6. BuildKit, Buildah, Kaniko, nerdctl/containerd
Rootless/daemonless BuildKit is excellent for Dockerfile builds and can run remotely over TLS. Its official rootless container recipe still needs relaxed seccomp/AppArmor/system-path restrictions, and it does not provide the runtime half of Compose. See BuildKit's official documentation.
Kaniko/Buildah similarly address image builds, not
compose up, service DNS, or published-port testing. Nerdctl needs containerd and recreates most of the nested-runtime problem.Verdict: useful future build-only optimization, not a solution to the stated requirement.
7. Sysbox/system-container runtimes
Sysbox makes nested rootful Docker safer by changing the outer host runtime. Apple Container does not expose Sysbox as a runtime, so adopting it would mean replacing the macOS backend rather than adding a bottle capability.
Verdict: not applicable.
Recommended near-term spike
Implement a throwaway, live-Mac proof before changing the manifest:
uidmap, and storage/network helpers.node.container runpath with the same custom network and--cap-drop CAP_NET_RAW; add no capabilities.dockerd-rootless.shasnodevia the normal token-bearing exec path.DOCKER_HOST=unix:///run/user/<uid>/docker.sock.Acceptance/security tests
The spike only passes if all of these hold:
docker buildanddocker compose buildcan pull through the gateway.docker compose upprovides service-name DNS..:/workspacesees and can update the bottle's workspace.curl localhost:<published-high-port>reaches a service.RUNfails without proxy configuration;--privilegedcontainer cannot access macOS, the ApplecontainerAPI, other bottles, the gateway control plane, or spoof another outer source IP;Fail closed if rootless Docker requires adding outer
NET_ADMIN,NET_RAW,SYS_ADMIN,--privileged, a host socket, or broad host mounts. In that case, test rootless Podman under the same no-new-capabilities rule. If both fail, proceed to the per-bottle remote worker design.Product recommendation
Keep the manifest intent as
docker_access: true, but make its backend implementation platform-specific:This path directly targets the urgent platform-repository build need while preserving the rule that no agent gets host runtime authority or an egress path outside bot-bottle's gateway.
Docker socket access for agent containers (docker_access bottle flag)to Run Docker/Compose inside bottles without bypassing isolationSpike outcome
Spiked in #449 (now closed unmerged; branch
spike/rootless-docker-macosis pushed and reusable). Write-ups:docs/research/rootless-docker-in-apple-container-spike.mdanddocs/research/egress-proxy-oom-on-large-downloads.md.What actually works
Not rootless Docker — that turns out to be impossible on the macOS backend. Apple Container's capability bounding set omits
CAP_SYS_ADMIN, which the kernel requires to write a multi-rangeuid_mapvianewuidmap. Nothing in packaging can route around it; the only fix is granting the bottleCAP_SYS_ADMIN, which is close to root in practical terms and would undercut the containment this issue is about. (Same constraint that makes upstream'sdind-rootlessneed--privileged.)Rootless podman does work, with zero added capabilities. With no subordinate UID range configured it falls back to a single-UID self-mapping that an unprivileged process can write itself, so
newuidmapis never invoked. Verified live on macOS 26.5.1 / Apple Container 1.0.0:docker compose up -d --waitpulls an image, starts a service, and serves the workspace on a published port — inside a registered bottle, with no host socket and no privilege grant. The agent still typesdockeranddocker compose; they talk to podman's Docker-compatible API socket.Note the backend divergence from this issue's non-goals: the working implementation is on
macos-container, not thedockerbackend.What "docker access" would mean in practice
Four constraints a consumer of this feature has to accept, none of them hypothetical:
rootinside is the agent user outside. This is by design: the goal is to run project Docker dependencies inside an existing bottle, not to add another isolation layer. Fine as a build/test workflow; not suitable for sandboxing untrusted code.Authorizationby design, and both require a client-fetched per-scope bearer token. Measured: token fetch200, manifest request with a valid token401, manifest with no auth401— identical. quay.io, mcr.microsoft.com, registry.k8s.io, and unauthenticated internal registries work today with a plain route.wget: bad address 'quay.io'). Compose can run images and serve locally; services that need to call out need proxy env, the CA, and DNS wired in first. Fail-closed, which is the right default, but it bounds what this feature currently delivers.mitmdumpto ~1GB RSS and the cgroup killed it. The gateway is a per-host singleton with no restart-on-death, so one bottle's pull removes egress for every bottle until it's recreated. This is a pre-existing defect that pulling images makes easy to hit; it is not specific to containers.Suggested reframing
If this issue stays open, it's worth rewriting around what the spike established:
macos-container, exposing a Docker-compatible socket.docker run, so agent scripts see failures as successes), plus a decision on the registry-auth strip and the egress-proxy OOM.@didericis-claude should add a flag for egress routes that turns off authorization stripping from certain routes. Can add it for docker hub and ghcr in bottle configs that use docker. Create a new PR for this/reference this issue.
Sounds right. The flag belongs on individual egress route entries — something like
strip_auth: false(or an explicitpreserve_auth: true) — so the gateway can make a per-route decision rather than applying a global policy. Docker Hub (registry-1.docker.io) and GHCR (ghcr.io) would get it set in whatever route block is emitted for bottles with nested container access.Opening a PR now.
@didericis-claude this commit was incorrect ->
ccd987a501. Issue is not fully close until we verify that we can use podman to run docker within a bottle. The last outstanding issue should be this:RE this issue:
Check to see if we've filed a new issue for it (thought we have). If we didn't, file one. But should not block the pr. We should simply turn off dlp for the image registries in the egress config.
So acceptance criteria for final PR to close this issue:
bot-bottle-devhas been updated so the egress options for docker registries are updated (do not strip authorization, do not do dlp), and once the dns issue is resolved (skip initially/make sure it wasn't a phantom issue), you should be able to clone thebot-bottle-platformrepo from within a bottle (along with it's submodules) and run the integration tests in the bottle withdocker composeConfirmed — the
Closes #392inccd987a(PR #453, thepreserve_authflag) auto-closed this on merge, and that was premature.preserve_authis one prerequisite, not the whole feature: nothing has verified podman-drivendocker composeend-to-end inside a bottle yet, and the nested-container DNS/outbound gap is still open. Reopened.OOM: no separate issue existed — filed as #455 (
Kind/Bug,Priority/High), split out from the finding here with thedmesg/RSS evidence and the two fix options (restart-on-death vs. cap scanned-body size). Per your note it does not block this PR: we sidestep it on the registry hosts by disabling DLP there (below), and #455 tracks the general fix.Egress config for docker registries (the
bot-bottle-devchange): each registry route getsregistry-1.docker.io+auth.docker.io,ghcr.io— needpreserve_auth: true(token dance) and DLP off.quay.io,mcr.microsoft.com,registry.k8s.io— already reachable on a plain route; just add DLP off so large pulls don't hit #455.(The manifest already supports both:
preserve_authfrom #453 anddlp.{outbound,inbound}_detectors: falseinmanifest_egress.py.)DNS gap (last real blocker per your comment): I'll treat "verify it wasn't a phantom" as step one — the earlier
wget: bad address 'quay.io'was from inside a nested container, whose resolver isn't wired to the bottle's. Before adding any plumbing I'll confirm whether podman's default netns/aardvark-dnsresolves once the gateway route + CA + proxy env are present in the container, and only wire DNS/proxy through if it's genuinely broken.Acceptance criteria for the closing PR (must run on the macOS/Apple-container host — I can't exercise Apple
containerfrom the Linux CI sandbox, so I'll run these live and paste results):bot-bottle-devegress updated: registry routespreserve_auth: true+ DLP offbot-bottle-platformwith submodules from inside a bottledocker composeStill gated on #451 (trixie base bump) — bookworm's podman 4.3.1 returns exit
0for everydocker runthrough the compat API, so the tests above would report false passes until that lands.