# Apple Container networking spike Issue: https://gitea.dideric.is/didericis/bot-bottle/issues/230 ## Summary Apple Container 1.0.0 on macOS 26 can support the core two-network sidecar shape, but not as a drop-in Docker Compose clone. The viable shape is: - agent container on one `--internal` host-only network; - sidecar bundle container on both the NAT egress network and the host-only agent network; - sidecar network flags ordered with the NAT network first, because Apple Container chooses the first network as the default route; - explicit DNS on the sidecar, because the tested NAT gateway routed packets but did not resolve DNS; - agent talks to sidecar by the sidecar's host-only-network IP, not by container name or host-published loopback alias. This is enough to unblock a cautious `macos-container` launch spike if the backend records inspect-derived IPs and avoids depending on Docker Compose-style aliases. It is not enough to reuse the Docker backend's service-name assumptions unchanged. ## Local Environment Tested on 2026-06-10: ```console $ sw_vers ProductName: macOS ProductVersion: 26.5.1 BuildVersion: 25F80 $ uname -m arm64 $ container --version container CLI version 1.0.0 (build: release, commit: ee848e3) $ container system version --format json [ { "appName": "container", "buildType": "release", "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779", "version": "1.0.0" }, { "appName": "container-apiserver", "buildType": "release", "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779", "version": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)" } ] $ container system status --format json { "apiServerAppName": "container-apiserver", "apiServerBuild": "release", "apiServerCommit": "ee848e3ebfd7c73b04dd419683be54fb450b8779", "apiServerVersion": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)", "appRoot": "/Users/didericis/Library/Application Support/com.apple.container/", "installRoot": "/usr/local/", "status": "running" } ``` Apple Container was installed from the official signed 1.0.0 GitHub release package, `container-1.0.0-installer-signed.pkg`. The package was signed by `Developer ID Installer: Apple Inc. - Containerization (UPBK2H6LZM)` and notarized by Apple. ## Commands Run Create the networks: ```bash container network create bb-spike-230-agent \ --internal \ --label bot-bottle.spike=apple-container-networking container network create bb-spike-230-egress \ --label bot-bottle.spike=apple-container-networking ``` `container network inspect bb-spike-230-agent bb-spike-230-egress` showed: ```json [ { "configuration": { "labels": {"bot-bottle.spike": "apple-container-networking"}, "mode": "hostOnly", "name": "bb-spike-230-agent", "plugin": "container-network-vmnet" }, "id": "bb-spike-230-agent", "status": { "ipv4Gateway": "192.168.128.1", "ipv4Subnet": "192.168.128.0/24" } }, { "configuration": { "labels": {"bot-bottle.spike": "apple-container-networking"}, "mode": "nat", "name": "bb-spike-230-egress", "plugin": "container-network-vmnet" }, "id": "bb-spike-230-egress", "status": { "ipv4Gateway": "192.168.66.1", "ipv4Subnet": "192.168.66.0/24" } } ] ``` Repeated `--network` flags are accepted. With the agent network first, the sidecar got two interfaces but the default route pointed at the host-only gateway, so egress failed: ```bash container run --name bb-spike-230-sidecar \ --label bot-bottle.spike=apple-container-networking \ --network bb-spike-230-agent \ --network bb-spike-230-egress \ --detach --rm docker.io/python:alpine \ sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0' container exec bb-spike-230-sidecar sh -c 'ip route && cat /etc/resolv.conf' ``` Observed: ```console default via 192.168.128.1 dev eth0 192.168.66.0/24 dev eth1 scope link src 192.168.66.3 192.168.128.0/24 dev eth0 scope link src 192.168.128.3 nameserver 192.168.128.1 ``` With the NAT network first and explicit DNS, the sidecar can egress: ```bash container run --name bb-spike-230-sidecar \ --label bot-bottle.spike=apple-container-networking \ --network bb-spike-230-egress \ --network bb-spike-230-agent \ --dns 1.1.1.1 \ --detach docker.io/python:alpine \ sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0' container exec bb-spike-230-sidecar sh -c 'ip route; cat /etc/resolv.conf; wget -T 8 -O- https://example.com' ``` Observed: ```console default via 192.168.66.1 dev eth0 192.168.66.0/24 dev eth0 scope link src 192.168.66.5 192.168.128.0/24 dev eth1 scope link src 192.168.128.7 nameserver 1.1.1.1 Connecting to example.com (172.66.147.243:443) ... 100% ``` Start an agent only on the host-only network: ```bash container run --name bb-spike-230-agent \ --label bot-bottle.spike=apple-container-networking \ --network bb-spike-230-agent \ --detach docker.io/alpine:latest sleep 600 ``` Agent network probes: ```bash container exec bb-spike-230-agent sh -c ' ip route cat /etc/resolv.conf wget -T 5 -O- http://192.168.128.7 wget -T 5 -O- http://bb-spike-230-sidecar || true ping -c 2 1.1.1.1 || true wget -T 5 -O- https://example.com || true ' ``` Observed: ```console default via 192.168.128.1 dev eth0 192.168.128.0/24 dev eth0 scope link src 192.168.128.8 nameserver 192.168.128.1 Connecting to 192.168.128.7 (192.168.128.7:80) ok wget: bad address 'bb-spike-230-sidecar' 2 packets transmitted, 0 packets received, 100% packet loss wget: bad address 'example.com' ``` Host-published loopback aliases work and are constrained to the bound alias on the host: ```bash container run --name bb-spike-230-sidecar-alias \ --label bot-bottle.spike=apple-container-networking \ --network bb-spike-230-egress \ --network bb-spike-230-agent \ --dns 1.1.1.1 \ --publish 127.0.0.31:18080:80 \ --detach docker.io/python:alpine \ sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0' curl -fsS --max-time 5 http://127.0.0.31:18080 curl -fsS --max-time 5 http://127.0.0.1:18080 lsof -nP -iTCP:18080 -sTCP:LISTEN ``` Observed: ```console $ curl -fsS --max-time 5 http://127.0.0.31:18080 ok $ curl -fsS --max-time 5 http://127.0.0.1:18080 curl: (7) Failed to connect to 127.0.0.1 port 18080 after 0 ms: Couldn't connect to server $ lsof -nP -iTCP:18080 -sTCP:LISTEN COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME container 17908 didericis 25u IPv4 ... 0t0 TCP 127.0.0.31:18080 (LISTEN) ``` The guest cannot reach that host loopback-published listener through the host-only gateway or through its own loopback address: ```bash container exec bb-spike-230-agent sh -c ' wget -T 5 -O- http://192.168.128.10 wget -T 5 -O- http://192.168.128.1:18080 || true wget -T 5 -O- http://127.0.0.31:18080 || true wget -T 5 -O- http://bb-spike-230-sidecar-alias || true ' ``` Observed: ```console Connecting to 192.168.128.10 (192.168.128.10:80) ok Connecting to 192.168.128.1:18080 (192.168.128.1:18080) wget: can't connect to remote host (192.168.128.1): Connection refused Connecting to 127.0.0.31:18080 (127.0.0.31:18080) wget: can't connect to remote host (127.0.0.31): Connection refused wget: bad address 'bb-spike-230-sidecar-alias' ``` ## Answers ### 1. Does `container network create --internal` prevent outbound internet access? Yes in this run. `--internal` produced a `hostOnly` network. An internal-only agent had a default route to the host-only gateway, but could not ping `1.1.1.1` and could not resolve or fetch `https://example.com`. ### 2. Can `container run` attach one container to multiple networks? Yes. Repeated `--network` flags produced multiple interfaces and the inspect JSON preserved both network attachments. Important caveat: network order matters. The first network became `eth0`, supplied the default route, and supplied `/etc/resolv.conf`. For a sidecar that needs internet egress, put the NAT network first and the internal agent network second. ### 3. Can the sidecar bundle sit on both an internal agent network and an egress-capable network? Yes. The sidecar had a NAT interface and a host-only interface. With the NAT network first and explicit DNS, it could fetch `https://example.com` while the agent on only the host-only network could not. ### 4. Can Apple Container provide stable network aliases or service discovery equivalent to Docker Compose aliases? Not by default in this run. The agent could not resolve `bb-spike-230-sidecar` or `bb-spike-230-sidecar-alias`, even though those were the container names and hostnames in inspect output. The agent could reach the sidecar by the sidecar's host-only-network IP. The backend should not assume Docker Compose-style aliases. It should read the sidecar's host-only IP from `container inspect` and inject that concrete endpoint into the agent environment/config, or run a small internal DNS/hosts-file setup as an explicit backend feature. ### 5. Can a published sidecar port bound to a per-bottle loopback alias be reached from another Apple Container guest and constrained to that alias? Host-side alias binding works and is constrained on the host: `127.0.0.31:18080` reached the sidecar, while `127.0.0.1:18080` failed. Guest-to-host-published-loopback did not work. From the agent, `192.168.128.1:18080` and `127.0.0.31:18080` both failed. For agent-to-sidecar traffic, use the sidecar's internal network IP rather than a host-published loopback alias. ### 6. What structured output is available for robust enumeration and cleanup? Confirmed structured output: - `container list --all --format json` - `container inspect ` as JSON - `container image inspect ` as JSON - `container network list --format json` - `container network inspect ` as JSON - `container system status --format json` - `container system version --format json` Useful fields observed: - containers: `id`, `configuration.labels`, `configuration.networks`, `configuration.publishedPorts`, `status.state`, `status.networks[].network`, `status.networks[].ipv4Address`, `status.networks[].ipv4Gateway`; - networks: `id`, `configuration.name`, `configuration.labels`, `configuration.mode`, `status.ipv4Gateway`, `status.ipv4Subnet`; - images: `id`, `configuration.name`, `configuration.descriptor`, `variants[].platform`, `variants[].size`. ### 7. Are labels supported on containers and networks enough to replace prefix-only discovery? Labels are present in container and network inspect/list JSON, so they are sufficient as metadata if the backend lists resources and filters client-side. I did not find or validate a server-side label filter for `container list` or `container network list`. ## Recommendation Proceed with a narrow `macos-container` launch prototype, but encode the Apple Container-specific constraints directly: - create one host-only agent network and one NAT egress network per bottle; - start the sidecar bundle with `--network ` before `--network `; - set sidecar DNS explicitly, ideally from the bottle/host policy rather than hardcoding a public resolver; - start the agent only on the host-only network; - discover the sidecar's host-only IP from `container inspect` and pass concrete URLs to the agent; - use host loopback publishing only for host-to-sidecar access, not guest-to-sidecar access; - enumerate and clean up by labels plus name prefixes until/unless the CLI adds label filters. Do not implement the backend as a direct clone of Docker Compose service aliases. That assumption failed in this run.