diff --git a/docs/research/apple-container-networking-spike.md b/docs/research/apple-container-networking-spike.md new file mode 100644 index 0000000..e4fc405 --- /dev/null +++ b/docs/research/apple-container-networking-spike.md @@ -0,0 +1,360 @@ +# Apple Container networking spike + +Issue: https://gitea.dideric.is/didericis/bot-bottle/issues/230 + +## Summary + +Apple Container 1.0.0 on macOS 26 can support the core two-network +sidecar shape, but not as a drop-in Docker Compose clone. + +The viable shape is: + +- agent container on one `--internal` host-only network; +- sidecar bundle container on both the NAT egress network and the + host-only agent network; +- sidecar network flags ordered with the NAT network first, because + Apple Container chooses the first network as the default route; +- explicit DNS on the sidecar, because the tested NAT gateway routed + packets but did not resolve DNS; +- agent talks to sidecar by the sidecar's host-only-network IP, not by + container name or host-published loopback alias. + +This is enough to unblock a cautious `macos-container` launch spike if +the backend records inspect-derived IPs and avoids depending on Docker +Compose-style aliases. It is not enough to reuse the Docker backend's +service-name assumptions unchanged. + +## Local Environment + +Tested on 2026-06-10: + +```console +$ sw_vers +ProductName: macOS +ProductVersion: 26.5.1 +BuildVersion: 25F80 + +$ uname -m +arm64 + +$ container --version +container CLI version 1.0.0 (build: release, commit: ee848e3) + +$ container system version --format json +[ + { + "appName": "container", + "buildType": "release", + "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779", + "version": "1.0.0" + }, + { + "appName": "container-apiserver", + "buildType": "release", + "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779", + "version": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)" + } +] + +$ container system status --format json +{ + "apiServerAppName": "container-apiserver", + "apiServerBuild": "release", + "apiServerCommit": "ee848e3ebfd7c73b04dd419683be54fb450b8779", + "apiServerVersion": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)", + "appRoot": "/Users/didericis/Library/Application Support/com.apple.container/", + "installRoot": "/usr/local/", + "status": "running" +} +``` + +Apple Container was installed from the official signed 1.0.0 GitHub +release package, `container-1.0.0-installer-signed.pkg`. The package was +signed by `Developer ID Installer: Apple Inc. - Containerization +(UPBK2H6LZM)` and notarized by Apple. + +## Commands Run + +Create the networks: + +```bash +container network create bb-spike-230-agent \ + --internal \ + --label bot-bottle.spike=apple-container-networking + +container network create bb-spike-230-egress \ + --label bot-bottle.spike=apple-container-networking +``` + +`container network inspect bb-spike-230-agent bb-spike-230-egress` +showed: + +```json +[ + { + "configuration": { + "labels": {"bot-bottle.spike": "apple-container-networking"}, + "mode": "hostOnly", + "name": "bb-spike-230-agent", + "plugin": "container-network-vmnet" + }, + "id": "bb-spike-230-agent", + "status": { + "ipv4Gateway": "192.168.128.1", + "ipv4Subnet": "192.168.128.0/24" + } + }, + { + "configuration": { + "labels": {"bot-bottle.spike": "apple-container-networking"}, + "mode": "nat", + "name": "bb-spike-230-egress", + "plugin": "container-network-vmnet" + }, + "id": "bb-spike-230-egress", + "status": { + "ipv4Gateway": "192.168.66.1", + "ipv4Subnet": "192.168.66.0/24" + } + } +] +``` + +Repeated `--network` flags are accepted. With the agent network first, +the sidecar got two interfaces but the default route pointed at the +host-only gateway, so egress failed: + +```bash +container run --name bb-spike-230-sidecar \ + --label bot-bottle.spike=apple-container-networking \ + --network bb-spike-230-agent \ + --network bb-spike-230-egress \ + --detach --rm docker.io/python:alpine \ + sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0' + +container exec bb-spike-230-sidecar sh -c 'ip route && cat /etc/resolv.conf' +``` + +Observed: + +```console +default via 192.168.128.1 dev eth0 +192.168.66.0/24 dev eth1 scope link src 192.168.66.3 +192.168.128.0/24 dev eth0 scope link src 192.168.128.3 +nameserver 192.168.128.1 +``` + +With the NAT network first and explicit DNS, the sidecar can egress: + +```bash +container run --name bb-spike-230-sidecar \ + --label bot-bottle.spike=apple-container-networking \ + --network bb-spike-230-egress \ + --network bb-spike-230-agent \ + --dns 1.1.1.1 \ + --detach docker.io/python:alpine \ + sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0' + +container exec bb-spike-230-sidecar sh -c 'ip route; cat /etc/resolv.conf; wget -T 8 -O- https://example.com' +``` + +Observed: + +```console +default via 192.168.66.1 dev eth0 +192.168.66.0/24 dev eth0 scope link src 192.168.66.5 +192.168.128.0/24 dev eth1 scope link src 192.168.128.7 +nameserver 1.1.1.1 +Connecting to example.com (172.66.147.243:443) +... 100% +``` + +Start an agent only on the host-only network: + +```bash +container run --name bb-spike-230-agent \ + --label bot-bottle.spike=apple-container-networking \ + --network bb-spike-230-agent \ + --detach docker.io/alpine:latest sleep 600 +``` + +Agent network probes: + +```bash +container exec bb-spike-230-agent sh -c ' + ip route + cat /etc/resolv.conf + wget -T 5 -O- http://192.168.128.7 + wget -T 5 -O- http://bb-spike-230-sidecar || true + ping -c 2 1.1.1.1 || true + wget -T 5 -O- https://example.com || true +' +``` + +Observed: + +```console +default via 192.168.128.1 dev eth0 +192.168.128.0/24 dev eth0 scope link src 192.168.128.8 +nameserver 192.168.128.1 +Connecting to 192.168.128.7 (192.168.128.7:80) +ok +wget: bad address 'bb-spike-230-sidecar' +2 packets transmitted, 0 packets received, 100% packet loss +wget: bad address 'example.com' +``` + +Host-published loopback aliases work and are constrained to the bound +alias on the host: + +```bash +container run --name bb-spike-230-sidecar-alias \ + --label bot-bottle.spike=apple-container-networking \ + --network bb-spike-230-egress \ + --network bb-spike-230-agent \ + --dns 1.1.1.1 \ + --publish 127.0.0.31:18080:80 \ + --detach docker.io/python:alpine \ + sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0' + +curl -fsS --max-time 5 http://127.0.0.31:18080 +curl -fsS --max-time 5 http://127.0.0.1:18080 +lsof -nP -iTCP:18080 -sTCP:LISTEN +``` + +Observed: + +```console +$ curl -fsS --max-time 5 http://127.0.0.31:18080 +ok + +$ curl -fsS --max-time 5 http://127.0.0.1:18080 +curl: (7) Failed to connect to 127.0.0.1 port 18080 after 0 ms: Couldn't connect to server + +$ lsof -nP -iTCP:18080 -sTCP:LISTEN +COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME +container 17908 didericis 25u IPv4 ... 0t0 TCP 127.0.0.31:18080 (LISTEN) +``` + +The guest cannot reach that host loopback-published listener through +the host-only gateway or through its own loopback address: + +```bash +container exec bb-spike-230-agent sh -c ' + wget -T 5 -O- http://192.168.128.10 + wget -T 5 -O- http://192.168.128.1:18080 || true + wget -T 5 -O- http://127.0.0.31:18080 || true + wget -T 5 -O- http://bb-spike-230-sidecar-alias || true +' +``` + +Observed: + +```console +Connecting to 192.168.128.10 (192.168.128.10:80) +ok +Connecting to 192.168.128.1:18080 (192.168.128.1:18080) +wget: can't connect to remote host (192.168.128.1): Connection refused +Connecting to 127.0.0.31:18080 (127.0.0.31:18080) +wget: can't connect to remote host (127.0.0.31): Connection refused +wget: bad address 'bb-spike-230-sidecar-alias' +``` + +## Answers + +### 1. Does `container network create --internal` prevent outbound internet access? + +Yes in this run. `--internal` produced a `hostOnly` network. An +internal-only agent had a default route to the host-only gateway, but +could not ping `1.1.1.1` and could not resolve or fetch +`https://example.com`. + +### 2. Can `container run` attach one container to multiple networks? + +Yes. Repeated `--network` flags produced multiple interfaces and the +inspect JSON preserved both network attachments. + +Important caveat: network order matters. The first network became +`eth0`, supplied the default route, and supplied `/etc/resolv.conf`. +For a sidecar that needs internet egress, put the NAT network first and +the internal agent network second. + +### 3. Can the sidecar bundle sit on both an internal agent network and an egress-capable network? + +Yes. The sidecar had a NAT interface and a host-only interface. With the +NAT network first and explicit DNS, it could fetch `https://example.com` +while the agent on only the host-only network could not. + +### 4. Can Apple Container provide stable network aliases or service discovery equivalent to Docker Compose aliases? + +Not by default in this run. The agent could not resolve +`bb-spike-230-sidecar` or `bb-spike-230-sidecar-alias`, even though +those were the container names and hostnames in inspect output. The +agent could reach the sidecar by the sidecar's host-only-network IP. + +The backend should not assume Docker Compose-style aliases. It should +read the sidecar's host-only IP from `container inspect` and inject +that concrete endpoint into the agent environment/config, or run a +small internal DNS/hosts-file setup as an explicit backend feature. + +### 5. Can a published sidecar port bound to a per-bottle loopback alias be reached from another Apple Container guest and constrained to that alias? + +Host-side alias binding works and is constrained on the host: +`127.0.0.31:18080` reached the sidecar, while `127.0.0.1:18080` failed. + +Guest-to-host-published-loopback did not work. From the agent, +`192.168.128.1:18080` and `127.0.0.31:18080` both failed. For +agent-to-sidecar traffic, use the sidecar's internal network IP rather +than a host-published loopback alias. + +### 6. What structured output is available for robust enumeration and cleanup? + +Confirmed structured output: + +- `container list --all --format json` +- `container inspect ` as JSON +- `container image inspect ` as JSON +- `container network list --format json` +- `container network inspect ` as JSON +- `container system status --format json` +- `container system version --format json` + +Useful fields observed: + +- containers: `id`, `configuration.labels`, + `configuration.networks`, `configuration.publishedPorts`, + `status.state`, `status.networks[].network`, + `status.networks[].ipv4Address`, `status.networks[].ipv4Gateway`; +- networks: `id`, `configuration.name`, `configuration.labels`, + `configuration.mode`, `status.ipv4Gateway`, `status.ipv4Subnet`; +- images: `id`, `configuration.name`, `configuration.descriptor`, + `variants[].platform`, `variants[].size`. + +### 7. Are labels supported on containers and networks enough to replace prefix-only discovery? + +Labels are present in container and network inspect/list JSON, so they +are sufficient as metadata if the backend lists resources and filters +client-side. I did not find or validate a server-side label filter for +`container list` or `container network list`. + +## Recommendation + +Proceed with a narrow `macos-container` launch prototype, but encode +the Apple Container-specific constraints directly: + +- create one host-only agent network and one NAT egress network per + bottle; +- start the sidecar bundle with `--network ` before + `--network `; +- set sidecar DNS explicitly, ideally from the bottle/host policy + rather than hardcoding a public resolver; +- start the agent only on the host-only network; +- discover the sidecar's host-only IP from `container inspect` and pass + concrete URLs to the agent; +- use host loopback publishing only for host-to-sidecar access, not + guest-to-sidecar access; +- enumerate and clean up by labels plus name prefixes until/unless the + CLI adds label filters. + +Do not implement the backend as a direct clone of Docker Compose +service aliases. That assumption failed in this run.