bot-bottle/docs/research/apple-container-networking-spike.md

Apple Container networking spike

Issue: #230

Summary

Apple Container 1.0.0 on macOS 26 can support the core two-network sidecar shape, but not as a drop-in Docker Compose clone.

The viable shape is:

• agent container on one --internal host-only network;
• sidecar bundle container on both the NAT egress network and the host-only agent network;
• sidecar network flags ordered with the NAT network first, because Apple Container chooses the first network as the default route;
• explicit DNS on the sidecar, because the tested NAT gateway routed packets but did not resolve DNS;
• agent talks to sidecar by the sidecar's host-only-network IP, not by container name or host-published loopback alias.

This is enough to unblock a cautious macos-container launch spike if the backend records inspect-derived IPs and avoids depending on Docker Compose-style aliases. It is not enough to reuse the Docker backend's service-name assumptions unchanged.

Local Environment

Tested on 2026-06-10:

$ sw_vers
ProductName:		macOS
ProductVersion:		26.5.1
BuildVersion:		25F80

$ uname -m
arm64

$ container --version
container CLI version 1.0.0 (build: release, commit: ee848e3)

$ container system version --format json
[
  {
    "appName": "container",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "1.0.0"
  },
  {
    "appName": "container-apiserver",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)"
  }
]

$ container system status --format json
{
  "apiServerAppName": "container-apiserver",
  "apiServerBuild": "release",
  "apiServerCommit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
  "apiServerVersion": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)",
  "appRoot": "/Users/didericis/Library/Application Support/com.apple.container/",
  "installRoot": "/usr/local/",
  "status": "running"
}

Apple Container was installed from the official signed 1.0.0 GitHub release package, container-1.0.0-installer-signed.pkg. The package was signed by Developer ID Installer: Apple Inc. - Containerization (UPBK2H6LZM) and notarized by Apple.

Commands Run

Create the networks:

container network create bb-spike-230-agent \
  --internal \
  --label bot-bottle.spike=apple-container-networking

container network create bb-spike-230-egress \
  --label bot-bottle.spike=apple-container-networking

container network inspect bb-spike-230-agent bb-spike-230-egress showed:

[
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "hostOnly",
      "name": "bb-spike-230-agent",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-agent",
    "status": {
      "ipv4Gateway": "192.168.128.1",
      "ipv4Subnet": "192.168.128.0/24"
    }
  },
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "nat",
      "name": "bb-spike-230-egress",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-egress",
    "status": {
      "ipv4Gateway": "192.168.66.1",
      "ipv4Subnet": "192.168.66.0/24"
    }
  }
]

Repeated --network flags are accepted. With the agent network first, the sidecar got two interfaces but the default route pointed at the host-only gateway, so egress failed:

container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --network bb-spike-230-egress \
  --detach --rm docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route && cat /etc/resolv.conf'

Observed:

default via 192.168.128.1 dev eth0
192.168.66.0/24 dev eth1 scope link src 192.168.66.3
192.168.128.0/24 dev eth0 scope link src 192.168.128.3
nameserver 192.168.128.1

With the NAT network first and explicit DNS, the sidecar can egress:

container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route; cat /etc/resolv.conf; wget -T 8 -O- https://example.com'

Observed:

default via 192.168.66.1 dev eth0
192.168.66.0/24 dev eth0 scope link src 192.168.66.5
192.168.128.0/24 dev eth1 scope link src 192.168.128.7
nameserver 1.1.1.1
Connecting to example.com (172.66.147.243:443)
... 100%

Start an agent only on the host-only network:

container run --name bb-spike-230-agent \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --detach docker.io/alpine:latest sleep 600

Agent network probes:

container exec bb-spike-230-agent sh -c '
  ip route
  cat /etc/resolv.conf
  wget -T 5 -O- http://192.168.128.7
  wget -T 5 -O- http://bb-spike-230-sidecar || true
  ping -c 2 1.1.1.1 || true
  wget -T 5 -O- https://example.com || true
'

Observed:

default via 192.168.128.1 dev eth0
192.168.128.0/24 dev eth0 scope link src 192.168.128.8
nameserver 192.168.128.1
Connecting to 192.168.128.7 (192.168.128.7:80)
ok
wget: bad address 'bb-spike-230-sidecar'
2 packets transmitted, 0 packets received, 100% packet loss
wget: bad address 'example.com'

Host-published loopback aliases work and are constrained to the bound alias on the host:

container run --name bb-spike-230-sidecar-alias \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --publish 127.0.0.31:18080:80 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

curl -fsS --max-time 5 http://127.0.0.31:18080
curl -fsS --max-time 5 http://127.0.0.1:18080
lsof -nP -iTCP:18080 -sTCP:LISTEN

Observed:

$ curl -fsS --max-time 5 http://127.0.0.31:18080
ok

$ curl -fsS --max-time 5 http://127.0.0.1:18080
curl: (7) Failed to connect to 127.0.0.1 port 18080 after 0 ms: Couldn't connect to server

$ lsof -nP -iTCP:18080 -sTCP:LISTEN
COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
container 17908 didericis   25u  IPv4 ...    0t0  TCP 127.0.0.31:18080 (LISTEN)

The guest cannot reach that host loopback-published listener through the host-only gateway or through its own loopback address:

container exec bb-spike-230-agent sh -c '
  wget -T 5 -O- http://192.168.128.10
  wget -T 5 -O- http://192.168.128.1:18080 || true
  wget -T 5 -O- http://127.0.0.31:18080 || true
  wget -T 5 -O- http://bb-spike-230-sidecar-alias || true
'

Observed:

Connecting to 192.168.128.10 (192.168.128.10:80)
ok
Connecting to 192.168.128.1:18080 (192.168.128.1:18080)
wget: can't connect to remote host (192.168.128.1): Connection refused
Connecting to 127.0.0.31:18080 (127.0.0.31:18080)
wget: can't connect to remote host (127.0.0.31): Connection refused
wget: bad address 'bb-spike-230-sidecar-alias'

Answers

1. Does container network create --internal prevent outbound internet access?

Yes in this run. --internal produced a hostOnly network. An internal-only agent had a default route to the host-only gateway, but could not ping 1.1.1.1 and could not resolve or fetch https://example.com.

2. Can container run attach one container to multiple networks?

Yes. Repeated --network flags produced multiple interfaces and the inspect JSON preserved both network attachments.

Important caveat: network order matters. The first network became eth0, supplied the default route, and supplied /etc/resolv.conf. For a sidecar that needs internet egress, put the NAT network first and the internal agent network second.

3. Can the sidecar bundle sit on both an internal agent network and an egress-capable network?

Yes. The sidecar had a NAT interface and a host-only interface. With the NAT network first and explicit DNS, it could fetch https://example.com while the agent on only the host-only network could not.

4. Can Apple Container provide stable network aliases or service discovery equivalent to Docker Compose aliases?

Not by default in this run. The agent could not resolve bb-spike-230-sidecar or bb-spike-230-sidecar-alias, even though those were the container names and hostnames in inspect output. The agent could reach the sidecar by the sidecar's host-only-network IP.

The backend should not assume Docker Compose-style aliases. It should read the sidecar's host-only IP from container inspect and inject that concrete endpoint into the agent environment/config, or run a small internal DNS/hosts-file setup as an explicit backend feature.

5. Can a published sidecar port bound to a per-bottle loopback alias be reached from another Apple Container guest and constrained to that alias?

Host-side alias binding works and is constrained on the host: 127.0.0.31:18080 reached the sidecar, while 127.0.0.1:18080 failed.

Guest-to-host-published-loopback did not work. From the agent, 192.168.128.1:18080 and 127.0.0.31:18080 both failed. For agent-to-sidecar traffic, use the sidecar's internal network IP rather than a host-published loopback alias.

6. What structured output is available for robust enumeration and cleanup?

Confirmed structured output:

• container list --all --format json
• container inspect <container...> as JSON
• container image inspect <image...> as JSON
• container network list --format json
• container network inspect <network...> as JSON
• container system status --format json
• container system version --format json

Useful fields observed:

• containers: id, configuration.labels, configuration.networks, configuration.publishedPorts, status.state, status.networks[].network, status.networks[].ipv4Address, status.networks[].ipv4Gateway;
• networks: id, configuration.name, configuration.labels, configuration.mode, status.ipv4Gateway, status.ipv4Subnet;
• images: id, configuration.name, configuration.descriptor, variants[].platform, variants[].size.

7. Are labels supported on containers and networks enough to replace prefix-only discovery?

Labels are present in container and network inspect/list JSON, so they are sufficient as metadata if the backend lists resources and filters client-side. I did not find or validate a server-side label filter for container list or container network list.

Recommendation

Proceed with a narrow macos-container launch prototype, but encode the Apple Container-specific constraints directly:

• create one host-only agent network and one NAT egress network per bottle;
• start the sidecar bundle with --network <egress> before --network <agent>;
• set sidecar DNS explicitly, ideally from the bottle/host policy rather than hardcoding a public resolver;
• start the agent only on the host-only network;
• discover the sidecar's host-only IP from container inspect and pass concrete URLs to the agent;
• use host loopback publishing only for host-to-sidecar access, not guest-to-sidecar access;
• enumerate and clean up by labels plus name prefixes until/unless the CLI adds label filters.

Do not implement the backend as a direct clone of Docker Compose service aliases. That assumption failed in this run.