bot-bottle/docs/research/apple-container-networking-spike.md

Apple Container networking spike

Issue: #230

Summary

Apple Container 1.0.0 on macOS 26 can support the core two-network sidecar shape, but not as a drop-in Docker Compose clone.

The viable shape is:

• agent container on one --internal host-only network;
• sidecar bundle container on both the NAT egress network and the host-only agent network;
• sidecar network flags ordered with the NAT network first, because Apple Container chooses the first network as the default route;
• explicit DNS on the sidecar, because the tested NAT gateway routed packets but did not resolve DNS;
• agent talks to sidecar by the sidecar's host-only-network IP, not by container name or host-published loopback alias.

This is enough to unblock a cautious macos-container launch spike if the backend records inspect-derived IPs and avoids depending on Docker Compose-style aliases. It is not enough to reuse the Docker backend's service-name assumptions unchanged.

Local Environment

Tested on 2026-06-10:

$ sw_vers
ProductName:		macOS
ProductVersion:		26.5.1
BuildVersion:		25F80

$ uname -m
arm64

$ container --version
container CLI version 1.0.0 (build: release, commit: ee848e3)

$ container system version --format json
[
  {
    "appName": "container",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "1.0.0"
  },
  {
    "appName": "container-apiserver",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)"
  }
]

$ container system status --format json
{
  "apiServerAppName": "container-apiserver",
  "apiServerBuild": "release",
  "apiServerCommit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
  "apiServerVersion": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)",
  "appRoot": "/Users/didericis/Library/Application Support/com.apple.container/",
  "installRoot": "/usr/local/",
  "status": "running"
}

Apple Container was installed from the official signed 1.0.0 GitHub release package, container-1.0.0-installer-signed.pkg. The package was signed by Developer ID Installer: Apple Inc. - Containerization (UPBK2H6LZM) and notarized by Apple.

Commands Run

Create the networks:

container network create bb-spike-230-agent \
  --internal \
  --label bot-bottle.spike=apple-container-networking

container network create bb-spike-230-egress \
  --label bot-bottle.spike=apple-container-networking

container network inspect bb-spike-230-agent bb-spike-230-egress showed:

[
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "hostOnly",
      "name": "bb-spike-230-agent",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-agent",
    "status": {
      "ipv4Gateway": "192.168.128.1",
      "ipv4Subnet": "192.168.128.0/24"
    }
  },
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "nat",
      "name": "bb-spike-230-egress",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-egress",
    "status": {
      "ipv4Gateway": "192.168.66.1",
      "ipv4Subnet": "192.168.66.0/24"
    }
  }
]

Repeated --network flags are accepted. With the agent network first, the sidecar got two interfaces but the default route pointed at the host-only gateway, so egress failed:

container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --network bb-spike-230-egress \
  --detach --rm docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route && cat /etc/resolv.conf'

Observed:

default via 192.168.128.1 dev eth0
192.168.66.0/24 dev eth1 scope link src 192.168.66.3
192.168.128.0/24 dev eth0 scope link src 192.168.128.3
nameserver 192.168.128.1

With the NAT network first and explicit DNS, the sidecar can egress:

container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route; cat /etc/resolv.conf; wget -T 8 -O- https://example.com'

Observed:

default via 192.168.66.1 dev eth0
192.168.66.0/24 dev eth0 scope link src 192.168.66.5
192.168.128.0/24 dev eth1 scope link src 192.168.128.7
nameserver 1.1.1.1
Connecting to example.com (172.66.147.243:443)
... 100%

Start an agent only on the host-only network:

container run --name bb-spike-230-agent \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --detach docker.io/alpine:latest sleep 600

Agent network probes:

container exec bb-spike-230-agent sh -c '
  ip route
  cat /etc/resolv.conf
  wget -T 5 -O- http://192.168.128.7
  wget -T 5 -O- http://bb-spike-230-sidecar || true
  ping -c 2 1.1.1.1 || true
  wget -T 5 -O- https://example.com || true
'

Observed:

default via 192.168.128.1 dev eth0
192.168.128.0/24 dev eth0 scope link src 192.168.128.8
nameserver 192.168.128.1
Connecting to 192.168.128.7 (192.168.128.7:80)
ok
wget: bad address 'bb-spike-230-sidecar'
2 packets transmitted, 0 packets received, 100% packet loss
wget: bad address 'example.com'

Host-published loopback aliases work and are constrained to the bound alias on the host:

container run --name bb-spike-230-sidecar-alias \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --publish 127.0.0.31:18080:80 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

curl -fsS --max-time 5 http://127.0.0.31:18080
curl -fsS --max-time 5 http://127.0.0.1:18080
lsof -nP -iTCP:18080 -sTCP:LISTEN

Observed:

$ curl -fsS --max-time 5 http://127.0.0.31:18080
ok

$ curl -fsS --max-time 5 http://127.0.0.1:18080
curl: (7) Failed to connect to 127.0.0.1 port 18080 after 0 ms: Couldn't connect to server

$ lsof -nP -iTCP:18080 -sTCP:LISTEN
COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
container 17908 didericis   25u  IPv4 ...    0t0  TCP 127.0.0.31:18080 (LISTEN)

The guest cannot reach that host loopback-published listener through the host-only gateway or through its own loopback address:

container exec bb-spike-230-agent sh -c '
  wget -T 5 -O- http://192.168.128.10
  wget -T 5 -O- http://192.168.128.1:18080 || true
  wget -T 5 -O- http://127.0.0.31:18080 || true
  wget -T 5 -O- http://bb-spike-230-sidecar-alias || true
'

Observed:

Connecting to 192.168.128.10 (192.168.128.10:80)
ok
Connecting to 192.168.128.1:18080 (192.168.128.1:18080)
wget: can't connect to remote host (192.168.128.1): Connection refused
Connecting to 127.0.0.31:18080 (127.0.0.31:18080)
wget: can't connect to remote host (127.0.0.31): Connection refused
wget: bad address 'bb-spike-230-sidecar-alias'

Answers

1. Does container network create --internal prevent outbound internet access?

Yes in this run. --internal produced a hostOnly network. An internal-only agent had a default route to the host-only gateway, but could not ping 1.1.1.1 and could not resolve or fetch https://example.com.

2. Can container run attach one container to multiple networks?

Yes. Repeated --network flags produced multiple interfaces and the inspect JSON preserved both network attachments.

Important caveat: network order matters. The first network became eth0, supplied the default route, and supplied /etc/resolv.conf. For a sidecar that needs internet egress, put the NAT network first and the internal agent network second.

3. Can the sidecar bundle sit on both an internal agent network and an egress-capable network?

Yes. The sidecar had a NAT interface and a host-only interface. With the NAT network first and explicit DNS, it could fetch https://example.com while the agent on only the host-only network could not.

4. Can Apple Container provide stable network aliases or service discovery equivalent to Docker Compose aliases?

Not by default in this run. The agent could not resolve bb-spike-230-sidecar or bb-spike-230-sidecar-alias, even though those were the container names and hostnames in inspect output. The agent could reach the sidecar by the sidecar's host-only-network IP.

The backend should not assume Docker Compose-style aliases. It should read the sidecar's host-only IP from container inspect and inject that concrete endpoint into the agent environment/config, or run a small internal DNS/hosts-file setup as an explicit backend feature.

5. Can a published sidecar port bound to a per-bottle loopback alias be reached from another Apple Container guest and constrained to that alias?

Host-side alias binding works and is constrained on the host: 127.0.0.31:18080 reached the sidecar, while 127.0.0.1:18080 failed.

Guest-to-host-published-loopback did not work. From the agent, 192.168.128.1:18080 and 127.0.0.31:18080 both failed. For agent-to-sidecar traffic, use the sidecar's internal network IP rather than a host-published loopback alias.

6. What structured output is available for robust enumeration and cleanup?

Confirmed structured output:

• container list --all --format json
• container inspect <container...> as JSON
• container image inspect <image...> as JSON
• container network list --format json
• container network inspect <network...> as JSON
• container system status --format json
• container system version --format json

Useful fields observed:

• containers: id, configuration.labels, configuration.networks, configuration.publishedPorts, status.state, status.networks[].network, status.networks[].ipv4Address, status.networks[].ipv4Gateway;
• networks: id, configuration.name, configuration.labels, configuration.mode, status.ipv4Gateway, status.ipv4Subnet;
• images: id, configuration.name, configuration.descriptor, variants[].platform, variants[].size.

7. Are labels supported on containers and networks enough to replace prefix-only discovery?

Labels are present in container and network inspect/list JSON, so they are sufficient as metadata if the backend lists resources and filters client-side. I did not find or validate a server-side label filter for container list or container network list.

Recommendation

Proceed with a narrow macos-container launch prototype, but encode the Apple Container-specific constraints directly:

• create one host-only agent network and one NAT egress network per bottle;
• start the sidecar bundle with --network <egress> before --network <agent>;
• set sidecar DNS explicitly, ideally from the bottle/host policy rather than hardcoding a public resolver;
• start the agent only on the host-only network;
• discover the sidecar's host-only IP from container inspect and pass concrete URLs to the agent;
• use host loopback publishing only for host-to-sidecar access, not guest-to-sidecar access;
• enumerate and clean up by labels plus name prefixes until/unless the CLI adds label filters.

Do not implement the backend as a direct clone of Docker Compose service aliases. That assumption failed in this run.

Addendum: consolidated-gateway findings (2026-07-17, PRD 0070)

Re-tested on Apple Container 1.0.0 while porting the backend to the per-host consolidated gateway (#351). The two-network shape above still holds; these are the additional constraints that shaped the port, each verified against the live CLI on this host.

No static IP for a container

container run --network accepts only <name>[,mac=XX:XX:XX:XX:XX:XX][,mtu=VALUE]. There is no --ip. The address comes from vmnet's DHCP and is knowable only after the container is running:

$ container run --name a --network bb-net --detach alpine sleep 900
$ container inspect a | jq -r '.[0].status.networks[0].ipv4Address'
192.168.128.3/24

Consequence: the docker backend's "allocate a free IP -> pin it with --ip -> register -> launch" order cannot be reproduced. macOS inverts it to "launch -> read the assigned address -> register". The identity token therefore cannot be in the agent's run-time env (registration mints it after the container exists) and is delivered at container exec time.

Networks are fixed at run time

There is no container network connect; container network exposes only create, delete, list, inspect, prune. A network cannot be attached to a running container, so a persistent shared gateway rules out per-bottle networks — they would force a gateway restart per launch. One shared host-only network, created up front, is the only shape that keeps the gateway a singleton.

No container DNS

Containers cannot resolve each other by name; the host-only network's resolver refuses the query:

$ container exec agent nslookup gw
;; connection timed out; no servers could be reached
$ container exec agent cat /etc/resolv.conf
nameserver 192.168.128.1

Consequence: the gateway is handed the control plane's IP, not a container name as on docker. That forces the startup order orchestrator -> read its address -> gateway.

The host can reach the host-only network directly

$ container inspect c | jq -r '.[0].status.networks[0].ipv4Address'
192.168.128.2/24
$ curl -s http://192.168.128.2:8099/i
ok

So no --publish hop is needed: the host CLI and the gateway use the same control-plane URL. Docker needs --publish 127.0.0.1:... plus a separate internal URL for the same job.

CAP_NET_RAW is granted by default — and matters for attribution

Apple grants NET_RAW but not NET_ADMIN. The agent therefore cannot change its own address or route:

$ container exec agent ip addr add 192.168.128.99/24 dev eth0
ip: RTNETLINK answers: Operation not permitted
$ container exec agent ip route replace default via 192.168.128.2 dev eth0
ip: RTNETLINK answers: Operation not permitted
$ container exec agent grep CapEff /proc/self/status
CapEff:	00000000a80425fb        # bit 13 (NET_RAW) set, bit 12 (NET_ADMIN) clear

But NET_RAW permits raw sockets, i.e. source-address forgery against neighbours on the shared segment — directly against PRD 0070's invariant ("a packet's source address, as seen by the orchestrator, provably identifies the originating bottle"). --cap-drop CAP_NET_RAW closes it:

$ container run --cap-drop CAP_NET_RAW ... alpine
$ container exec nr grep CapEff /proc/self/status
CapEff:	00000000a80405fb        # bit 13 cleared
$ container exec nr ping -c1 192.168.128.2
ping: permission denied (are you root?)

The agent is run with --cap-drop CAP_NET_RAW for this reason.

container exec inherits run-time env, and --env overrides it

$ container run --name e --env FOO=from_run --detach alpine sleep 120
$ container exec e sh -c 'echo $FOO'
from_run
$ container exec --env FOO=from_exec e sh -c 'echo $FOO'
from_exec

This is what makes exec-time identity-token delivery work: the token-less proxy URL baked in at launch is superseded by the token-bearing one at exec. Bare --env NAME (inherit from the parent process) keeps the token value off argv.