bot-bottle/docs/research/apple-container-networking-spike.md

# Apple Container networking spike

Issue: https://gitea.dideric.is/didericis/bot-bottle/issues/230

## Summary

Apple Container 1.0.0 on macOS 26 can support the core two-network
sidecar shape, but not as a drop-in Docker Compose clone.

The viable shape is:

- agent container on one `--internal` host-only network;
- sidecar bundle container on both the NAT egress network and the
  host-only agent network;
- sidecar network flags ordered with the NAT network first, because
  Apple Container chooses the first network as the default route;
- explicit DNS on the sidecar, because the tested NAT gateway routed
  packets but did not resolve DNS;
- agent talks to sidecar by the sidecar's host-only-network IP, not by
  container name or host-published loopback alias.

This is enough to unblock a cautious `macos-container` launch spike if
the backend records inspect-derived IPs and avoids depending on Docker
Compose-style aliases. It is not enough to reuse the Docker backend's
service-name assumptions unchanged.

## Local Environment

Tested on 2026-06-10:

```console
$ sw_vers
ProductName:		macOS
ProductVersion:		26.5.1
BuildVersion:		25F80

$ uname -m
arm64

$ container --version
container CLI version 1.0.0 (build: release, commit: ee848e3)

$ container system version --format json
[
  {
    "appName": "container",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "1.0.0"
  },
  {
    "appName": "container-apiserver",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)"
  }
]

$ container system status --format json
{
  "apiServerAppName": "container-apiserver",
  "apiServerBuild": "release",
  "apiServerCommit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
  "apiServerVersion": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)",
  "appRoot": "/Users/didericis/Library/Application Support/com.apple.container/",
  "installRoot": "/usr/local/",
  "status": "running"
}
```

Apple Container was installed from the official signed 1.0.0 GitHub
release package, `container-1.0.0-installer-signed.pkg`. The package was
signed by `Developer ID Installer: Apple Inc. - Containerization
(UPBK2H6LZM)` and notarized by Apple.

## Commands Run

Create the networks:

```bash
container network create bb-spike-230-agent \
  --internal \
  --label bot-bottle.spike=apple-container-networking

container network create bb-spike-230-egress \
  --label bot-bottle.spike=apple-container-networking
```

`container network inspect bb-spike-230-agent bb-spike-230-egress`
showed:

```json
[
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "hostOnly",
      "name": "bb-spike-230-agent",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-agent",
    "status": {
      "ipv4Gateway": "192.168.128.1",
      "ipv4Subnet": "192.168.128.0/24"
    }
  },
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "nat",
      "name": "bb-spike-230-egress",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-egress",
    "status": {
      "ipv4Gateway": "192.168.66.1",
      "ipv4Subnet": "192.168.66.0/24"
    }
  }
]
```

Repeated `--network` flags are accepted. With the agent network first,
the sidecar got two interfaces but the default route pointed at the
host-only gateway, so egress failed:

```bash
container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --network bb-spike-230-egress \
  --detach --rm docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route && cat /etc/resolv.conf'
```

Observed:

```console
default via 192.168.128.1 dev eth0
192.168.66.0/24 dev eth1 scope link src 192.168.66.3
192.168.128.0/24 dev eth0 scope link src 192.168.128.3
nameserver 192.168.128.1
```

With the NAT network first and explicit DNS, the sidecar can egress:

```bash
container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route; cat /etc/resolv.conf; wget -T 8 -O- https://example.com'
```

Observed:

```console
default via 192.168.66.1 dev eth0
192.168.66.0/24 dev eth0 scope link src 192.168.66.5
192.168.128.0/24 dev eth1 scope link src 192.168.128.7
nameserver 1.1.1.1
Connecting to example.com (172.66.147.243:443)
... 100%
```

Start an agent only on the host-only network:

```bash
container run --name bb-spike-230-agent \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --detach docker.io/alpine:latest sleep 600
```

Agent network probes:

```bash
container exec bb-spike-230-agent sh -c '
  ip route
  cat /etc/resolv.conf
  wget -T 5 -O- http://192.168.128.7
  wget -T 5 -O- http://bb-spike-230-sidecar || true
  ping -c 2 1.1.1.1 || true
  wget -T 5 -O- https://example.com || true
'
```

Observed:

```console
default via 192.168.128.1 dev eth0
192.168.128.0/24 dev eth0 scope link src 192.168.128.8
nameserver 192.168.128.1
Connecting to 192.168.128.7 (192.168.128.7:80)
ok
wget: bad address 'bb-spike-230-sidecar'
2 packets transmitted, 0 packets received, 100% packet loss
wget: bad address 'example.com'
```

Host-published loopback aliases work and are constrained to the bound
alias on the host:

```bash
container run --name bb-spike-230-sidecar-alias \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --publish 127.0.0.31:18080:80 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

curl -fsS --max-time 5 http://127.0.0.31:18080
curl -fsS --max-time 5 http://127.0.0.1:18080
lsof -nP -iTCP:18080 -sTCP:LISTEN
```

Observed:

```console
$ curl -fsS --max-time 5 http://127.0.0.31:18080
ok

$ curl -fsS --max-time 5 http://127.0.0.1:18080
curl: (7) Failed to connect to 127.0.0.1 port 18080 after 0 ms: Couldn't connect to server

$ lsof -nP -iTCP:18080 -sTCP:LISTEN
COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
container 17908 didericis   25u  IPv4 ...    0t0  TCP 127.0.0.31:18080 (LISTEN)
```

The guest cannot reach that host loopback-published listener through
the host-only gateway or through its own loopback address:

```bash
container exec bb-spike-230-agent sh -c '
  wget -T 5 -O- http://192.168.128.10
  wget -T 5 -O- http://192.168.128.1:18080 || true
  wget -T 5 -O- http://127.0.0.31:18080 || true
  wget -T 5 -O- http://bb-spike-230-sidecar-alias || true
'
```

Observed:

```console
Connecting to 192.168.128.10 (192.168.128.10:80)
ok
Connecting to 192.168.128.1:18080 (192.168.128.1:18080)
wget: can't connect to remote host (192.168.128.1): Connection refused
Connecting to 127.0.0.31:18080 (127.0.0.31:18080)
wget: can't connect to remote host (127.0.0.31): Connection refused
wget: bad address 'bb-spike-230-sidecar-alias'
```

## Answers

### 1. Does `container network create --internal` prevent outbound internet access?

Yes in this run. `--internal` produced a `hostOnly` network. An
internal-only agent had a default route to the host-only gateway, but
could not ping `1.1.1.1` and could not resolve or fetch
`https://example.com`.

### 2. Can `container run` attach one container to multiple networks?

Yes. Repeated `--network` flags produced multiple interfaces and the
inspect JSON preserved both network attachments.

Important caveat: network order matters. The first network became
`eth0`, supplied the default route, and supplied `/etc/resolv.conf`.
For a sidecar that needs internet egress, put the NAT network first and
the internal agent network second.

### 3. Can the sidecar bundle sit on both an internal agent network and an egress-capable network?

Yes. The sidecar had a NAT interface and a host-only interface. With the
NAT network first and explicit DNS, it could fetch `https://example.com`
while the agent on only the host-only network could not.

### 4. Can Apple Container provide stable network aliases or service discovery equivalent to Docker Compose aliases?

Not by default in this run. The agent could not resolve
`bb-spike-230-sidecar` or `bb-spike-230-sidecar-alias`, even though
those were the container names and hostnames in inspect output. The
agent could reach the sidecar by the sidecar's host-only-network IP.

The backend should not assume Docker Compose-style aliases. It should
read the sidecar's host-only IP from `container inspect` and inject
that concrete endpoint into the agent environment/config, or run a
small internal DNS/hosts-file setup as an explicit backend feature.

### 5. Can a published sidecar port bound to a per-bottle loopback alias be reached from another Apple Container guest and constrained to that alias?

Host-side alias binding works and is constrained on the host:
`127.0.0.31:18080` reached the sidecar, while `127.0.0.1:18080` failed.

Guest-to-host-published-loopback did not work. From the agent,
`192.168.128.1:18080` and `127.0.0.31:18080` both failed. For
agent-to-sidecar traffic, use the sidecar's internal network IP rather
than a host-published loopback alias.

### 6. What structured output is available for robust enumeration and cleanup?

Confirmed structured output:

- `container list --all --format json`
- `container inspect <container...>` as JSON
- `container image inspect <image...>` as JSON
- `container network list --format json`
- `container network inspect <network...>` as JSON
- `container system status --format json`
- `container system version --format json`

Useful fields observed:

- containers: `id`, `configuration.labels`,
  `configuration.networks`, `configuration.publishedPorts`,
  `status.state`, `status.networks[].network`,
  `status.networks[].ipv4Address`, `status.networks[].ipv4Gateway`;
- networks: `id`, `configuration.name`, `configuration.labels`,
  `configuration.mode`, `status.ipv4Gateway`, `status.ipv4Subnet`;
- images: `id`, `configuration.name`, `configuration.descriptor`,
  `variants[].platform`, `variants[].size`.

### 7. Are labels supported on containers and networks enough to replace prefix-only discovery?

Labels are present in container and network inspect/list JSON, so they
are sufficient as metadata if the backend lists resources and filters
client-side. I did not find or validate a server-side label filter for
`container list` or `container network list`.

## Recommendation

Proceed with a narrow `macos-container` launch prototype, but encode
the Apple Container-specific constraints directly:

- create one host-only agent network and one NAT egress network per
  bottle;
- start the sidecar bundle with `--network <egress>` before
  `--network <agent>`;
- set sidecar DNS explicitly, ideally from the bottle/host policy
  rather than hardcoding a public resolver;
- start the agent only on the host-only network;
- discover the sidecar's host-only IP from `container inspect` and pass
  concrete URLs to the agent;
- use host loopback publishing only for host-to-sidecar access, not
  guest-to-sidecar access;
- enumerate and clean up by labels plus name prefixes until/unless the
  CLI adds label filters.

Do not implement the backend as a direct clone of Docker Compose
service aliases. That assumption failed in this run.