bot-bottle/docs/research/apple-container-networking-spike.md

# Apple Container networking spike

Issue: https://gitea.dideric.is/didericis/bot-bottle/issues/230

## Summary

Apple Container 1.0.0 on macOS 26 can support the core two-network
sidecar shape, but not as a drop-in Docker Compose clone.

The viable shape is:

- agent container on one `--internal` host-only network;
- sidecar bundle container on both the NAT egress network and the
  host-only agent network;
- sidecar network flags ordered with the NAT network first, because
  Apple Container chooses the first network as the default route;
- explicit DNS on the sidecar, because the tested NAT gateway routed
  packets but did not resolve DNS;
- agent talks to sidecar by the sidecar's host-only-network IP, not by
  container name or host-published loopback alias.

This is enough to unblock a cautious `macos-container` launch spike if
the backend records inspect-derived IPs and avoids depending on Docker
Compose-style aliases. It is not enough to reuse the Docker backend's
service-name assumptions unchanged.

## Local Environment

Tested on 2026-06-10:

```console
$ sw_vers
ProductName:		macOS
ProductVersion:		26.5.1
BuildVersion:		25F80

$ uname -m
arm64

$ container --version
container CLI version 1.0.0 (build: release, commit: ee848e3)

$ container system version --format json
[
  {
    "appName": "container",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "1.0.0"
  },
  {
    "appName": "container-apiserver",
    "buildType": "release",
    "commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
    "version": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)"
  }
]

$ container system status --format json
{
  "apiServerAppName": "container-apiserver",
  "apiServerBuild": "release",
  "apiServerCommit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
  "apiServerVersion": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)",
  "appRoot": "/Users/didericis/Library/Application Support/com.apple.container/",
  "installRoot": "/usr/local/",
  "status": "running"
}
```

Apple Container was installed from the official signed 1.0.0 GitHub
release package, `container-1.0.0-installer-signed.pkg`. The package was
signed by `Developer ID Installer: Apple Inc. - Containerization
(UPBK2H6LZM)` and notarized by Apple.

## Commands Run

Create the networks:

```bash
container network create bb-spike-230-agent \
  --internal \
  --label bot-bottle.spike=apple-container-networking

container network create bb-spike-230-egress \
  --label bot-bottle.spike=apple-container-networking
```

`container network inspect bb-spike-230-agent bb-spike-230-egress`
showed:

```json
[
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "hostOnly",
      "name": "bb-spike-230-agent",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-agent",
    "status": {
      "ipv4Gateway": "192.168.128.1",
      "ipv4Subnet": "192.168.128.0/24"
    }
  },
  {
    "configuration": {
      "labels": {"bot-bottle.spike": "apple-container-networking"},
      "mode": "nat",
      "name": "bb-spike-230-egress",
      "plugin": "container-network-vmnet"
    },
    "id": "bb-spike-230-egress",
    "status": {
      "ipv4Gateway": "192.168.66.1",
      "ipv4Subnet": "192.168.66.0/24"
    }
  }
]
```

Repeated `--network` flags are accepted. With the agent network first,
the sidecar got two interfaces but the default route pointed at the
host-only gateway, so egress failed:

```bash
container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --network bb-spike-230-egress \
  --detach --rm docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route && cat /etc/resolv.conf'
```

Observed:

```console
default via 192.168.128.1 dev eth0
192.168.66.0/24 dev eth1 scope link src 192.168.66.3
192.168.128.0/24 dev eth0 scope link src 192.168.128.3
nameserver 192.168.128.1
```

With the NAT network first and explicit DNS, the sidecar can egress:

```bash
container run --name bb-spike-230-sidecar \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

container exec bb-spike-230-sidecar sh -c 'ip route; cat /etc/resolv.conf; wget -T 8 -O- https://example.com'
```

Observed:

```console
default via 192.168.66.1 dev eth0
192.168.66.0/24 dev eth0 scope link src 192.168.66.5
192.168.128.0/24 dev eth1 scope link src 192.168.128.7
nameserver 1.1.1.1
Connecting to example.com (172.66.147.243:443)
... 100%
```

Start an agent only on the host-only network:

```bash
container run --name bb-spike-230-agent \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-agent \
  --detach docker.io/alpine:latest sleep 600
```

Agent network probes:

```bash
container exec bb-spike-230-agent sh -c '
  ip route
  cat /etc/resolv.conf
  wget -T 5 -O- http://192.168.128.7
  wget -T 5 -O- http://bb-spike-230-sidecar || true
  ping -c 2 1.1.1.1 || true
  wget -T 5 -O- https://example.com || true
'
```

Observed:

```console
default via 192.168.128.1 dev eth0
192.168.128.0/24 dev eth0 scope link src 192.168.128.8
nameserver 192.168.128.1
Connecting to 192.168.128.7 (192.168.128.7:80)
ok
wget: bad address 'bb-spike-230-sidecar'
2 packets transmitted, 0 packets received, 100% packet loss
wget: bad address 'example.com'
```

Host-published loopback aliases work and are constrained to the bound
alias on the host:

```bash
container run --name bb-spike-230-sidecar-alias \
  --label bot-bottle.spike=apple-container-networking \
  --network bb-spike-230-egress \
  --network bb-spike-230-agent \
  --dns 1.1.1.1 \
  --publish 127.0.0.31:18080:80 \
  --detach docker.io/python:alpine \
  sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'

curl -fsS --max-time 5 http://127.0.0.31:18080
curl -fsS --max-time 5 http://127.0.0.1:18080
lsof -nP -iTCP:18080 -sTCP:LISTEN
```

Observed:

```console
$ curl -fsS --max-time 5 http://127.0.0.31:18080
ok

$ curl -fsS --max-time 5 http://127.0.0.1:18080
curl: (7) Failed to connect to 127.0.0.1 port 18080 after 0 ms: Couldn't connect to server

$ lsof -nP -iTCP:18080 -sTCP:LISTEN
COMMAND     PID      USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
container 17908 didericis   25u  IPv4 ...    0t0  TCP 127.0.0.31:18080 (LISTEN)
```

The guest cannot reach that host loopback-published listener through
the host-only gateway or through its own loopback address:

```bash
container exec bb-spike-230-agent sh -c '
  wget -T 5 -O- http://192.168.128.10
  wget -T 5 -O- http://192.168.128.1:18080 || true
  wget -T 5 -O- http://127.0.0.31:18080 || true
  wget -T 5 -O- http://bb-spike-230-sidecar-alias || true
'
```

Observed:

```console
Connecting to 192.168.128.10 (192.168.128.10:80)
ok
Connecting to 192.168.128.1:18080 (192.168.128.1:18080)
wget: can't connect to remote host (192.168.128.1): Connection refused
Connecting to 127.0.0.31:18080 (127.0.0.31:18080)
wget: can't connect to remote host (127.0.0.31): Connection refused
wget: bad address 'bb-spike-230-sidecar-alias'
```

## Answers

### 1. Does `container network create --internal` prevent outbound internet access?

Yes in this run. `--internal` produced a `hostOnly` network. An
internal-only agent had a default route to the host-only gateway, but
could not ping `1.1.1.1` and could not resolve or fetch
`https://example.com`.

### 2. Can `container run` attach one container to multiple networks?

Yes. Repeated `--network` flags produced multiple interfaces and the
inspect JSON preserved both network attachments.

Important caveat: network order matters. The first network became
`eth0`, supplied the default route, and supplied `/etc/resolv.conf`.
For a sidecar that needs internet egress, put the NAT network first and
the internal agent network second.

### 3. Can the sidecar bundle sit on both an internal agent network and an egress-capable network?

Yes. The sidecar had a NAT interface and a host-only interface. With the
NAT network first and explicit DNS, it could fetch `https://example.com`
while the agent on only the host-only network could not.

### 4. Can Apple Container provide stable network aliases or service discovery equivalent to Docker Compose aliases?

Not by default in this run. The agent could not resolve
`bb-spike-230-sidecar` or `bb-spike-230-sidecar-alias`, even though
those were the container names and hostnames in inspect output. The
agent could reach the sidecar by the sidecar's host-only-network IP.

The backend should not assume Docker Compose-style aliases. It should
read the sidecar's host-only IP from `container inspect` and inject
that concrete endpoint into the agent environment/config, or run a
small internal DNS/hosts-file setup as an explicit backend feature.

### 5. Can a published sidecar port bound to a per-bottle loopback alias be reached from another Apple Container guest and constrained to that alias?

Host-side alias binding works and is constrained on the host:
`127.0.0.31:18080` reached the sidecar, while `127.0.0.1:18080` failed.

Guest-to-host-published-loopback did not work. From the agent,
`192.168.128.1:18080` and `127.0.0.31:18080` both failed. For
agent-to-sidecar traffic, use the sidecar's internal network IP rather
than a host-published loopback alias.

### 6. What structured output is available for robust enumeration and cleanup?

Confirmed structured output:

- `container list --all --format json`
- `container inspect <container...>` as JSON
- `container image inspect <image...>` as JSON
- `container network list --format json`
- `container network inspect <network...>` as JSON
- `container system status --format json`
- `container system version --format json`

Useful fields observed:

- containers: `id`, `configuration.labels`,
  `configuration.networks`, `configuration.publishedPorts`,
  `status.state`, `status.networks[].network`,
  `status.networks[].ipv4Address`, `status.networks[].ipv4Gateway`;
- networks: `id`, `configuration.name`, `configuration.labels`,
  `configuration.mode`, `status.ipv4Gateway`, `status.ipv4Subnet`;
- images: `id`, `configuration.name`, `configuration.descriptor`,
  `variants[].platform`, `variants[].size`.

### 7. Are labels supported on containers and networks enough to replace prefix-only discovery?

Labels are present in container and network inspect/list JSON, so they
are sufficient as metadata if the backend lists resources and filters
client-side. I did not find or validate a server-side label filter for
`container list` or `container network list`.

## Recommendation

Proceed with a narrow `macos-container` launch prototype, but encode
the Apple Container-specific constraints directly:

- create one host-only agent network and one NAT egress network per
  bottle;
- start the sidecar bundle with `--network <egress>` before
  `--network <agent>`;
- set sidecar DNS explicitly, ideally from the bottle/host policy
  rather than hardcoding a public resolver;
- start the agent only on the host-only network;
- discover the sidecar's host-only IP from `container inspect` and pass
  concrete URLs to the agent;
- use host loopback publishing only for host-to-sidecar access, not
  guest-to-sidecar access;
- enumerate and clean up by labels plus name prefixes until/unless the
  CLI adds label filters.

Do not implement the backend as a direct clone of Docker Compose
service aliases. That assumption failed in this run.

## Addendum: consolidated-gateway findings (2026-07-17, PRD 0070)

Re-tested on Apple Container 1.0.0 while porting the backend to the
per-host consolidated gateway (#351). The two-network shape above still
holds; these are the additional constraints that shaped the port, each
verified against the live CLI on this host.

### No static IP for a container

`container run --network` accepts only
`<name>[,mac=XX:XX:XX:XX:XX:XX][,mtu=VALUE]`. There is no `--ip`. The
address comes from vmnet's DHCP and is knowable only after the container
is running:

```console
$ container run --name a --network bb-net --detach alpine sleep 900
$ container inspect a | jq -r '.[0].status.networks[0].ipv4Address'
192.168.128.3/24
```

Consequence: the docker backend's "allocate a free IP -> pin it with
`--ip` -> register -> launch" order cannot be reproduced. macOS inverts
it to "launch -> read the assigned address -> register". The identity
token therefore cannot be in the agent's run-time env (registration mints
it after the container exists) and is delivered at `container exec` time.

### Networks are fixed at run time

There is no `container network connect`; `container network` exposes only
`create`, `delete`, `list`, `inspect`, `prune`. A network cannot be
attached to a running container, so a *persistent* shared gateway rules
out per-bottle networks — they would force a gateway restart per launch.
One shared host-only network, created up front, is the only shape that
keeps the gateway a singleton.

### No container DNS

Containers cannot resolve each other by name; the host-only network's
resolver refuses the query:

```console
$ container exec agent nslookup gw
;; connection timed out; no servers could be reached
$ container exec agent cat /etc/resolv.conf
nameserver 192.168.128.1
```

Consequence: the gateway is handed the control plane's **IP**, not a
container name as on docker. That forces the startup order
orchestrator -> read its address -> gateway.

### The host can reach the host-only network directly

```console
$ container inspect c | jq -r '.[0].status.networks[0].ipv4Address'
192.168.128.2/24
$ curl -s http://192.168.128.2:8099/i
ok
```

So no `--publish` hop is needed: the host CLI and the gateway use the
same control-plane URL. Docker needs `--publish 127.0.0.1:...` plus a
separate internal URL for the same job.

### CAP_NET_RAW is granted by default — and matters for attribution

Apple grants NET_RAW but not NET_ADMIN. The agent therefore cannot change
its own address or route:

```console
$ container exec agent ip addr add 192.168.128.99/24 dev eth0
ip: RTNETLINK answers: Operation not permitted
$ container exec agent ip route replace default via 192.168.128.2 dev eth0
ip: RTNETLINK answers: Operation not permitted
$ container exec agent grep CapEff /proc/self/status
CapEff:	00000000a80425fb        # bit 13 (NET_RAW) set, bit 12 (NET_ADMIN) clear
```

But NET_RAW permits raw sockets, i.e. source-address forgery against
neighbours on the shared segment — directly against PRD 0070's invariant
("a packet's source address, as seen by the orchestrator, provably
identifies the originating bottle"). `--cap-drop CAP_NET_RAW` closes it:

```console
$ container run --cap-drop CAP_NET_RAW ... alpine
$ container exec nr grep CapEff /proc/self/status
CapEff:	00000000a80405fb        # bit 13 cleared
$ container exec nr ping -c1 192.168.128.2
ping: permission denied (are you root?)
```

The agent is run with `--cap-drop CAP_NET_RAW` for this reason.

### `container exec` inherits run-time env, and `--env` overrides it

```console
$ container run --name e --env FOO=from_run --detach alpine sleep 120
$ container exec e sh -c 'echo $FOO'
from_run
$ container exec --env FOO=from_exec e sh -c 'echo $FOO'
from_exec
```

This is what makes exec-time identity-token delivery work: the token-less
proxy URL baked in at launch is superseded by the token-bearing one at
exec. Bare `--env NAME` (inherit from the parent process) keeps the token
value off argv.