didericis/bot-bottle
1
0
Fork 0
Code Issues 7 Pull Requests 15 Actions Packages Projects Releases Wiki Activity

docs: add Apple Container networking spike

Browse Source
This commit is contained in:
didericis
2026-06-10 18:26:54 -04:00
parent f2d5307573
commit 13e4af421d
1 changed files with 360 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/research/apple-container-networking-spike.md
+360
View File
@@ -0,0 +1,360 @@
# Apple Container networking spike
Issue: https://gitea.dideric.is/didericis/bot-bottle/issues/230
## Summary
Apple Container 1.0.0 on macOS 26 can support the core two-network
sidecar shape, but not as a drop-in Docker Compose clone.
The viable shape is:
- agent container on one `--internal` host-only network;
- sidecar bundle container on both the NAT egress network and the
host-only agent network;
- sidecar network flags ordered with the NAT network first, because
Apple Container chooses the first network as the default route;
- explicit DNS on the sidecar, because the tested NAT gateway routed
packets but did not resolve DNS;
- agent talks to sidecar by the sidecar's host-only-network IP, not by
container name or host-published loopback alias.
This is enough to unblock a cautious `macos-container` launch spike if
the backend records inspect-derived IPs and avoids depending on Docker
Compose-style aliases. It is not enough to reuse the Docker backend's
service-name assumptions unchanged.
## Local Environment
Tested on 2026-06-10:
```console
$ sw_vers
ProductName: macOS
ProductVersion: 26.5.1
BuildVersion: 25F80
$ uname -m
arm64
$ container --version
container CLI version 1.0.0 (build: release, commit: ee848e3)
$ container system version --format json
[
{
"appName": "container",
"buildType": "release",
"commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
"version": "1.0.0"
},
{
"appName": "container-apiserver",
"buildType": "release",
"commit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
"version": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)"
}
]
$ container system status --format json
{
"apiServerAppName": "container-apiserver",
"apiServerBuild": "release",
"apiServerCommit": "ee848e3ebfd7c73b04dd419683be54fb450b8779",
"apiServerVersion": "container-apiserver version 1.0.0 (build: release, commit: ee848e3)",
"appRoot": "/Users/didericis/Library/Application Support/com.apple.container/",
"installRoot": "/usr/local/",
"status": "running"
}
```
Apple Container was installed from the official signed 1.0.0 GitHub
release package, `container-1.0.0-installer-signed.pkg`. The package was
signed by `Developer ID Installer: Apple Inc. - Containerization
(UPBK2H6LZM)` and notarized by Apple.
## Commands Run
Create the networks:
```bash
container network create bb-spike-230-agent \
--internal \
--label bot-bottle.spike=apple-container-networking
container network create bb-spike-230-egress \
--label bot-bottle.spike=apple-container-networking
```
`container network inspect bb-spike-230-agent bb-spike-230-egress`
showed:
```json
[
{
"configuration": {
"labels": {"bot-bottle.spike": "apple-container-networking"},
"mode": "hostOnly",
"name": "bb-spike-230-agent",
"plugin": "container-network-vmnet"
},
"id": "bb-spike-230-agent",
"status": {
"ipv4Gateway": "192.168.128.1",
"ipv4Subnet": "192.168.128.0/24"
}
},
{
"configuration": {
"labels": {"bot-bottle.spike": "apple-container-networking"},
"mode": "nat",
"name": "bb-spike-230-egress",
"plugin": "container-network-vmnet"
},
"id": "bb-spike-230-egress",
"status": {
"ipv4Gateway": "192.168.66.1",
"ipv4Subnet": "192.168.66.0/24"
}
}
]
```
Repeated `--network` flags are accepted. With the agent network first,
the sidecar got two interfaces but the default route pointed at the
host-only gateway, so egress failed:
```bash
container run --name bb-spike-230-sidecar \
--label bot-bottle.spike=apple-container-networking \
--network bb-spike-230-agent \
--network bb-spike-230-egress \
--detach --rm docker.io/python:alpine \
sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'
container exec bb-spike-230-sidecar sh -c 'ip route && cat /etc/resolv.conf'
```
Observed:
```console
default via 192.168.128.1 dev eth0
192.168.66.0/24 dev eth1 scope link src 192.168.66.3
192.168.128.0/24 dev eth0 scope link src 192.168.128.3
nameserver 192.168.128.1
```
With the NAT network first and explicit DNS, the sidecar can egress:
```bash
container run --name bb-spike-230-sidecar \
--label bot-bottle.spike=apple-container-networking \
--network bb-spike-230-egress \
--network bb-spike-230-agent \
--dns 1.1.1.1 \
--detach docker.io/python:alpine \
sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'
container exec bb-spike-230-sidecar sh -c 'ip route; cat /etc/resolv.conf; wget -T 8 -O- https://example.com'
```
Observed:
```console
default via 192.168.66.1 dev eth0
192.168.66.0/24 dev eth0 scope link src 192.168.66.5
192.168.128.0/24 dev eth1 scope link src 192.168.128.7
nameserver 1.1.1.1
Connecting to example.com (172.66.147.243:443)
... 100%
```
Start an agent only on the host-only network:
```bash
container run --name bb-spike-230-agent \
--label bot-bottle.spike=apple-container-networking \
--network bb-spike-230-agent \
--detach docker.io/alpine:latest sleep 600
```
Agent network probes:
```bash
container exec bb-spike-230-agent sh -c '
ip route
cat /etc/resolv.conf
wget -T 5 -O- http://192.168.128.7
wget -T 5 -O- http://bb-spike-230-sidecar || true
ping -c 2 1.1.1.1 || true
wget -T 5 -O- https://example.com || true
'
```
Observed:
```console
default via 192.168.128.1 dev eth0
192.168.128.0/24 dev eth0 scope link src 192.168.128.8
nameserver 192.168.128.1
Connecting to 192.168.128.7 (192.168.128.7:80)
ok
wget: bad address 'bb-spike-230-sidecar'
2 packets transmitted, 0 packets received, 100% packet loss
wget: bad address 'example.com'
```
Host-published loopback aliases work and are constrained to the bound
alias on the host:
```bash
container run --name bb-spike-230-sidecar-alias \
--label bot-bottle.spike=apple-container-networking \
--network bb-spike-230-egress \
--network bb-spike-230-agent \
--dns 1.1.1.1 \
--publish 127.0.0.31:18080:80 \
--detach docker.io/python:alpine \
sh -c 'mkdir -p /srv && printf ok >/srv/index.html && cd /srv && python3 -m http.server 80 --bind 0.0.0.0'
curl -fsS --max-time 5 http://127.0.0.31:18080
curl -fsS --max-time 5 http://127.0.0.1:18080
lsof -nP -iTCP:18080 -sTCP:LISTEN
```
Observed:
```console
$ curl -fsS --max-time 5 http://127.0.0.31:18080
ok
$ curl -fsS --max-time 5 http://127.0.0.1:18080
curl: (7) Failed to connect to 127.0.0.1 port 18080 after 0 ms: Couldn't connect to server
$ lsof -nP -iTCP:18080 -sTCP:LISTEN
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
container 17908 didericis 25u IPv4 ... 0t0 TCP 127.0.0.31:18080 (LISTEN)
```
The guest cannot reach that host loopback-published listener through
the host-only gateway or through its own loopback address:
```bash
container exec bb-spike-230-agent sh -c '
wget -T 5 -O- http://192.168.128.10
wget -T 5 -O- http://192.168.128.1:18080 || true
wget -T 5 -O- http://127.0.0.31:18080 || true
wget -T 5 -O- http://bb-spike-230-sidecar-alias || true
'
```
Observed:
```console
Connecting to 192.168.128.10 (192.168.128.10:80)
ok
Connecting to 192.168.128.1:18080 (192.168.128.1:18080)
wget: can't connect to remote host (192.168.128.1): Connection refused
Connecting to 127.0.0.31:18080 (127.0.0.31:18080)
wget: can't connect to remote host (127.0.0.31): Connection refused
wget: bad address 'bb-spike-230-sidecar-alias'
```
## Answers
### 1. Does `container network create --internal` prevent outbound internet access?
Yes in this run. `--internal` produced a `hostOnly` network. An
internal-only agent had a default route to the host-only gateway, but
could not ping `1.1.1.1` and could not resolve or fetch
`https://example.com`.
### 2. Can `container run` attach one container to multiple networks?
Yes. Repeated `--network` flags produced multiple interfaces and the
inspect JSON preserved both network attachments.
Important caveat: network order matters. The first network became
`eth0`, supplied the default route, and supplied `/etc/resolv.conf`.
For a sidecar that needs internet egress, put the NAT network first and
the internal agent network second.
### 3. Can the sidecar bundle sit on both an internal agent network and an egress-capable network?
Yes. The sidecar had a NAT interface and a host-only interface. With the
NAT network first and explicit DNS, it could fetch `https://example.com`
while the agent on only the host-only network could not.
### 4. Can Apple Container provide stable network aliases or service discovery equivalent to Docker Compose aliases?
Not by default in this run. The agent could not resolve
`bb-spike-230-sidecar` or `bb-spike-230-sidecar-alias`, even though
those were the container names and hostnames in inspect output. The
agent could reach the sidecar by the sidecar's host-only-network IP.
The backend should not assume Docker Compose-style aliases. It should
read the sidecar's host-only IP from `container inspect` and inject
that concrete endpoint into the agent environment/config, or run a
small internal DNS/hosts-file setup as an explicit backend feature.
### 5. Can a published sidecar port bound to a per-bottle loopback alias be reached from another Apple Container guest and constrained to that alias?
Host-side alias binding works and is constrained on the host:
`127.0.0.31:18080` reached the sidecar, while `127.0.0.1:18080` failed.
Guest-to-host-published-loopback did not work. From the agent,
`192.168.128.1:18080` and `127.0.0.31:18080` both failed. For
agent-to-sidecar traffic, use the sidecar's internal network IP rather
than a host-published loopback alias.
### 6. What structured output is available for robust enumeration and cleanup?
Confirmed structured output:
- `container list --all --format json`
- `container inspect <container...>` as JSON
- `container image inspect <image...>` as JSON
- `container network list --format json`
- `container network inspect <network...>` as JSON
- `container system status --format json`
- `container system version --format json`
Useful fields observed:
- containers: `id`, `configuration.labels`,
`configuration.networks`, `configuration.publishedPorts`,
`status.state`, `status.networks[].network`,
`status.networks[].ipv4Address`, `status.networks[].ipv4Gateway`;
- networks: `id`, `configuration.name`, `configuration.labels`,
`configuration.mode`, `status.ipv4Gateway`, `status.ipv4Subnet`;
- images: `id`, `configuration.name`, `configuration.descriptor`,
`variants[].platform`, `variants[].size`.
### 7. Are labels supported on containers and networks enough to replace prefix-only discovery?
Labels are present in container and network inspect/list JSON, so they
are sufficient as metadata if the backend lists resources and filters
client-side. I did not find or validate a server-side label filter for
`container list` or `container network list`.
## Recommendation
Proceed with a narrow `macos-container` launch prototype, but encode
the Apple Container-specific constraints directly:
- create one host-only agent network and one NAT egress network per
bottle;
- start the sidecar bundle with `--network <egress>` before
`--network <agent>`;
- set sidecar DNS explicitly, ideally from the bottle/host policy
rather than hardcoding a public resolver;
- start the agent only on the host-only network;
- discover the sidecar's host-only IP from `container inspect` and pass
concrete URLs to the agent;
- use host loopback publishing only for host-to-sidecar access, not
guest-to-sidecar access;
- enumerate and clean up by labels plus name prefixes until/unless the
CLI adds label filters.
Do not implement the backend as a direct clone of Docker Compose
service aliases. That assumption failed in this run.