didericis/bot-bottle
1
0
Fork 0
Code Issues 33 Pull Requests 9 Actions Packages Releases Wiki Activity

macos-container: host loses route to orchestrator control plane once the multi-homed gateway attaches (bottle launch times out) #484

New Issue
Closed
opened 2026-07-25 22:34:31 -04:00 by didericis-claude · 1 comment
didericis-claude commented 2026-07-25 22:34:31 -04:00
Collaborator
Copy Link
Copy Source

Symptom

On the macos-container (Apple Container) backend, launching a bottle fails: every host-side orchestrator control-plane call connect-times-out (<urlopen error timed out>):

bot-bottle: orchestrator healthy [url=http://192.168.129.6:8099]      <-- host GET /health SUCCEEDS
bot-bottle: egress token reprovision skipped: GET /bottles: timed out <-- next host call: TIMEOUT
bot-bottle: registry reconciliation skipped: POST /reconcile: timed out
OrchestratorClientError: POST /bottles: <urlopen error timed out>     <-- launch dies here

Surfaced by the advisory integration-macos CI job added in #479 (TestSandboxEscape.setUpClass). Run: https://gitea.dideric.is/didericis/bot-bottle/actions/runs/3200/jobs/6189

This is not the gateway_init.py supervisor regression (which fails as GatewayError: gateway CA not available), and not related to the job's trigger config.

Root cause (host -> control-network routing on vmnet)

The orchestrator stays running and listening on 0.0.0.0:8099 the entire time (on-failure diagnostics confirm: state running, IP 192.168.129.6/24 on bot-bottle-mac-control, log orchestrator control plane listening). The host simply loses its TCP route to it.

The break happens at a precise point in infra.py:ensure_running:

  1. Orchestrator comes up alone on the control net -> host reaches http://192.168.129.6:8099/health OK (logs orchestrator healthy).
  2. gateway.connect_to_orchestrator() brings up the triple-homed gateway (gateway.py:126-152) with --network order egress, agent, control -- NAT egress first (its default route).
  3. From that instant the host can no longer TCP-connect to 192.168.129.6:8099. A connect timeout occurs at the TCP handshake, before any auth header / method / body -- so it is a routing failure, not an HTTP-layer one.

I.e. bringing the NAT-defaulted, multi-homed gateway onto the shared control network breaks the host's route to the orchestrator.

This is a real backend bug, not a CI artifact: bot-bottle start walks the same ensure_gateway -> register bottle path, so a real launch on this host hits the identical timeout.

Aggravating factor: shared per-host singleton

The host also runs a live bot-bottle instance. The macOS backend uses per-host singletons (bot-bottle-mac-infra, bot-bottle-mac-orchestrator), so the CI job force-restarts them (connect_to_orchestrator does force_remove_container) and the if: always() teardown removes them. Two consequences:

  • Running this job on a host with live bottles disrupts/kills those bottles (shared gateway+orchestrator torn down).
  • The pre-existing instance's vmnet networks/routes may be contributing to the routing break; worth isolating (dedicated runner vs shared host).

Proposed fix

Stop depending on host->vmnet routing for the control plane. Have the orchestrator publish its port to host loopback (--publish 127.0.0.1::8099) and point the host-side OrchestratorClient at 127.0.0.1:<published>. Container->container (gateway->orchestrator) keeps using the control-net IP (gateway_url()), which works fine. This sidesteps the vmnet multi-network routing limitation entirely.

Separately: guard the integration-macos job (or require a dedicated runner) so it can't clobber co-located live bottles via the shared singleton.

Repro (leaves infra up, unlike CI)

BOT_BOTTLE_BACKEND=macos-container python3 -m unittest tests.integration.test_sandbox_escape -v   # fails
ORCH=$(container ls | awk '/bot-bottle-mac-orchestrator/{print $5}' | cut -d/ -f1)
curl -m5 http://$ORCH:8099/health          # times out (gateway up)
container rm -f bot-bottle-mac-infra        # drop the gateway
curl -m5 http://$ORCH:8099/health          # 200 again -> proves gateway attach breaks host routing
## Symptom On the `macos-container` (Apple Container) backend, launching a bottle fails: every host-side orchestrator control-plane call connect-times-out (`<urlopen error timed out>`): ``` bot-bottle: orchestrator healthy [url=http://192.168.129.6:8099] <-- host GET /health SUCCEEDS bot-bottle: egress token reprovision skipped: GET /bottles: timed out <-- next host call: TIMEOUT bot-bottle: registry reconciliation skipped: POST /reconcile: timed out OrchestratorClientError: POST /bottles: <urlopen error timed out> <-- launch dies here ``` Surfaced by the advisory `integration-macos` CI job added in #479 (`TestSandboxEscape.setUpClass`). Run: https://gitea.dideric.is/didericis/bot-bottle/actions/runs/3200/jobs/6189 This is **not** the `gateway_init.py` supervisor regression (which fails as `GatewayError: gateway CA not available`), and **not** related to the job's trigger config. ## Root cause (host -> control-network routing on vmnet) The orchestrator stays `running` and listening on `0.0.0.0:8099` the entire time (on-failure diagnostics confirm: state `running`, IP `192.168.129.6/24` on `bot-bottle-mac-control`, log `orchestrator control plane listening`). The host simply loses its TCP route to it. The break happens at a precise point in `infra.py:ensure_running`: 1. Orchestrator comes up **alone** on the control net -> host reaches `http://192.168.129.6:8099/health` OK (logs `orchestrator healthy`). 2. `gateway.connect_to_orchestrator()` brings up the **triple-homed** gateway (`gateway.py:126-152`) with `--network` order egress, agent, control -- NAT egress **first** (its default route). 3. From that instant the host can no longer TCP-connect to `192.168.129.6:8099`. A connect timeout occurs at the TCP handshake, before any auth header / method / body -- so it is a routing failure, not an HTTP-layer one. I.e. bringing the NAT-defaulted, multi-homed gateway onto the shared control network breaks the host's route to the orchestrator. **This is a real backend bug, not a CI artifact:** `bot-bottle start` walks the same `ensure_gateway -> register bottle` path, so a real launch on this host hits the identical timeout. ## Aggravating factor: shared per-host singleton The host also runs a live `bot-bottle` instance. The macOS backend uses **per-host singletons** (`bot-bottle-mac-infra`, `bot-bottle-mac-orchestrator`), so the CI job **force-restarts** them (`connect_to_orchestrator` does `force_remove_container`) and the `if: always()` teardown **removes** them. Two consequences: - Running this job on a host with live bottles **disrupts/kills those bottles** (shared gateway+orchestrator torn down). - The pre-existing instance's vmnet networks/routes may be contributing to the routing break; worth isolating (dedicated runner vs shared host). ## Proposed fix Stop depending on host->vmnet routing for the control plane. Have the orchestrator **publish** its port to host loopback (`--publish 127.0.0.1::8099`) and point the **host-side** `OrchestratorClient` at `127.0.0.1:<published>`. Container->container (gateway->orchestrator) keeps using the control-net IP (`gateway_url()`), which works fine. This sidesteps the vmnet multi-network routing limitation entirely. Separately: guard the `integration-macos` job (or require a dedicated runner) so it can't clobber co-located live bottles via the shared singleton. ## Repro (leaves infra up, unlike CI) ``` BOT_BOTTLE_BACKEND=macos-container python3 -m unittest tests.integration.test_sandbox_escape -v # fails ORCH=$(container ls | awk '/bot-bottle-mac-orchestrator/{print $5}' | cut -d/ -f1) curl -m5 http://$ORCH:8099/health # times out (gateway up) container rm -f bot-bottle-mac-infra # drop the gateway curl -m5 http://$ORCH:8099/health # 200 again -> proves gateway attach breaks host routing ```
gitea-actions bot added the Status/Needs Triage label 2026-07-25 22:34:50 -04:00
didericis-claude commented 2026-07-25 22:42:03 -04:00
Author
Collaborator
Copy Link
Copy Source

Closing per decision: keep the integration-macos CI job disabled (dispatch-only, not auto-run) and run macOS integration tests manually for now instead of chasing the vmnet routing fix.

Closing per decision: keep the integration-macos CI job disabled (dispatch-only, not auto-run) and run macOS integration tests manually for now instead of chasing the vmnet routing fix.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label Status/Needs Triage
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#484
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?