Resolves via ENV VAR -> orchestrator DB config -> default (30 s, up from 5 s):
BOT_BOTTLE_ORCHESTRATOR_TEARDOWN_TIMEOUT_SECONDS
teardown_timeout_seconds key in new orchestrator_config table (bot-bottle.db)
New OrchestratorConfigStore (same DbStore/TableMigrations pattern as the
registry) stores the DB-level setting. resolve_teardown_timeout() implements
the priority chain and is called at stack.callback registration time in all
three backends (macos_container, docker, firecracker).
A failed container list or a partial per-container inspect were
previously indistinguishable from a legitimately empty/partial live
set, so reconciliation could silently unregister healthy bottles.
- enumerate_active() now raises EnumerationError instead of returning []
when `container list` fails
- Add inspect_container_network_ip() to util, which returns None on
inspect failure (vs "" for "no DHCP address yet"), so live_source_ips
can tell the two apart
- live_source_ips() raises EnumerationError on either failure mode;
register_agent() catches it and skips reconciliation, same as
OrchestratorClientError
- Update tests: rename test_empty_when_the_cli_fails to
test_raises_when_the_cli_fails, update patching to the new function,
add coverage for list failure, per-container inspect failure, and the
launch-not-blocked path
A registry row only ever left the registry two ways: an explicit
teardown_bottle (the launcher's cleanup callback) or the same-IP supersede
sweep in register(). Neither runs when the launching CLI dies hard, so the
row outlives its container.
That orphan is not inert. Source IPs are recycled by the backend's DHCP and
by_source_ip fail-closes on ambiguity, so a leftover row at a reused address
resolves no policy at all for the next bottle that lands there — and a
bottle with no policy denies every host, which surfaces to the agent as
"host X is not in the allowlist" for hosts that were never the problem.
Add reap_absent/reconcile and call it from the macOS launch path before
registering, so each launch self-heals the registry. Restores the invariant
the data plane needs: at most one active row per live address, and none for
a dead one. The second half matters as much as the first — when several rows
claim a *live* address the newest wins and the rest are swept, otherwise a
recycled address stays ambiguous, which is exactly the bricked state.
The host supplies the live set because the orchestrator runs inside the
infra container and cannot see the backend. A grace window exempts rows
younger than it, so reconciliation cannot race a bottle still coming up, and
a reconcile failure is logged rather than blocking an otherwise-fine launch.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Adopts the firecracker infra-VM pattern for macOS: the orchestrator control
plane and the gateway data plane now run in a SINGLE Apple container instead of
two. Apple Containers are lightweight VMs with separate kernels, so the prior
two-container design had both guests writing one bot-bottle.db over virtiofs,
where fcntl locks are not coherent across kernels — concurrent writes (the
orchestrator's registry vs the gateway supervise daemon's queue) could corrupt
it. One container = one kernel = coherent locking.
The DB moves onto a container-only Apple volume (bot-bottle-mac-db), never
bind-mounted from the host, so no host process opens the live file either. The
host CLI already reaches registry + supervise state over the control-plane HTTP
surface (cli/supervise.py uses OrchestratorClient), exactly as firecracker's
VM-only DB requires.
Two simplifications fall out of the single container:
- No DNS dance: the control plane and gateway daemons reach each other over
127.0.0.1, so the orchestrator-before-gateway ordering (a workaround for
Apple having no container DNS) is gone, along with the moved-IP recreate
logic it needed.
- Net -243 lines.
Mechanics: the infra container runs from the gateway image with the
control-plane source bind-mounted read-only (like the docker orchestrator, so a
code change needs no rebuild) and a small sh -c init that starts both processes
(mirrors firecracker's _infra_init). Also implements the macOS backend's
ensure_orchestrator() and adds it to discover_orchestrator_url, so operator
tools (supervise) can bring up / find the control plane on demand — previously
the macOS backend died with "no orchestrator control plane".
Verified end-to-end on real Apple Container 1.0.0: the single infra container
comes up healthy (one address for control plane + gateway), both processes run,
the DB is written on the container-only volume, host-side supervise works over
HTTP, and a registered agent gets 200 for an allowed host / 403 for a denied
one. 1824 unit tests pass with `container` absent (CI parity), pyright clean,
pylint 9.89.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Re-enables the macos-container backend on the shared per-host orchestrator +
gateway, replacing the per-bottle companion container removed in #385. This is
the last backend in PRD 0070's roadmap.
Apple Container 1.0.0 forced three departures from the docker shape, each
verified against the live CLI (findings recorded in the networking spike):
- No `--ip`. The address is DHCP-assigned and knowable only once the container
runs, so the order inverts: gateway up -> run agent -> read its address ->
register. The identity token is minted by registration and therefore cannot
be in the agent's run-time env; it rides the proxy URL applied at
`container exec` time (bare `--env` names keep it off argv).
- No container DNS. The gateway can only be handed the control plane's IP, so
the orchestrator starts first and the gateway is pointed at its address.
- No `network connect`. Networks are fixed at run time, so the shared host-only
network is created up front; per-bottle networks would restart the gateway
on every launch and defeat the consolidation.
The agent runs with `--cap-drop CAP_NET_RAW`: Apple grants NET_RAW by default,
which would let an agent forge a neighbour's source address on the shared
segment. NET_ADMIN is already absent, so this closes the source-address half of
PRD 0070's attribution invariant.
Verified end-to-end on real Apple Container 1.0.0: both images build, the
control plane comes up healthy, the gateway reaches it by IP, and a registered
agent gets 200 for a host in its routes and 403 for one outside them. Bring-up
is idempotent — a second launch does not churn the singletons.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-claude
didericis