fix(orchestrator): address review on host control server transport (#468)
prd-number-check / require-numbered-prds (pull_request) Failing after 12s
test / integration-docker (pull_request) Successful in 19s
tracker-policy-pr / check-pr (pull_request) Successful in 9s
test / unit (pull_request) Failing after 52s
lint / lint (push) Successful in 1m1s
test / coverage (pull_request) Has been skipped

Codex review on #496:

- **High — ambiguous delivery no longer orphans a launched bottle.** A
  timeout / dropped response from the host controller is now the ambiguous
  BrokerUnavailableError (distinct from the definite BrokerAuthError /
  BrokerClientError). OrchestratorCore.launch_bottle keeps the registry
  row on the ambiguous case instead of deregistering — deregistering would
  orphan a running container with no record (reconcile reaps rows, never
  containers). The row is left for reconcile to reap iff the bottle is not
  actually live. Definite failures still roll back, so a real failure
  leaves no orphan row.
- **Medium — the privileged endpoint bounds request bodies.** The host
  server rejects an oversized Content-Length with 413 before reading it,
  and sets a per-request socket timeout, so a caller that can merely reach
  the socket (no signed token) can't exhaust memory or a handler thread.

Tests: ambiguous-keep vs definite-rollback in the launch path; the
BrokerUnavailableError/BrokerClientError split in BrokerClient; the 413
body cap + handler error paths (driven in-thread, since daemon request
threads lose coverage); and the __main__ entrypoint broker selection.
Diff-coverage 98%; pyright clean; pylint 9.88.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 08:43:00 +00:00
parent 79dd4926bb
commit 7ab85e9ea6
8 changed files with 334 additions and 30 deletions
+16 -7
View File
@@ -25,7 +25,7 @@ import json
from collections.abc import Iterable
from datetime import datetime, timezone
from .broker import LaunchRequest, SubmitBroker, sign_request
from .broker import BrokerUnavailableError, LaunchRequest, SubmitBroker, sign_request
from .store.registry_store import DEFAULT_REAP_GRACE_SECONDS, BottleRecord, RegistryStore
from .supervisor import (
AuditEntry,
@@ -111,14 +111,23 @@ class OrchestratorCore:
image_ref=image_ref,
slot=slot,
)
launched = False
try:
self._broker.submit(sign_request(req, self._secret))
launched = True
finally:
if not launched:
self.registry.deregister(rec.bottle_id)
self._tokens.pop(rec.bottle_id, None)
except BrokerUnavailableError:
# Ambiguous delivery failure (timeout / dropped response): the broker
# may already have launched the bottle before the response was lost.
# Do NOT deregister — that would orphan a running container with no
# registry row (reconcile reaps rows, never containers). Keep the row
# so reconcile reaps it iff the bottle is not actually live; surface
# the error so the caller knows the launch is unconfirmed.
raise
except Exception:
# A definite failure — a fail-closed rejection, a backend launch
# error, or the host reporting it did not launch: nothing is running,
# so roll the registry entry back to leave no orphan.
self.registry.deregister(rec.bottle_id)
self._tokens.pop(rec.bottle_id, None)
raise
return rec
def teardown_bottle(self, bottle_id: str) -> bool: