fix(docker): heal a poisoned IPv6 gateway network, not just avoid creating one
test / integration-docker (pull_request) Successful in 1m15s
test / image-input-builds (pull_request) Successful in 1m14s
lint / lint (push) Successful in 3m28s
tracker-policy-pr / check-pr (pull_request) Failing after 10m28s
test / unit (pull_request) Failing after 10m39s
prd-number-check / require-numbered-prds (pull_request) Failing after 10m45s
test / coverage (pull_request) Has been skipped

PR #515 stopped bot-bottle from *creating* a gateway network with a
malformed IPv6 subnet (--ipv6=false), but it can't recover a network that
is *already* poisoned. On a daemon that default-enables IPv6, the fixed-name
`bot-bottle-gateway` network gets an `fdd0::/64` subnet whose `::1/64`
gateway trips docker's own netip.ParseAddr, so `docker network inspect`/`ls`
exit non-zero — poisoning every command that reads networks.

Such a network can survive on a shared runner from a pre-fix or concurrent
launch. `_ensure_network` never healed it: its migrate/recreate branch only
ran when `network inspect` *succeeded*, but a poisoned network makes inspect
*fail*, so the code fell through to `network create`, which no-ops on
"already exists" — leaving the poison in place. The next subnet read then
failed with ConsolidatedLaunchError (test_multitenant_isolation), and a bare
`docker network ls` failed too (test_orphan_cleanup).

Fix, two parts:
- gateway.py: when `network inspect` fails for a reason other than "no such
  network", treat the network as poisoned and force-remove + recreate it
  IPv4-only. Absent-vs-poisoned is distinguished by the inspect stderr.
- test.yml: add an integration-docker preflight that drops the leftover
  gateway network (and its container) before the suite, so direct `network
  ls`/`inspect` calls in tests are clean even on a daemon where the in-code
  heal can't run because inspect itself is what's broken.

Adds unit coverage for the poisoned-heal and the absent-create split.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 01:23:25 +00:00
parent 85fb6b0c98
commit bd8a146a46
3 changed files with 90 additions and 4 deletions
+55
View File
@@ -248,6 +248,61 @@ class TestDockerGateway(unittest.TestCase):
calls,
)
def test_ensure_running_replaces_poisoned_ipv6_network(self) -> None:
# A daemon that default-enables IPv6 leaves the gateway network with a
# malformed fdd0::/64 gateway, so `docker network inspect` exits
# non-zero with a ParseAddr error (not "No such network"). `--ipv6=false`
# can't heal an already-poisoned network — the create just no-ops on
# "already exists" — so _ensure_network must force-remove and recreate
# it, else every later subnet read keeps failing.
calls: list[list[str]] = []
def fake(argv: list[str], **_kw: object) -> Mock:
calls.append(argv)
if argv[:2] == ["docker", "ps"]:
return _proc(stdout="")
if argv[:3] == ["docker", "network", "inspect"]:
return _proc(
returncode=1,
stderr='ParseAddr("fdd0:0:0:4::1/64"): unexpected character, '
'want colon (at "/64")',
)
return _proc()
with patch(_RUN_DOCKER, side_effect=fake):
self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
self.assertIn(["docker", "rm", "--force", self.sc.name], calls)
self.assertIn(["docker", "network", "rm", self.sc.network], calls)
creates = [c for c in calls if c[:3] == ["docker", "network", "create"]]
self.assertEqual(
[[
"docker", "network", "create",
"--ipv6=false",
"--subnet", DEFAULT_GATEWAY_SUBNET,
"--label",
f"bot-bottle.gateway-subnet={DEFAULT_GATEWAY_SUBNET}",
self.sc.network,
]],
creates,
)
def test_ensure_running_creates_network_when_inspect_reports_absent(self) -> None:
# The absent case (inspect fails with "No such network") must NOT try to
# remove anything — it just creates. Guards the poisoned-vs-absent split.
calls: list[list[str]] = []
def fake(argv: list[str], **_kw: object) -> Mock:
calls.append(argv)
if argv[:3] == ["docker", "network", "inspect"]:
return _proc(returncode=1, stderr="Error: No such network: x")
return _proc(stdout="") if argv[:2] == ["docker", "ps"] else _proc()
with patch(_RUN_DOCKER, side_effect=fake):
self.sc.connect_to_orchestrator(_ORCH_URL, _TOKEN)
self.assertNotIn(["docker", "network", "rm", self.sc.network], calls)
creates = [c for c in calls if c[:3] == ["docker", "network", "create"]]
self.assertEqual(1, len(creates))
def test_ca_cert_pem_reads_from_container(self) -> None:
with patch(_RUN_DOCKER, return_value=_proc(stdout=_CA_PEM)) as m:
self.assertEqual(_CA_PEM, self.sc.ca_cert_pem())