fix(docker): heal a poisoned IPv6 gateway network, not just avoid creating one
test / integration-docker (pull_request) Successful in 1m15s
test / image-input-builds (pull_request) Successful in 1m14s
lint / lint (push) Successful in 3m28s
tracker-policy-pr / check-pr (pull_request) Failing after 10m28s
test / unit (pull_request) Failing after 10m39s
prd-number-check / require-numbered-prds (pull_request) Failing after 10m45s
test / coverage (pull_request) Has been skipped

PR #515 stopped bot-bottle from *creating* a gateway network with a
malformed IPv6 subnet (--ipv6=false), but it can't recover a network that
is *already* poisoned. On a daemon that default-enables IPv6, the fixed-name
`bot-bottle-gateway` network gets an `fdd0::/64` subnet whose `::1/64`
gateway trips docker's own netip.ParseAddr, so `docker network inspect`/`ls`
exit non-zero — poisoning every command that reads networks.

Such a network can survive on a shared runner from a pre-fix or concurrent
launch. `_ensure_network` never healed it: its migrate/recreate branch only
ran when `network inspect` *succeeded*, but a poisoned network makes inspect
*fail*, so the code fell through to `network create`, which no-ops on
"already exists" — leaving the poison in place. The next subnet read then
failed with ConsolidatedLaunchError (test_multitenant_isolation), and a bare
`docker network ls` failed too (test_orphan_cleanup).

Fix, two parts:
- gateway.py: when `network inspect` fails for a reason other than "no such
  network", treat the network as poisoned and force-remove + recreate it
  IPv4-only. Absent-vs-poisoned is distinguished by the inspect stderr.
- test.yml: add an integration-docker preflight that drops the leftover
  gateway network (and its container) before the suite, so direct `network
  ls`/`inspect` calls in tests are clean even on a daemon where the in-code
  heal can't run because inspect itself is what's broken.

Adds unit coverage for the poisoned-heal and the absent-create split.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-27 01:23:25 +00:00
parent 85fb6b0c98
commit bd8a146a46
3 changed files with 90 additions and 4 deletions
+21 -4
View File
@@ -140,12 +140,29 @@ class DockerGateway(Gateway):
marker = inspected.stdout.strip()
if marker in {"", self._subnet}:
return
if inspected.returncode == 0:
# Migrate the stale auto-IPAM network created by older releases.
# Removing the fixed gateway is safe here: this launch recreates it.
# Inspectable but mislabelled: the stale auto-IPAM network created
# by older releases. Replace it below.
stale = True
else:
# inspect failed: either the network is absent (create it below) or
# it exists but docker can't parse it. A daemon that default-enables
# IPv6 attaches an fdd0::/64 subnet whose `::1/64` gateway trips
# docker's own netip.ParseAddr in `network inspect`/`ls`, so the
# command exits non-zero. A fixed release never *creates* such a
# network, but one can survive on a shared host from an older or
# concurrent launch — and `--ipv6=false` alone can't heal it, since
# the create below only no-ops on "already exists". Force-replace it
# so later reads (e.g. `_network_cidr` pinning a source IP) stop
# failing. An absent network's inspect error is "No such network";
# anything else means it exists but is poisoned.
err = inspected.stderr.lower()
stale = "no such network" not in err and "not found" not in err
if stale:
# Migrate the stale/poisoned network. Removing the fixed gateway is
# safe here: this launch recreates it.
run_docker(["docker", "rm", "--force", self.name])
removed = run_docker(["docker", "network", "rm", self.network])
if removed.returncode != 0:
if removed.returncode != 0 and "no such network" not in removed.stderr.lower():
raise GatewayError(
f"gateway network {self.network} needs explicit subnet "
f"{self._subnet} but could not be replaced: "