fix(docker): heal a poisoned IPv6 gateway network, not just avoid creating one
test / integration-docker (pull_request) Successful in 1m15s
test / image-input-builds (pull_request) Successful in 1m14s
lint / lint (push) Successful in 3m28s
tracker-policy-pr / check-pr (pull_request) Failing after 10m28s
test / unit (pull_request) Failing after 10m39s
prd-number-check / require-numbered-prds (pull_request) Failing after 10m45s
test / coverage (pull_request) Has been skipped
test / integration-docker (pull_request) Successful in 1m15s
test / image-input-builds (pull_request) Successful in 1m14s
lint / lint (push) Successful in 3m28s
tracker-policy-pr / check-pr (pull_request) Failing after 10m28s
test / unit (pull_request) Failing after 10m39s
prd-number-check / require-numbered-prds (pull_request) Failing after 10m45s
test / coverage (pull_request) Has been skipped
PR #515 stopped bot-bottle from *creating* a gateway network with a malformed IPv6 subnet (--ipv6=false), but it can't recover a network that is *already* poisoned. On a daemon that default-enables IPv6, the fixed-name `bot-bottle-gateway` network gets an `fdd0::/64` subnet whose `::1/64` gateway trips docker's own netip.ParseAddr, so `docker network inspect`/`ls` exit non-zero — poisoning every command that reads networks. Such a network can survive on a shared runner from a pre-fix or concurrent launch. `_ensure_network` never healed it: its migrate/recreate branch only ran when `network inspect` *succeeded*, but a poisoned network makes inspect *fail*, so the code fell through to `network create`, which no-ops on "already exists" — leaving the poison in place. The next subnet read then failed with ConsolidatedLaunchError (test_multitenant_isolation), and a bare `docker network ls` failed too (test_orphan_cleanup). Fix, two parts: - gateway.py: when `network inspect` fails for a reason other than "no such network", treat the network as poisoned and force-remove + recreate it IPv4-only. Absent-vs-poisoned is distinguished by the inspect stderr. - test.yml: add an integration-docker preflight that drops the leftover gateway network (and its container) before the suite, so direct `network ls`/`inspect` calls in tests are clean even on a daemon where the in-code heal can't run because inspect itself is what's broken. Adds unit coverage for the poisoned-heal and the absent-create split. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -140,12 +140,29 @@ class DockerGateway(Gateway):
|
||||
marker = inspected.stdout.strip()
|
||||
if marker in {"", self._subnet}:
|
||||
return
|
||||
if inspected.returncode == 0:
|
||||
# Migrate the stale auto-IPAM network created by older releases.
|
||||
# Removing the fixed gateway is safe here: this launch recreates it.
|
||||
# Inspectable but mislabelled: the stale auto-IPAM network created
|
||||
# by older releases. Replace it below.
|
||||
stale = True
|
||||
else:
|
||||
# inspect failed: either the network is absent (create it below) or
|
||||
# it exists but docker can't parse it. A daemon that default-enables
|
||||
# IPv6 attaches an fdd0::/64 subnet whose `::1/64` gateway trips
|
||||
# docker's own netip.ParseAddr in `network inspect`/`ls`, so the
|
||||
# command exits non-zero. A fixed release never *creates* such a
|
||||
# network, but one can survive on a shared host from an older or
|
||||
# concurrent launch — and `--ipv6=false` alone can't heal it, since
|
||||
# the create below only no-ops on "already exists". Force-replace it
|
||||
# so later reads (e.g. `_network_cidr` pinning a source IP) stop
|
||||
# failing. An absent network's inspect error is "No such network";
|
||||
# anything else means it exists but is poisoned.
|
||||
err = inspected.stderr.lower()
|
||||
stale = "no such network" not in err and "not found" not in err
|
||||
if stale:
|
||||
# Migrate the stale/poisoned network. Removing the fixed gateway is
|
||||
# safe here: this launch recreates it.
|
||||
run_docker(["docker", "rm", "--force", self.name])
|
||||
removed = run_docker(["docker", "network", "rm", self.network])
|
||||
if removed.returncode != 0:
|
||||
if removed.returncode != 0 and "no such network" not in removed.stderr.lower():
|
||||
raise GatewayError(
|
||||
f"gateway network {self.network} needs explicit subnet "
|
||||
f"{self._subnet} but could not be replaced: "
|
||||
|
||||
Reference in New Issue
Block a user