fix(firecracker): reap legacy pre-split infra VM on cutover
tracker-policy-pr / check-pr (pull_request) Successful in 12s
test / integration-docker (pull_request) Successful in 19s
lint / lint (push) Successful in 59s
test / unit (pull_request) Successful in 1m53s
test / integration-firecracker (pull_request) Successful in 3m36s
test / coverage (pull_request) Successful in 14s
test / publish-infra (pull_request) Has been skipped

The two-VM stop() only knew the new orchestrator/gateway config paths
(<infra>/{orchestrator,gateway}/config.json), so a host still running the
OLD single combined-VM (booted from <infra>/config.json on the orchestrator
link) kept it alive across the migration — and it held the orchestrator TAP,
so the first split boot died with "Open tap device failed: Resource busy"
(seen on the CI runner's bborchci0 pool).

stop() now also reaps that legacy singleton (both its drifted pidfile and any
firecracker whose --config-file is the legacy path), so the cutover is
self-healing with no manual per-host teardown. Scoped to the legacy infra
config path, so pool agent VMs are untouched.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-25 14:37:32 -04:00
parent eb9723027b
commit 06718ca761
2 changed files with 29 additions and 12 deletions
+17 -5
View File
@@ -263,13 +263,19 @@ def stop() -> None:
recorded VMMs AND any orphaned firecracker still bound to either infra
config — the PID files drift after crashes / out-of-band kills, and a
survivor would hold a link's TAP so the next boot dies with "tap …
Resource busy". Drops the version marker so a stopped pair is never
treated as adoptable."""
Resource busy". Also reaps a surviving *legacy* single combined-VM (the
pre-split layout booted from `<infra>/config.json` on the orchestrator
link): the two-VM `stop` otherwise wouldn't know about it, and it would
hold the orchestrator TAP so the first split boot fails — so the cutover
is self-healing, no manual host teardown. Drops the version marker so a
stopped pair is never treated as adoptable."""
_kill_pidfile(_orch_dir())
_kill_pidfile(_gw_dir())
_kill_pidfile(_infra_dir()) # legacy pre-split combined VM (migration)
_kill_infra_firecrackers()
_pid_file(_orch_dir()).unlink(missing_ok=True)
_pid_file(_gw_dir()).unlink(missing_ok=True)
_pid_file(_infra_dir()).unlink(missing_ok=True) # legacy
_version_file().unlink(missing_ok=True)
@@ -543,9 +549,15 @@ def _kill_infra_firecrackers(proc_root: Path = Path("/proc")) -> None:
"""SIGKILL any firecracker VMM whose `--config-file` is one of this host's
infra configs (orchestrator or gateway), independent of the PID files —
reaps orphans it lost track of so both links' TAPs are free to rebind.
Scoped to the two infra config paths, so the interactive pool's agent VMs
(other config paths) are untouched."""
cfgs = {str(_orch_dir() / "config.json"), str(_gw_dir() / "config.json")}
Also matches the *legacy* pre-split combined-VM config (`<infra>/config.json`)
so a surviving old singleton is cleared off the orchestrator link during the
cutover. Scoped to these infra config paths, so the pool's agent VMs (other
config paths) are untouched."""
cfgs = {
str(_orch_dir() / "config.json"),
str(_gw_dir() / "config.json"),
str(_infra_dir() / "config.json"), # legacy pre-split combined VM
}
for entry in proc_root.iterdir():
if not entry.name.isdigit():
continue