Flaky firecracker CI: parallel jobs reboot the shared infra-VM singleton (blocks kvm runner parallelization) #425

New Issue

2026-07-20T10:41:37-04:00

didericis-claude commented

2026-07-20 10:41:37 -04:00

Summary

Running more than one firecracker bottle test in parallel on the delphi-kvm runner is unsafe. Parallel jobs share the per-host infra-VM singleton, and one job can reboot it out from under the others, causing flaky failures. As an immediate mitigation the kvm runner has been pinned to capacity = 1 in gitea-runner.nix (serializes bottle tests). This issue tracks the real fix so we can restore parallelism.

Symptoms (as seen in CI)

tests/integration/test_sandbox_escape.py, firecracker backend:

test_3_http_exfil_blocked — egress proxy 10.244.0.0:9099 fails with a ~3s connect timeout.
test_5_readme_push_blocked — git push to 10.244.0.0:9420 returns repository '.../throwaway.git/' not found; gitleaks never runs.

These are not an nft/netpool misconfiguration — verified on delphi that the CI netpool unit, table names (bot_bottle_fc_ci), the byte-identical bring-up script, and the bbci* :9099,9100,9420 -> 10.244.255.1 DNAT are all correct and persistent.

Root cause

The gitea kvm runner runs multiple jobs concurrently (was capacity = 2), each running the full suite.
The firecracker infra VM is a per-host singleton (bot_bottle/backend/firecracker/infra_vm.py: ensure_running). The gateway data plane is correctly multi-tenant (git-gate namespaced by unique bottle_id; egress attributed per guest /31 IP), so shared state is fine.
The singleton lifecycle is not concurrency-safe. ensure_running() calls _adoptable(key, url, want); when it returns False the job takes the singleton lock and calls stop() + boot() — destroying the VM other in-flight jobs are using.
_adoptable returns False when EITHER:
1. booted != want, where want = infra_artifact_version() is a content hash of the whole bot_bottle package + Dockerfiles + init (infra_artifact.py). Parallel jobs on different commits (two PRs, or PR vs main) disagree on the version and reboot-ping-pong the VM; OR
2. _health_ok(url) — a single 1.0s probe — flakes under concurrent boot load.
Each reboot (~10–30s) kills the egress proxy (→ 9099 timeout) and wipes the ephemeral rootfs holding /git/<bottle_id>/ repos (→ 9420 "repository not found"; registry.ext4 survives, git-gate repos do not).

Immediate mitigation (done)

gitea-runner.nix: delphi-kvm settings.runner.capacity = 2 → 1. Needs a nixos-rebuild switch on delphi to take effect.

Proposed fix (to restore parallelism)

Refcount active bottles and never reboot a healthy singleton while tenants exist; only apply a version-driven reboot when the VM is idle.
Retry the _health_ok probe a few times before condemning the VM, so a single transient flake can't trigger a destructive reboot.
Pin/converge the CI infra artifact version so concurrent jobs don't disagree on want (e.g. adopt a published tag rather than each job hashing its own checkout).
Optional/heavier alternative: per-job isolated infra VM + pool (removes the singleton; costs RAM).
Separately, add a gateway liveness preflight (is 10.244.255.1:{9099,9420} actually answering?) so failures surface as a clear fail-fast rather than a mid-test timeout.

Acceptance

kvm runner can run ≥ 2 firecracker bottle jobs concurrently without cross-job infra-VM reboots.
test_sandbox_escape passes reliably under concurrent load; restore capacity > 1.

## Summary Running more than one firecracker bottle test in parallel on the `delphi-kvm` runner is unsafe. Parallel jobs share the per-host infra-VM singleton, and one job can reboot it out from under the others, causing flaky failures. As an immediate mitigation the kvm runner has been pinned to `capacity = 1` in `gitea-runner.nix` (serializes bottle tests). This issue tracks the real fix so we can restore parallelism. ## Symptoms (as seen in CI) `tests/integration/test_sandbox_escape.py`, firecracker backend: - `test_3_http_exfil_blocked` — egress proxy `10.244.0.0:9099` fails with a ~3s connect timeout. - `test_5_readme_push_blocked` — git push to `10.244.0.0:9420` returns `repository '.../throwaway.git/' not found`; gitleaks never runs. These are **not** an nft/netpool misconfiguration — verified on delphi that the CI netpool unit, table names (`bot_bottle_fc_ci`), the byte-identical bring-up script, and the `bbci* :9099,9100,9420 -> 10.244.255.1` DNAT are all correct and persistent. ## Root cause - The gitea kvm runner runs multiple jobs concurrently (was `capacity = 2`), each running the full suite. - The firecracker infra VM is a per-host singleton (`bot_bottle/backend/firecracker/infra_vm.py: ensure_running`). The gateway data plane is correctly multi-tenant (git-gate namespaced by unique `bottle_id`; egress attributed per guest /31 IP), so shared *state* is fine. - The singleton **lifecycle** is not concurrency-safe. `ensure_running()` calls `_adoptable(key, url, want)`; when it returns False the job takes the singleton lock and calls `stop()` + `boot()` — destroying the VM other in-flight jobs are using. - `_adoptable` returns False when EITHER: 1. `booted != want`, where `want = infra_artifact_version()` is a content hash of the whole `bot_bottle` package + Dockerfiles + init (`infra_artifact.py`). Parallel jobs on **different commits** (two PRs, or PR vs main) disagree on the version and reboot-ping-pong the VM; OR 2. `_health_ok(url)` — a single 1.0s probe — flakes under concurrent boot load. - Each reboot (~10–30s) kills the egress proxy (→ 9099 timeout) and wipes the ephemeral rootfs holding `/git/<bottle_id>/` repos (→ 9420 "repository not found"; `registry.ext4` survives, git-gate repos do not). ## Immediate mitigation (done) `gitea-runner.nix`: `delphi-kvm` `settings.runner.capacity = 2 → 1`. Needs a `nixos-rebuild switch` on delphi to take effect. ## Proposed fix (to restore parallelism) - Refcount active bottles and **never reboot a healthy singleton while tenants exist**; only apply a version-driven reboot when the VM is idle. - Retry the `_health_ok` probe a few times before condemning the VM, so a single transient flake can't trigger a destructive reboot. - Pin/converge the CI infra artifact version so concurrent jobs don't disagree on `want` (e.g. adopt a published tag rather than each job hashing its own checkout). - Optional/heavier alternative: per-job isolated infra VM + pool (removes the singleton; costs RAM). - Separately, add a gateway **liveness** preflight (is `10.244.255.1:{9099,9420}` actually answering?) so failures surface as a clear fail-fast rather than a mid-test timeout. ## Acceptance - kvm runner can run ≥ 2 firecracker bottle jobs concurrently without cross-job infra-VM reboots. - `test_sandbox_escape` passes reliably under concurrent load; restore `capacity` > 1.

didericis-claude added the Kind/Bug

Priority

Low

4

labels 2026-07-20 10:41:37 -04:00

didericis referenced this issue

2026-07-20 13:36:47 -04:00

CI: no macOS (Apple Container) runner — macos_container backend is unexercised #426

Sign in to join this conversation.