ci: artifact-based coverage and local Firecracker candidate flow #446

New Issue

2026-07-20T23:41:23-04:00

didericis-codex commented

2026-07-20 23:41:23 -04:00

Background

The current CI pipeline builds a compressed Firecracker infra candidate in a separate build-infra job, uploads the approximately 194 MB artifact, downloads it in integration-firecracker, then downloads it again in coverage. The coverage job also reruns the full unit and integration suites.

In a representative run, candidate construction took about 122 seconds, upload took about 70 seconds, and the two downstream downloads took about 83 seconds combined. The repeated artifact transfer and duplicate test execution dominate the critical path.

This issue supersedes the artifact-based coverage task previously tracked in #414 and expands it around a locally built, exactly-once Firecracker candidate.

Target architecture

`unit`

Run the unit suite once under coverage.
Upload a small, uniquely named coverage data artifact such as .coverage.unit.

`integration-docker`

Run the Docker integration suite once under coverage.
Upload .coverage.integration-docker.

`integration-firecracker`

Run on the self-hosted KVM runner.
Build the infra images and rootfs locally from the checked-out revision exactly once.
Run the Firecracker integration suite against that rootfs under coverage.
Upload .coverage.integration-firecracker.
Retain the exact tested rootfs in a revision/run-specific location for conditional publication.

Use the existing local-build support (BOT_BOTTLE_INFRA_BUILD=local) where appropriate rather than introducing a second build implementation. Provision host Docker on the KVM runner if this path requires it.

`coverage`

Depend on unit, integration-docker, and integration-firecracker.
Download only their small coverage data artifacts.
Run coverage combine, the aggregate coverage report, and scripts/diff_coverage.py.
Do not run any tests and do not require the KVM runner.
Ensure all producers record compatible repository-relative source paths.

`publish-infra`

Run only for pushes to main, after all required lint, test, and coverage gates pass.
Publish the exact rootfs exercised by integration-firecracker; do not rebuild it.
Pull requests must not upload or download the large rootfs through the Actions artifact service.

Requirements

Keep integration job naming consistent: integration-docker and integration-firecracker.
Each test suite executes once per workflow run.
Coverage is a pure aggregation/gating job.
PRs avoid large infra-candidate artifact transfers.
Main publishes byte-for-byte the rootfs tested by integration-firecracker.
Retained candidates are keyed by workflow run and revision, cannot collide across concurrent runs, and are cleaned up after publication or expiry.
A failed or cancelled run cannot publish an untested or stale candidate.
Preserve the current same-repository restriction for executing untrusted code on the privileged KVM runner.

Verification

Unit, Docker integration, and Firecracker integration coverage files combine successfully.
The changed-lines coverage gate produces the same result without rerunning tests.
A pull-request run transfers only coverage artifacts and does not transfer rootfs.ext4.gz.
A main run publishes the checksum of the exact rootfs used by integration-firecracker.
Concurrent and cancelled workflow runs cannot cross-publish candidates.

## Background The current CI pipeline builds a compressed Firecracker infra candidate in a separate `build-infra` job, uploads the approximately 194 MB artifact, downloads it in `integration-firecracker`, then downloads it again in `coverage`. The `coverage` job also reruns the full unit and integration suites. In a representative run, candidate construction took about 122 seconds, upload took about 70 seconds, and the two downstream downloads took about 83 seconds combined. The repeated artifact transfer and duplicate test execution dominate the critical path. This issue supersedes the artifact-based coverage task previously tracked in #414 and expands it around a locally built, exactly-once Firecracker candidate. ## Target architecture ### `unit` - Run the unit suite once under coverage. - Upload a small, uniquely named coverage data artifact such as `.coverage.unit`. ### `integration-docker` - Run the Docker integration suite once under coverage. - Upload `.coverage.integration-docker`. ### `integration-firecracker` - Run on the self-hosted KVM runner. - Build the infra images and rootfs locally from the checked-out revision exactly once. - Run the Firecracker integration suite against that rootfs under coverage. - Upload `.coverage.integration-firecracker`. - Retain the exact tested rootfs in a revision/run-specific location for conditional publication. Use the existing local-build support (`BOT_BOTTLE_INFRA_BUILD=local`) where appropriate rather than introducing a second build implementation. Provision host Docker on the KVM runner if this path requires it. ### `coverage` - Depend on `unit`, `integration-docker`, and `integration-firecracker`. - Download only their small coverage data artifacts. - Run `coverage combine`, the aggregate coverage report, and `scripts/diff_coverage.py`. - Do not run any tests and do not require the KVM runner. - Ensure all producers record compatible repository-relative source paths. ### `publish-infra` - Run only for pushes to `main`, after all required lint, test, and coverage gates pass. - Publish the exact rootfs exercised by `integration-firecracker`; do not rebuild it. - Pull requests must not upload or download the large rootfs through the Actions artifact service. ## Requirements - Keep integration job naming consistent: `integration-docker` and `integration-firecracker`. - Each test suite executes once per workflow run. - Coverage is a pure aggregation/gating job. - PRs avoid large infra-candidate artifact transfers. - Main publishes byte-for-byte the rootfs tested by `integration-firecracker`. - Retained candidates are keyed by workflow run and revision, cannot collide across concurrent runs, and are cleaned up after publication or expiry. - A failed or cancelled run cannot publish an untested or stale candidate. - Preserve the current same-repository restriction for executing untrusted code on the privileged KVM runner. ## Verification - Unit, Docker integration, and Firecracker integration coverage files combine successfully. - The changed-lines coverage gate produces the same result without rerunning tests. - A pull-request run transfers only coverage artifacts and does not transfer `rootfs.ext4.gz`. - A `main` run publishes the checksum of the exact rootfs used by `integration-firecracker`. - Concurrent and cancelled workflow runs cannot cross-publish candidates. ## Related - #414 — backend-agnostic integration tests and explicit backend skip visibility - #349 — KVM runner migration - `docs/decisions/0004-coverage-policy.md`

gitea-actions bot added the Status/Needs Triage label 2026-07-20 23:41:36 -04:00

didericis-codex referenced this issue

2026-07-20 23:41:37 -04:00

ci: backend-agnostic integration tests and explicit backend skip visibility #414

didericis-codex added the Kind/Enhancement Kind/Testing labels 2026-07-20 23:41:38 -04:00

didericis added

Priority

Medium

3

and removed Status/Needs Triage labels 2026-07-20 23:42:57 -04:00

didericis referenced this issue from a commit

2026-07-21 00:04:39 -04:00