didericis/bot-bottle
1
0
Fork 0
Code Issues 34 Pull Requests 11 Actions Packages Releases Wiki Activity

Make CI assurance gates fail closed and measure the actual critical core #498

New Issue
Closed
opened 2026-07-26 03:36:15 -04:00 by didericis-codex · 0 comments
didericis-codex commented 2026-07-26 03:36:15 -04:00
Collaborator
Copy Link
Copy Source

Problem

A repository-wide quality review after #491 found that the green checks and coverage badges currently prove less than their names imply. Several related assurance mechanisms have drifted together:

  • scripts/critical-modules.txt lists 15 paths, but 8 no longer exist after the manifest/egress/git-gate/supervise package decompositions. Coverage silently measures only the seven surviving paths and omits the replacement critical modules.
  • The documented >=90% critical-core target is printed by scripts/coverage.sh, but .gitea/workflows/test.yml does not enforce it with --fail-under or an equivalent check.
  • .gitea/workflows/update-badges.yml runs the unit suite with || true, so a failed or partial run may still update the README badges.
  • tests/canaries/ contains only __init__.py. The scheduled canary workflow therefore has no canary tests to execute. No canary or pre-release run was visible in the available Actions history during the review.
  • The PR #491 Docker integration run discovered 22 tests but ran only 4; 18 were skipped under act_runner, including gateway-image, sandbox-escape, authentication, and multitenant-isolation coverage.
  • Automated test path filters omit tested runtime/build inputs such as install.sh, setup.py, MANIFEST.in, flake.nix, nix/firecracker-netpool.nix, and the pre-release workflow itself.

The result is false confidence rather than merely a low coverage number: badges and green jobs can remain healthy while the security-critical behavior they describe is absent from the measurement.

Proposed work

  1. Replace every stale critical-module path with its current package modules and fail if any configured path does not exist.
  2. Enforce the critical-core threshold in CI, independently of the diff-coverage gate.
  3. Remove || true from badge generation and refuse to publish badges unless the unit suite and coverage extraction succeed.
  4. Add real canary tests or remove/disable the nominal canary workflow until a suite exists.
  5. Make the automated Docker job exercise the security/isolation scenarios it advertises, or split those scenarios into a clearly named required job whose runner topology supports them.
  6. Expand workflow path filters to every source/build input covered by the tests.
  7. Update README.md and docs/ci.md so automated versus manually dispatched backend coverage is described accurately.

Acceptance criteria

  • CI fails when scripts/critical-modules.txt contains a missing path.
  • All current security/logic core modules are included and aggregate core coverage is >=90%.
  • A failing unit test cannot produce or commit a coverage badge.
  • The canary job executes at least one test and has a recorded successful run.
  • Required Docker security/isolation tests run rather than skip in CI, with an explicit minimum executed-test assertion.
  • Changes to every tested build/runtime input trigger the relevant workflow.
  • CI documentation matches the workflow topology.

Found during the whole-project quality evaluation following #444 / PR #491.

## Problem A repository-wide quality review after #491 found that the green checks and coverage badges currently prove less than their names imply. Several related assurance mechanisms have drifted together: - `scripts/critical-modules.txt` lists 15 paths, but 8 no longer exist after the manifest/egress/git-gate/supervise package decompositions. Coverage silently measures only the seven surviving paths and omits the replacement critical modules. - The documented >=90% critical-core target is printed by `scripts/coverage.sh`, but `.gitea/workflows/test.yml` does not enforce it with `--fail-under` or an equivalent check. - `.gitea/workflows/update-badges.yml` runs the unit suite with `|| true`, so a failed or partial run may still update the README badges. - `tests/canaries/` contains only `__init__.py`. The scheduled canary workflow therefore has no canary tests to execute. No canary or pre-release run was visible in the available Actions history during the review. - The PR #491 Docker integration run discovered 22 tests but ran only 4; 18 were skipped under `act_runner`, including gateway-image, sandbox-escape, authentication, and multitenant-isolation coverage. - Automated test path filters omit tested runtime/build inputs such as `install.sh`, `setup.py`, `MANIFEST.in`, `flake.nix`, `nix/firecracker-netpool.nix`, and the pre-release workflow itself. The result is false confidence rather than merely a low coverage number: badges and green jobs can remain healthy while the security-critical behavior they describe is absent from the measurement. ## Proposed work 1. Replace every stale critical-module path with its current package modules and fail if any configured path does not exist. 2. Enforce the critical-core threshold in CI, independently of the diff-coverage gate. 3. Remove `|| true` from badge generation and refuse to publish badges unless the unit suite and coverage extraction succeed. 4. Add real canary tests or remove/disable the nominal canary workflow until a suite exists. 5. Make the automated Docker job exercise the security/isolation scenarios it advertises, or split those scenarios into a clearly named required job whose runner topology supports them. 6. Expand workflow path filters to every source/build input covered by the tests. 7. Update `README.md` and `docs/ci.md` so automated versus manually dispatched backend coverage is described accurately. ## Acceptance criteria - CI fails when `scripts/critical-modules.txt` contains a missing path. - All current security/logic core modules are included and aggregate core coverage is >=90%. - A failing unit test cannot produce or commit a coverage badge. - The canary job executes at least one test and has a recorded successful run. - Required Docker security/isolation tests run rather than skip in CI, with an explicit minimum executed-test assertion. - Changes to every tested build/runtime input trigger the relevant workflow. - CI documentation matches the workflow topology. Found during the whole-project quality evaluation following #444 / PR #491.
didericis-codex added the
Priority
High
2
Kind/BugKind/Testing labels 2026-07-26 03:36:16 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label Kind/Bug Kind/Testing
Priority
High
2
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#498
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?