didericis/bot-bottle
1
0
Fork 0
Code Issues 34 Pull Requests 7 Actions Packages Releases Wiki Activity

ci: backend-agnostic integration tests, per-backend skip visibility, and artifact-based coverage #414

New Issue
Open
opened 2026-07-18 14:13:41 -04:00 by didericis-claude · 0 comments
didericis-claude commented 2026-07-18 14:13:41 -04:00
Collaborator
Copy Link
Copy Source

Background

PR #349 moved the coverage job to a self-hosted KVM runner. During review, the following improvements were identified but deferred to keep the PR scope manageable.

Tasks

1. Backend-agnostic integration tests

Integration tests should not assume a specific backend. Currently:

  • Several test files are named test_orchestrator_docker_* with Docker-specific assumptions baked into fixtures and networking
  • test_sandbox_escape.py carries @skip_unless_docker() even though the Firecracker backend is fully Docker-independent at runtime (Docker is only a build-time dep for the gateway image — confirmed: the Firecracker gateway runs in a microVM via SSH, not a Docker container)
  • setUpClass reads BOT_BOTTLE_BACKEND already, but the fixture layer (networking, container/VM lifecycle) needs abstraction

Goal: each integration test reads BOT_BOTTLE_BACKEND and exercises the full test via whatever backend is specified. Backend-specific skip guards should check capability (e.g. /dev/kvm accessible) not Docker availability.

2. Explicit skip visibility per backend

When running the integration suite, skipped backends should be surfaced clearly — not silent no-ops inside test output. A preflight step per backend job should print a pass/fail capability check so a missing backend shows up as a distinct skipped/cancelled job in CI rather than hidden unittest.skip lines.

3. Coverage via artifact combination (deferred from #349)

Instead of re-running tests inside the coverage job, each test job should:

  1. Run with coverage run and upload the resulting .coverage file as an artifact
  2. The coverage job downloads all artifacts, runs coverage combine, then diff_coverage.py

This eliminates the double test-run and makes coverage a pure aggregation step. Requires actions/upload-artifact + actions/download-artifact in each job and a reworked coverage job.

Related

  • PR #349 (the KVM runner migration that prompted this)
  • docs/decisions/0004-coverage-policy.md
## Background PR #349 moved the `coverage` job to a self-hosted KVM runner. During review, the following improvements were identified but deferred to keep the PR scope manageable. ## Tasks ### 1. Backend-agnostic integration tests Integration tests should not assume a specific backend. Currently: - Several test files are named `test_orchestrator_docker_*` with Docker-specific assumptions baked into fixtures and networking - `test_sandbox_escape.py` carries `@skip_unless_docker()` even though the Firecracker backend is fully Docker-independent at runtime (Docker is only a build-time dep for the gateway image — confirmed: the Firecracker gateway runs in a microVM via SSH, not a Docker container) - `setUpClass` reads `BOT_BOTTLE_BACKEND` already, but the fixture layer (networking, container/VM lifecycle) needs abstraction Goal: each integration test reads `BOT_BOTTLE_BACKEND` and exercises the full test via whatever backend is specified. Backend-specific skip guards should check capability (e.g. `/dev/kvm` accessible) not Docker availability. ### 2. Explicit skip visibility per backend When running the integration suite, skipped backends should be surfaced clearly — not silent no-ops inside test output. A preflight step per backend job should print a pass/fail capability check so a missing backend shows up as a distinct skipped/cancelled job in CI rather than hidden `unittest.skip` lines. ### 3. Coverage via artifact combination (deferred from #349) Instead of re-running tests inside the `coverage` job, each test job should: 1. Run with `coverage run` and upload the resulting `.coverage` file as an artifact 2. The `coverage` job downloads all artifacts, runs `coverage combine`, then `diff_coverage.py` This eliminates the double test-run and makes coverage a pure aggregation step. Requires `actions/upload-artifact` + `actions/download-artifact` in each job and a reworked `coverage` job. ## Related - PR #349 (the KVM runner migration that prompted this) - `docs/decisions/0004-coverage-policy.md`
didericis-claude added the Kind/TestingKind/Enhancement labels 2026-07-18 14:13:41 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label Kind/Enhancement Kind/Testing
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#414
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?