didericis/bot-bottle
1
0
Fork 0
Code Issues 36 Pull Requests 3 Actions Packages Releases Wiki Activity

CI: cross-distro Firecracker end-to-end tests (Cirrus vs alternatives + cost) #387

New Issue
Closed
opened 2026-07-16 14:37:51 -04:00 by didericis-claude · 1 comment
didericis-claude commented 2026-07-16 14:37:51 -04:00
Collaborator
Copy Link
Copy Source

Goal

Verify the Firecracker backend installed the way a real user would, on bare-metal Linux distros beyond NixOS (Debian/Ubuntu, Fedora/RHEL, Arch, ...): provision prereqs via the distro package manager, run backend setup --backend=firecracker, then run the e2e suite (headless launch asserting isolation, egress 200, git-gate). Complements #348 ("firecracker + KVM only, no Docker") and #386 (once images are pulled, the install flow is pull, not build).

Hard constraint that drives everything: KVM + root

Firecracker needs /dev/kvm, and backend setup needs root / CAP_NET_ADMIN (TAP pool + nftables). That rules out ordinary container-based CI runners. The compute must be a full VM or bare-metal host, run as root, with KVM (native or nested).

Key realization: this is the self-hosted-runner model, and we already have it

The thing that made Cirrus attractive -- "attach specific systems with specific quirks as workers, run privileged/KVM tasks" -- is the self-hosted-runner model, which Gitea Actions supports natively. Register an act_runner on each bare-metal distro box, give it labels (fedora40, arch, debian12), run it in host mode (jobs execute directly on the machine, not in a container). Target with runs-on: [self-hosted, fedora40]. If act_runner runs as root -> root + CAP_NET_ADMIN + /dev/kvm, exactly the firecracker requirements. No GitHub, no mirror, no external SaaS. (Corrects the earlier "Cirrus -> GitHub mirror" framing below: we don't need to reach for Cirrus to get persistent workers.)

Why Cirrus specifically would require GitHub (for the record)

Cirrus CI is a hosted SaaS, not a runner we install. Its servers ingest webhooks, clone with an SCM token, and post status/checks -- all implemented for GitHub App + Bitbucket only. There's no way to point Cirrus's backend at self-hosted Gitea; using it would mean mirroring the repo to GitHub. Its open-source cirrus-cli can run .cirrus.yml locally, but then we provide the compute anyway, so it collapses to "Gitea Actions running Cirrus YAML." Net: Cirrus is only worth the GitHub mirror if we specifically want its managed FreeBSD/macOS/ARM fleet -- not our use case.

Options

Option Gitea-native Specific-systems / KVM Cost Notes
Gitea Actions + labeled host-mode act_runners yes (we run it) yes -- one runner per system, host mode, root ~free + hardware simplest; corrects for the persistent-worker use case
Woodpecker CI (or Drone) yes (native Gitea OAuth) yes -- privileged pipelines on our agents ~free + hardware dedicated pipeline CI if we want more than Actions, still Gitea-native
Cirrus CI no (needs GitHub mirror) yes (persistent workers / compute_engine_instance nested-virt) subscription + compute + mirror only for managed FreeBSD/macOS/ARM breadth

Cost of the compute backend (the real driver -- order-of-magnitude, verify)

Backend ~Cost KVM Notes
Own bare metal (delphi + distro VMs) ~$0 marginal (electricity ~$5-15/mo) native we maintain it; concurrency = hardware
Hetzner dedicated / auction box ~EUR 30-45/mo/box native cheap real bare metal; can host several distro VMs
GCP n2-standard-4 + nested virt ~$0.19/hr on-demand, ~$0.05/hr spot (no nested-virt surcharge) nested (Intel) pay-per-run; slower nested. ~5 distros x 15min ~ $0.25/PR
AWS *.metal ~$0.5-5+/hr native expensive; overkill

CI orchestration itself is ~free in the Gitea-native options (we self-host); the cost is compute + hardware upkeep.

Recommendation

  1. Default: Gitea Actions + one labeled host-mode act_runner per target system (delphi, plus a ~EUR 35/mo Hetzner box hosting per-distro VMs). Native, ~free, full root+KVM, gives the "specific systems" experience directly.
  2. If we outgrow Actions' ergonomics: Woodpecker CI (Gitea-native, privileged agents) before considering anything that needs a GitHub mirror.
  3. Skip Cirrus unless we later want its managed macOS/FreeBSD/ARM fleet.

First deliverable (CI-agnostic)

A scripts/firecracker-e2e.sh harness: install prereqs -> provision kernel+dropbear (the documented user path) -> backend setup -> headless launch -> assert isolation + egress 200 + git-gate. Same script runs under whichever CI/runner we pick; the CI YAML is then thin (just runs-on: the right labeled runner).

## Goal Verify the Firecracker backend **installed the way a real user would**, on bare-metal Linux distros beyond NixOS (Debian/Ubuntu, Fedora/RHEL, Arch, ...): provision prereqs via the distro package manager, run `backend setup --backend=firecracker`, then run the e2e suite (headless launch asserting isolation, egress 200, git-gate). Complements #348 ("firecracker + KVM only, no Docker") and #386 (once images are pulled, the install flow is `pull`, not `build`). ## Hard constraint that drives everything: KVM + root Firecracker needs `/dev/kvm`, and `backend setup` needs **root / CAP_NET_ADMIN** (TAP pool + nftables). That rules out ordinary container-based CI runners. The compute must be a **full VM or bare-metal host**, run as root, with KVM (native or nested). ## Key realization: this is the self-hosted-runner model, and we already have it The thing that made Cirrus attractive -- "attach specific systems with specific quirks as workers, run privileged/KVM tasks" -- **is the self-hosted-runner model, which Gitea Actions supports natively.** Register an `act_runner` on each bare-metal distro box, give it **labels** (`fedora40`, `arch`, `debian12`), run it in **host mode** (jobs execute directly on the machine, not in a container). Target with `runs-on: [self-hosted, fedora40]`. If act_runner runs as root -> **root + CAP_NET_ADMIN + /dev/kvm**, exactly the firecracker requirements. No GitHub, no mirror, no external SaaS. (Corrects the earlier "Cirrus -> GitHub mirror" framing below: we don't need to reach for Cirrus to get persistent workers.) ## Why Cirrus specifically would require GitHub (for the record) Cirrus CI is a **hosted SaaS**, not a runner we install. Its servers ingest **webhooks**, **clone** with an SCM token, and post **status/checks** -- all implemented for **GitHub App + Bitbucket only**. There's no way to point Cirrus's backend at self-hosted Gitea; using it would mean **mirroring the repo to GitHub**. Its open-source `cirrus-cli` can run `.cirrus.yml` locally, but then we provide the compute anyway, so it collapses to "Gitea Actions running Cirrus YAML." Net: Cirrus is only worth the GitHub mirror if we specifically want its managed FreeBSD/macOS/ARM fleet -- not our use case. ## Options | Option | Gitea-native | Specific-systems / KVM | Cost | Notes | |---|---|---|---|---| | **Gitea Actions + labeled host-mode `act_runner`s** | yes (we run it) | yes -- one runner per system, host mode, root | ~free + hardware | simplest; corrects for the persistent-worker use case | | **Woodpecker CI** (or Drone) | yes (native Gitea OAuth) | yes -- privileged pipelines on our agents | ~free + hardware | dedicated pipeline CI if we want more than Actions, still Gitea-native | | Cirrus CI | no (needs GitHub mirror) | yes (persistent workers / `compute_engine_instance` nested-virt) | subscription + compute + mirror | only for managed FreeBSD/macOS/ARM breadth | ## Cost of the compute backend (the real driver -- order-of-magnitude, verify) | Backend | ~Cost | KVM | Notes | |---|---|---|---| | Own bare metal (delphi + distro VMs) | ~$0 marginal (electricity ~$5-15/mo) | native | we maintain it; concurrency = hardware | | Hetzner dedicated / auction box | ~EUR 30-45/mo/box | native | cheap real bare metal; can host several distro VMs | | GCP `n2-standard-4` + nested virt | ~$0.19/hr on-demand, ~$0.05/hr spot (no nested-virt surcharge) | nested (Intel) | pay-per-run; slower nested. ~5 distros x 15min ~ $0.25/PR | | AWS `*.metal` | ~$0.5-5+/hr | native | expensive; overkill | CI orchestration itself is ~free in the Gitea-native options (we self-host); the cost is compute + hardware upkeep. ## Recommendation 1. **Default: Gitea Actions + one labeled host-mode `act_runner` per target system** (delphi, plus a ~EUR 35/mo Hetzner box hosting per-distro VMs). Native, ~free, full root+KVM, gives the "specific systems" experience directly. 2. **If we outgrow Actions' ergonomics:** Woodpecker CI (Gitea-native, privileged agents) before considering anything that needs a GitHub mirror. 3. Skip Cirrus unless we later want its managed macOS/FreeBSD/ARM fleet. ## First deliverable (CI-agnostic) A `scripts/firecracker-e2e.sh` harness: install prereqs -> provision kernel+dropbear (the documented user path) -> `backend setup` -> headless launch -> assert isolation + egress 200 + git-gate. Same script runs under whichever CI/runner we pick; the CI YAML is then thin (just `runs-on:` the right labeled runner).
didericis added the Kind/Testing
Priority
Low
4
 labels 2026-07-16 22:18:33 -04:00
didericis-codex commented 2026-07-21 01:10:09 -04:00
Collaborator
Copy Link
Copy Source

Partially superseded by the self-hosted KVM runner, which now exercises the Firecracker backend and end-to-end isolation suite. The original cross-distro installation and backend setup coverage remains incomplete, but is low priority; we can reopen this issue later if we decide to complete that work.

Partially superseded by the self-hosted KVM runner, which now exercises the Firecracker backend and end-to-end isolation suite. The original cross-distro installation and `backend setup` coverage remains incomplete, but is low priority; we can reopen this issue later if we decide to complete that work.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label Kind/Testing
Priority
Low
4
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#387
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?