bot-bottle/docs/prds/0083-packaged-infra-artifacts.md

PRD 0083: Packaged infrastructure artifacts

• Status: Draft
• Author: Codex
• Created: 2026-07-27
• Issue: #536

Summary

Publish the orchestrator and gateway infrastructure before packaging bot-bottle, record their immutable identities in the application package, and make every production backend acquire those exact artifacts instead of building infrastructure during bottle startup. Docker and Apple Container pull digest-pinned OCI images; Firecracker pulls checksum-verified rootfs artifacts selected by the same packaged release manifest.

Problem

Starting a bottle currently invokes an infrastructure build on Docker and Apple Container even when a healthy orchestrator is already running. Both infra composers call ensure_built() before their health/currentness checks, and both implementations unconditionally invoke their OCI builder. Layer caching can make this less expensive, but startup still depends on a working builder, build context, base-image registry, and package installation network.

The Apple Container orchestrator also bind-mounts the installed source over the package baked into its image. A pinned image alone would therefore not pin the code that actually runs.

Firecracker already downloads fixed rootfs artifacts, but it derives their versions from the package contents at launch. There is no package-level record that says which already-published infrastructure release belongs to an installed bot-bottle release.

The result is three different delivery contracts:

• Docker and Apple Container build mutable :latest images on the launch host.
• Firecracker computes an artifact version locally and downloads it.
• An installed wheel contains Dockerfiles specifically so production startup can rebuild its own infrastructure.

That makes startup slower and less reliable, prevents a packaged application from identifying the infrastructure it was tested with, and weakens the supply-chain boundary between release production and runtime.

Goals / Success criteria

• A packaged bot-bottle release contains one validated release manifest with immutable orchestrator and gateway identities.
• Docker and Apple Container acquire and run the manifest's digest-pinned OCI images without invoking docker build or container build.
• Firecracker acquires the manifest's versioned, checksum-verified orchestrator and gateway rootfs artifacts rather than deriving versions at launch.
• All three backends expose the same backend-neutral ensure_available lifecycle contract.
• A healthy running orchestrator is retained only when its recorded release identity matches the installed package.
• Apple Container runs the package baked into the orchestrator image; no host source overlays production code.
• Missing, mutable, malformed, or internally inconsistent release metadata fails closed before infrastructure starts.
• Source-checkout development retains an explicit local-build mode.
• Publishing produces the infrastructure first and the installable Python package last, so the package cannot name artifacts that were not published.
• Any branch, tag, or commit can be resolved once to a full source SHA and published as one immutable, commit-addressed artifact bundle.
• The installer defaults to the production channel, can select staging or an exact commit, and warns for commit snapshots that have not been qualified as a release.
• Staging and production releases promote the same tested artifact bytes; promotion never rebuilds them.
• A release is published only after the complete pre-release suite and a clean install from the published bundle succeed.

Non-goals

• Changing agent/provider image builds. User Dockerfiles continue to build through the backend-specific agent-image paths.
• Installing Docker, Apple Container, Firecracker, or other host prerequisites.
• Replacing Gitea's OCI or generic-package registries.
• Adding image signing or a transparency log. Immutable digests and existing Firecracker SHA-256 verification are the integrity boundary for this version.
• Forcing a network download when an immutable, verified artifact is already cached locally. "Pull" means acquire the packaged identity, with safe cache reuse.
• Supporting live source overlays in packaged production installs.
• Defining the project's version-numbering policy beyond stable vX.Y.Z and staging vX.Y.Z-rc.N tag shapes.

Design

Release manifest

bot_bottle/release-manifest.json is generated during package production and shipped as package data:

{
  "schema": 1,
  "source_commit": "<40 lowercase hex>",
  "oci": {
    "orchestrator": "gitea.dideric.is/didericis/bot-bottle-orchestrator@sha256:<64 hex>",
    "gateway": "gitea.dideric.is/didericis/bot-bottle-gateway@sha256:<64 hex>"
  },
  "firecracker": {
    "orchestrator": {
      "version": "<artifact version>",
      "sha256": "<rootfs.ext4.gz sha256>"
    },
    "gateway": {
      "version": "<artifact version>",
      "sha256": "<rootfs.ext4.gz sha256>"
    }
  }
}

The runtime loader accepts only schema 1, digest-qualified OCI references, non-empty Firecracker versions, 64-character lowercase SHA-256 values, and a full source commit. It returns immutable value objects rather than passing raw dictionaries through backend code.

The repository carries no pretend production pins. A source checkout or ad-hoc Git wheel carries a development: true marker and selects local infrastructure builds, preserving the contributor and current install.sh paths. The release packaging workflow must replace that marker with the validated schema above; it refuses mutable or incomplete release inputs.

Packaging order

The release job operates on one tested commit:

1. Build the multi-architecture orchestrator and gateway OCI images.
2. Push them and resolve their registry manifest digests.
3. Build Firecracker's orchestrator and gateway rootfs artifacts from the same source and pinned OCI bases.
4. Publish both generic-package artifacts and retain their versions and compressed-file checksums.
5. Generate release-manifest.json from those published identities.
6. Build the wheel with that manifest. Do not produce a source distribution: rebuilding an sdist outside this release boundary could replace the pins with the development manifest.
7. Install the wheel in a clean environment and assert that every manifest identity is valid and retrievable.
8. Publish only that verified wheel.

The build hook receives the manifest as a file input. It must not query a registry or choose versions itself: package builds stay deterministic and network-independent once their release inputs exist.

Commit-addressed bundle index

Every publication resolves its requested branch, tag, or commit to one 40-character source SHA before doing any work. The publisher writes an immutable external bundle index under that SHA. The index contains:

• schema version and source SHA;
• wheel URL and SHA-256;
• orchestrator and gateway OCI digest references;
• Firecracker package versions and compressed-file SHA-256 values;
• producing workflow/run identity and publication timestamp; and
• qualification records, if any, naming the test workflow and release tag.

The embedded wheel manifest contains the runtime subset of the same identities. Publication fails if the embedded manifest and external index differ.

Bundle coordinates are immutable. Re-running publication for a SHA verifies and reuses a byte-identical complete bundle; it never replaces one artifact or mixes outputs from different runs. A partial existing bundle is an error and requires an explicit administrative cleanup before retrying.

Commit bundles are durable release inputs, not expiring CI artifacts. The project may garbage-collect unqualified snapshots only under a documented retention policy; qualified staging and production bundles are retained.

Branches, tags, and promotion

The protected promotion topology is:

feature -> main -> staging -> production
                       |           |
                 vX.Y.Z-rc.N     vX.Y.Z

Changes reach staging and production through promotion pull requests; those branches do not accept direct development commits. A staging tag must point to a commit reachable from staging and uses vX.Y.Z-rc.N. A production tag must point to a commit reachable from production and uses vX.Y.Z.

Channels are small, mutable pointers to immutable qualified bundles:

• production points to the newest promoted stable tag;
• staging points to the newest promoted release-candidate tag; and
• main is not a release channel; its commits are installable snapshots.

Channel updates are serialized and monotonic. Moving a channel to an older release requires a distinct, explicit rollback operation that records the reason and target. Deleting or moving a Git tag does not mutate a published bundle or channel silently.

Promotion reuses the exact bundle selected by source SHA. It does not rebuild the wheel, OCI images, or Firecracker artifacts. Thus production runs the same bytes qualified in staging.

Publication workflows

publish-artifacts is manually dispatchable for any branch, tag, or commit. It resolves the input to a source SHA, checks out that SHA, builds and publishes all infrastructure artifacts, generates the bundle index and embedded manifest, builds the wheel, installs the wheel in a clean environment, and publishes the verified wheel. It does not create a release or update staging or production. Its output is an unqualified snapshot and publication is serialized per source SHA.

After required CI succeeds, main may invoke the same publication operation automatically. Manual publication remains available so a branch can be tested before merge. Automatic and manual runs converge on the same idempotent bundle.

The release workflow is triggered by an eligible staging or production tag:

1. Resolve and validate the tag, source SHA, tag shape, and branch reachability.
2. Run the complete pre-release suite against that SHA, including Docker, Firecracker, and Apple Container.
3. After the suite passes, ensure the immutable commit bundle exists. Build it then if absent; otherwise verify every existing identity and checksum.
4. Install using the published installer and bundle index on clean supported hosts, and run the post-install smoke test against the acquired artifacts.
5. Attach a qualification record to the bundle, create the Gitea release, and advance the matching channel pointer.

No release artifact or channel update occurs before qualification succeeds. Jobs use per-SHA and per-channel concurrency locks so concurrent dispatches cannot publish twice or race a promotion.

Installer selection and warnings

The installer installs the latest production channel by default. It accepts:

• BOT_BOTTLE_CHANNEL=production or staging to select the latest release in that channel;
• BOT_BOTTLE_REF=<40-character-sha> to install an exact commit bundle; and
• BOT_BOTTLE_VERSION=<tag> to install an exact qualified release.

Selectors are mutually exclusive. Branch names are accepted by the publication workflow but not by the installer because they are mutable; callers resolve and publish them first, then install the reported SHA.

An exact-commit install prints a prominent warning that the snapshot may not have passed release qualification, even if the same SHA is later associated with a tag. A tag or channel install suppresses that warning only when the external index contains a successful qualification record for that tag and the tag selects the same bundle SHA.

The installer downloads the wheel named by the external index, verifies its SHA-256 before invoking pipx or pip, and reports the selected channel/tag, source SHA, and bundle identity. It never builds a Git checkout for a packaged install and never trusts a mutable channel response without resolving it to an immutable bundle index.

Backend-neutral lifecycle

Rename the host-side control-plane lifecycle operation from ensure_built() to ensure_available(). The production meaning is "make the package-selected artifact locally available and verified." Infrastructure composers call it before starting either plane.

The gateway receives the same contract so startup does not continue to rebuild the adjacent half of the fixed infrastructure pair.

A source checkout is itself a development boundary and selects the existing builders. BOT_BOTTLE_INFRA_BUILD=local provides the same explicit override for release debugging. Both paths deliberately bypass release metadata. Packaged production startup never silently falls back from a failed pull to a local build.

Docker

The Docker backend runs docker pull with each digest-qualified manifest reference. Docker may reuse its content-addressed local store. Containers are created from the immutable reference, and the running container's image ID is compared with the locally resolved ID for currentness.

The orchestrator source-hash label is replaced by a release-identity label. The label is diagnostic; the image ID comparison remains authoritative.

Apple Container

The Apple Container backend pulls the same multi-architecture OCI references. Because Apple Container's accepted syntax for pull, tag, inspect, and run is not identical to Docker's, one utility owns normalization from a digest reference to the local runnable name. Integration coverage on the macOS runner guards that adapter.

Production orchestrator launch removes the build-root bind mount, PYTHONPATH, and BOT_BOTTLE_SOURCE_HASH. It runs only the code baked into the pulled image. Currentness is based on the packaged release identity/image ID.

Firecracker

The publishing tool may continue computing content-derived artifact versions; that is a producer concern. Runtime launch instead reads the version and expected compressed checksum from the release manifest.

The artifact cache validates the packaged expected checksum even when a previous .verified marker exists. The marker records the checksum it validated, rather than an unqualified ok, so changing package metadata cannot adopt an artifact verified against a different expectation.

The existing candidate-directory and explicit local-build paths remain for CI and development. Candidate metadata must match the packaged identity unless the caller selected local development mode.

Overrides and failure behavior

BOT_BOTTLE_ORCHESTRATOR_IMAGE and BOT_BOTTLE_GATEWAY_IMAGE remain useful for testing but production accepts them only when they are digest-qualified. Mutable tag overrides require explicit local-build mode.

Artifact acquisition errors identify the packaged reference and backend. No backend catches an acquisition failure and rebuilds from local source.

Testing strategy

• Unit-test manifest parsing, immutability validation, missing-manifest errors, package-data inclusion, and deterministic build-hook injection.
• Assert Docker and Apple production preparation pulls and never builds.
• Assert explicit local mode retains existing build behavior.
• Assert the Apple orchestrator command has no source bind mount or PYTHONPATH.
• Assert running-image mismatches recreate infrastructure while matching, healthy instances are retained.
• Assert Firecracker uses packaged versions/checksums and rejects mismatches in downloaded, cached, and candidate artifacts.
• Extend wheel-install coverage to inspect the installed manifest.
• Run the full unit and Docker integration gates; run advisory Firecracker and macOS integration before publishing a release.
• Test ref resolution, tag-shape and branch-reachability enforcement, bundle idempotency, partial-publication rejection, and per-SHA concurrency.
• Test installer channel, tag, and exact-commit selection; checksum rejection; mutually exclusive selectors; and snapshot warning behavior.
• Test that release qualification happens before publication and that a failed suite or clean-install smoke test cannot create a release or move a channel.
• Test promotion reuses the qualified bundle byte-for-byte and that monotonic channel movement rejects an accidental rollback.

Rollout

Land runtime, installer, bundle publication, and release qualification support together, but do not move the production installer away from its current source-install path until an end-to-end staging release has passed on Docker, Firecracker, and Apple Container.

Create and protect staging and production, then promote one commit from main through both branches. Publish its commit bundle, qualify an RC tag, perform clean installs through the staging channel, promote the same SHA, qualify a stable tag, and only then make the production channel the installer's default. The stable promotion must demonstrate that every production artifact identity matches the staging-qualified bundle.

Existing source checkouts use explicit local mode. Existing mutable local images are ignored by production acquisition and may be pruned independently. Snapshot retention and rollback procedures must be documented before automatic publication from every successful main commit is enabled.