bot-bottle/docs/prds/0083-packaged-infra-artifacts.md

PRD 0083: Packaged infrastructure artifacts

• Status: Active
• Author: Codex
• Created: 2026-07-27
• Issue: #536

Summary

Publish the orchestrator and gateway infrastructure before packaging bot-bottle, record their immutable identities in the application package, and make every production backend acquire those exact artifacts instead of building infrastructure during bottle startup. Docker and Apple Container pull digest-pinned OCI images; Firecracker pulls checksum-verified rootfs artifacts selected by the same packaged release manifest.

Problem

Starting a bottle currently invokes an infrastructure build on Docker and Apple Container even when a healthy orchestrator is already running. Both infra composers call ensure_built() before their health/currentness checks, and both implementations unconditionally invoke their OCI builder. Layer caching can make this less expensive, but startup still depends on a working builder, build context, base-image registry, and package installation network.

The Apple Container orchestrator also bind-mounts the installed source over the package baked into its image. A pinned image alone would therefore not pin the code that actually runs.

Firecracker already downloads fixed rootfs artifacts, but it derives their versions from the package contents at launch. There is no package-level record that says which already-published infrastructure release belongs to an installed bot-bottle release.

The result is three different delivery contracts:

• Docker and Apple Container build mutable :latest images on the launch host.
• Firecracker computes an artifact version locally and downloads it.
• An installed wheel contains Dockerfiles specifically so production startup can rebuild its own infrastructure.

That makes startup slower and less reliable, prevents a packaged application from identifying the infrastructure it was tested with, and weakens the supply-chain boundary between release production and runtime.

Goals / Success criteria

• A packaged bot-bottle release contains one validated release manifest with immutable orchestrator and gateway identities.
• Docker and Apple Container acquire and run the manifest's digest-pinned OCI images without invoking docker build or container build.
• Firecracker acquires the manifest's versioned, checksum-verified orchestrator and gateway rootfs artifacts rather than deriving versions at launch.
• All three backends expose the same backend-neutral ensure_available lifecycle contract.
• A healthy running orchestrator is retained only when its recorded release identity matches the installed package.
• Apple Container runs the package baked into the orchestrator image; no host source overlays production code.
• Missing, mutable, malformed, or internally inconsistent release metadata fails closed before infrastructure starts.
• Source-checkout development retains an explicit local-build mode.
• Publishing produces the infrastructure first and the installable Python package last, so the package cannot name artifacts that were not published.

Non-goals

• Changing agent/provider image builds. User Dockerfiles continue to build through the backend-specific agent-image paths.
• Installing Docker, Apple Container, Firecracker, or other host prerequisites.
• Replacing Gitea's OCI or generic-package registries.
• Adding image signing or a transparency log. Immutable digests and existing Firecracker SHA-256 verification are the integrity boundary for this version.
• Forcing a network download when an immutable, verified artifact is already cached locally. "Pull" means acquire the packaged identity, with safe cache reuse.
• Supporting live source overlays in packaged production installs.

Design

Release manifest

bot_bottle/release-manifest.json is generated during package production and shipped as package data:

{
  "schema": 1,
  "source_commit": "<40 lowercase hex>",
  "oci": {
    "orchestrator": "gitea.dideric.is/didericis/bot-bottle-orchestrator@sha256:<64 hex>",
    "gateway": "gitea.dideric.is/didericis/bot-bottle-gateway@sha256:<64 hex>"
  },
  "firecracker": {
    "orchestrator": {
      "version": "<artifact version>",
      "sha256": "<rootfs.ext4.gz sha256>"
    },
    "gateway": {
      "version": "<artifact version>",
      "sha256": "<rootfs.ext4.gz sha256>"
    }
  }
}

The runtime loader accepts only schema 1, digest-qualified OCI references, non-empty Firecracker versions, 64-character lowercase SHA-256 values, and a full source commit. It returns immutable value objects rather than passing raw dictionaries through backend code.

The repository carries no pretend production pins. A source checkout or ad-hoc Git wheel carries a development: true marker and selects local infrastructure builds, preserving the contributor and current install.sh paths. The release packaging workflow must replace that marker with the validated schema above; it refuses mutable or incomplete release inputs.

Packaging order

The release job operates on one tested commit:

1. Build the multi-architecture orchestrator and gateway OCI images.
2. Push them and resolve their registry manifest digests.
3. Build Firecracker's orchestrator and gateway rootfs artifacts from the same source and pinned OCI bases.
4. Publish both generic-package artifacts and retain their versions and compressed-file checksums.
5. Generate release-manifest.json from those published identities.
6. Build the wheel with that manifest. Do not produce a source distribution: rebuilding an sdist outside this release boundary could replace the pins with the development manifest.
7. Install the wheel in a clean environment and assert that every manifest identity is valid and retrievable.
8. Publish only that verified wheel.

The build hook receives the manifest as a file input. It must not query a registry or choose versions itself: package builds stay deterministic and network-independent once their release inputs exist.

Backend-neutral lifecycle

Rename the host-side control-plane lifecycle operation from ensure_built() to ensure_available(). The production meaning is "make the package-selected artifact locally available and verified." Infrastructure composers call it before starting either plane.

The gateway receives the same contract so startup does not continue to rebuild the adjacent half of the fixed infrastructure pair.

A source checkout is itself a development boundary and selects the existing builders. BOT_BOTTLE_INFRA_BUILD=local provides the same explicit override for release debugging. Both paths deliberately bypass release metadata. Packaged production startup never silently falls back from a failed pull to a local build.

Docker

The Docker backend runs docker pull with each digest-qualified manifest reference. Docker may reuse its content-addressed local store. Containers are created from the immutable reference, and the running container's image ID is compared with the locally resolved ID for currentness.

The orchestrator source-hash label is replaced by a release-identity label. The label is diagnostic; the image ID comparison remains authoritative.

Apple Container

The Apple Container backend pulls the same multi-architecture OCI references. Because Apple Container's accepted syntax for pull, tag, inspect, and run is not identical to Docker's, one utility owns normalization from a digest reference to the local runnable name. Integration coverage on the macOS runner guards that adapter.

Production orchestrator launch removes the build-root bind mount, PYTHONPATH, and BOT_BOTTLE_SOURCE_HASH. It runs only the code baked into the pulled image. Currentness is based on the packaged release identity/image ID.

Firecracker

The publishing tool may continue computing content-derived artifact versions; that is a producer concern. Runtime launch instead reads the version and expected compressed checksum from the release manifest.

The artifact cache validates the packaged expected checksum even when a previous .verified marker exists. The marker records the checksum it validated, rather than an unqualified ok, so changing package metadata cannot adopt an artifact verified against a different expectation.

The existing candidate-directory and explicit local-build paths remain for CI and development. Candidate metadata must match the packaged identity unless the caller selected local development mode.

Overrides and failure behavior

BOT_BOTTLE_ORCHESTRATOR_IMAGE and BOT_BOTTLE_GATEWAY_IMAGE remain useful for testing but production accepts them only when they are digest-qualified. Mutable tag overrides require explicit local-build mode.

Artifact acquisition errors identify the packaged reference and backend. No backend catches an acquisition failure and rebuilds from local source.

Testing strategy

• Unit-test manifest parsing, immutability validation, missing-manifest errors, package-data inclusion, and deterministic build-hook injection.
• Assert Docker and Apple production preparation pulls and never builds.
• Assert explicit local mode retains existing build behavior.
• Assert the Apple orchestrator command has no source bind mount or PYTHONPATH.
• Assert running-image mismatches recreate infrastructure while matching, healthy instances are retained.
• Assert Firecracker uses packaged versions/checksums and rejects mismatches in downloaded, cached, and candidate artifacts.
• Extend wheel-install coverage to inspect the installed manifest.
• Run the full unit and Docker integration gates; run advisory Firecracker and macOS integration before publishing a release.

Rollout

Land runtime and publishing support together, but do not publish an installable release until its infrastructure artifacts have been published and the generated manifest has passed clean-install validation. Existing source checkouts use explicit local mode. Existing mutable local images are ignored by production acquisition and may be pruned independently.