bot-bottle/docs/prds/0083-packaged-infra-artifacts.md

# PRD 0083: Packaged infrastructure artifacts

- **Status:** Draft
- **Author:** Codex
- **Created:** 2026-07-27
- **Issue:** #536

## Summary

Publish the orchestrator and gateway infrastructure before packaging
bot-bottle, record their immutable identities in the application package, and
make every production backend acquire those exact artifacts instead of
building infrastructure during bottle startup. Docker and Apple Container pull
digest-pinned OCI images; Firecracker pulls checksum-verified rootfs artifacts
selected by the same packaged release manifest.

## Problem

Starting a bottle currently invokes an infrastructure build on Docker and
Apple Container even when a healthy orchestrator is already running. Both
infra composers call `ensure_built()` before their health/currentness checks,
and both implementations unconditionally invoke their OCI builder. Layer
caching can make this less expensive, but startup still depends on a working
builder, build context, base-image registry, and package installation network.

The Apple Container orchestrator also bind-mounts the installed source over the
package baked into its image. A pinned image alone would therefore not pin the
code that actually runs.

Firecracker already downloads fixed rootfs artifacts, but it derives their
versions from the package contents at launch. There is no package-level record
that says which already-published infrastructure release belongs to an
installed bot-bottle release.

The result is three different delivery contracts:

- Docker and Apple Container build mutable `:latest` images on the launch host.
- Firecracker computes an artifact version locally and downloads it.
- An installed wheel contains Dockerfiles specifically so production startup
  can rebuild its own infrastructure.

That makes startup slower and less reliable, prevents a packaged application
from identifying the infrastructure it was tested with, and weakens the
supply-chain boundary between release production and runtime.

## Goals / Success criteria

- A packaged bot-bottle release contains one validated release manifest with
  immutable orchestrator and gateway identities.
- Docker and Apple Container acquire and run the manifest's digest-pinned OCI
  images without invoking `docker build` or `container build`.
- Firecracker acquires the manifest's versioned, checksum-verified
  orchestrator and gateway rootfs artifacts rather than deriving versions at
  launch.
- All three backends expose the same backend-neutral `ensure_available`
  lifecycle contract.
- A healthy running orchestrator is retained only when its recorded release
  identity matches the installed package.
- Apple Container runs the package baked into the orchestrator image; no host
  source overlays production code.
- Missing, mutable, malformed, or internally inconsistent release metadata
  fails closed before infrastructure starts.
- Source-checkout development retains an explicit local-build mode.
- Publishing produces the infrastructure first and the installable Python
  package last, so the package cannot name artifacts that were not published.
- Any branch, tag, or commit can be resolved once to a full source SHA and
  published as one immutable, commit-addressed artifact bundle.
- The installer defaults to the production channel, can select staging or an
  exact commit, and warns for commit snapshots that have not been qualified as
  a release.
- Staging and production releases promote the same tested artifact bytes;
  promotion never rebuilds them.
- A release is published only after the complete pre-release suite and a clean
  install from the published bundle succeed.

## Non-goals

- Changing agent/provider image builds. User Dockerfiles continue to build
  through the backend-specific agent-image paths.
- Installing Docker, Apple Container, Firecracker, or other host prerequisites.
- Replacing Gitea's OCI or generic-package registries.
- Adding image signing or a transparency log. Immutable digests and existing
  Firecracker SHA-256 verification are the integrity boundary for this version.
- Forcing a network download when an immutable, verified artifact is already
  cached locally. "Pull" means acquire the packaged identity, with safe cache
  reuse.
- Supporting live source overlays in packaged production installs.
- Defining the project's version-numbering policy beyond stable
  `vX.Y.Z` and staging `vX.Y.Z-rc.N` tag shapes.

## Design

### Release manifest

`bot_bottle/release-manifest.json` is generated during package production and
shipped as package data:

```json
{
  "schema": 1,
  "source_commit": "<40 lowercase hex>",
  "oci": {
    "orchestrator": "gitea.dideric.is/didericis/bot-bottle-orchestrator@sha256:<64 hex>",
    "gateway": "gitea.dideric.is/didericis/bot-bottle-gateway@sha256:<64 hex>"
  },
  "firecracker": {
    "orchestrator": {
      "version": "<artifact version>",
      "sha256": "<rootfs.ext4.gz sha256>"
    },
    "gateway": {
      "version": "<artifact version>",
      "sha256": "<rootfs.ext4.gz sha256>"
    }
  }
}
```

The runtime loader accepts only schema 1, digest-qualified OCI references,
non-empty Firecracker versions, 64-character lowercase SHA-256 values, and a
full source commit. It returns immutable value objects rather than passing raw
dictionaries through backend code.

The repository carries no pretend production pins. A source checkout or ad-hoc
Git wheel carries a `development: true` marker and selects local infrastructure
builds, preserving the contributor and current `install.sh` paths. The release
packaging workflow must replace that marker with the validated schema above;
it refuses mutable or incomplete release inputs.

### Packaging order

The release job operates on one tested commit:

1. Build the multi-architecture orchestrator and gateway OCI images.
2. Push them and resolve their registry manifest digests.
3. Build Firecracker's orchestrator and gateway rootfs artifacts from the same
   source and pinned OCI bases.
4. Publish both generic-package artifacts and retain their versions and
   compressed-file checksums.
5. Generate `release-manifest.json` from those published identities.
6. Build the wheel with that manifest. Do not produce a source distribution:
   rebuilding an sdist outside this release boundary could replace the pins
   with the development manifest.
7. Install the wheel in a clean environment and assert that every manifest
   identity is valid and retrievable.
8. Publish only that verified wheel.

The build hook receives the manifest as a file input. It must not query a
registry or choose versions itself: package builds stay deterministic and
network-independent once their release inputs exist.

### Commit-addressed bundle index

Every publication resolves its requested branch, tag, or commit to one
40-character source SHA before doing any work. The publisher writes an
immutable external bundle index under that SHA. The index contains:

- schema version and source SHA;
- wheel URL and SHA-256;
- orchestrator and gateway OCI digest references;
- Firecracker package versions and compressed-file SHA-256 values;
- producing workflow/run identity and publication timestamp; and
- qualification records, if any, naming the test workflow and release tag.

The embedded wheel manifest contains the runtime subset of the same identities.
Publication fails if the embedded manifest and external index differ.

Bundle coordinates are immutable. Re-running publication for a SHA verifies
and reuses a byte-identical complete bundle; it never replaces one artifact or
mixes outputs from different runs. A partial existing bundle is an error and
requires an explicit administrative cleanup before retrying.

Commit bundles are durable release inputs, not expiring CI artifacts. The
project may garbage-collect unqualified snapshots only under a documented
retention policy; qualified staging and production bundles are retained.

### Branches, tags, and promotion

The protected promotion topology is:

```text
feature -> main -> staging -> production
                       |           |
                 vX.Y.Z-rc.N     vX.Y.Z
```

Changes reach `staging` and `production` through promotion pull requests; those
branches do not accept direct development commits. A staging tag must point to
a commit reachable from `staging` and uses `vX.Y.Z-rc.N`. A production tag must
point to a commit reachable from `production` and uses `vX.Y.Z`.

Channels are small, mutable pointers to immutable qualified bundles:

- `production` points to the newest promoted stable tag;
- `staging` points to the newest promoted release-candidate tag; and
- `main` is not a release channel; its commits are installable snapshots.

Channel updates are serialized and monotonic. Moving a channel to an older
release requires a distinct, explicit rollback operation that records the
reason and target. Deleting or moving a Git tag does not mutate a published
bundle or channel silently.

Promotion reuses the exact bundle selected by source SHA. It does not rebuild
the wheel, OCI images, or Firecracker artifacts. Thus production runs the same
bytes qualified in staging.

### Publication workflows

`publish-artifacts` is manually dispatchable for any branch, tag, or commit.
It resolves the input to a source SHA, checks out that SHA, builds and publishes
all infrastructure artifacts, generates the bundle index and embedded
manifest, builds the wheel, installs the wheel in a clean environment, and
publishes the verified wheel. It does not create a release or update staging or
production. Its output is an unqualified snapshot and publication is
serialized per source SHA.

After required CI succeeds, `main` may invoke the same publication operation
automatically. Manual publication remains available so a branch can be tested
before merge. Automatic and manual runs converge on the same idempotent bundle.

The release workflow is triggered by an eligible staging or production tag:

1. Resolve and validate the tag, source SHA, tag shape, and branch reachability.
2. Run the complete pre-release suite against that SHA, including Docker,
   Firecracker, and Apple Container.
3. After the suite passes, ensure the immutable commit bundle exists. Build it
   then if absent; otherwise verify every existing identity and checksum.
4. Install using the published installer and bundle index on clean supported
   hosts, and run the post-install smoke test against the acquired artifacts.
5. Attach a qualification record to the bundle, create the Gitea release, and
   advance the matching channel pointer.

No release artifact or channel update occurs before qualification succeeds.
Jobs use per-SHA and per-channel concurrency locks so concurrent dispatches
cannot publish twice or race a promotion.

### Installer selection and warnings

The installer installs the latest `production` channel by default. It accepts:

- `BOT_BOTTLE_CHANNEL=production` or `staging` to select the latest release in
  that channel;
- `BOT_BOTTLE_REF=<40-character-sha>` to install an exact commit bundle; and
- `BOT_BOTTLE_VERSION=<tag>` to install an exact qualified release.

Selectors are mutually exclusive. Branch names are accepted by the publication
workflow but not by the installer because they are mutable; callers resolve
and publish them first, then install the reported SHA.

An exact-commit install prints a prominent warning that the snapshot may not
have passed release qualification, even if the same SHA is later associated
with a tag. A tag or channel install suppresses that warning only when the
external index contains a successful qualification record for that tag and
the tag selects the same bundle SHA.

The installer downloads the wheel named by the external index, verifies its
SHA-256 before invoking pipx or pip, and reports the selected channel/tag,
source SHA, and bundle identity. It never builds a Git checkout for a packaged
install and never trusts a mutable channel response without resolving it to an
immutable bundle index.

### Backend-neutral lifecycle

Rename the host-side control-plane lifecycle operation from `ensure_built()` to
`ensure_available()`. The production meaning is "make the package-selected
artifact locally available and verified." Infrastructure composers call it
before starting either plane.

The gateway receives the same contract so startup does not continue to rebuild
the adjacent half of the fixed infrastructure pair.

A source checkout is itself a development boundary and selects the existing
builders. `BOT_BOTTLE_INFRA_BUILD=local` provides the same explicit override
for release debugging. Both paths deliberately bypass release metadata.
Packaged production startup never silently falls back from a failed pull to a
local build.

### Docker

The Docker backend runs `docker pull` with each digest-qualified manifest
reference. Docker may reuse its content-addressed local store. Containers are
created from the immutable reference, and the running container's image ID is
compared with the locally resolved ID for currentness.

The orchestrator source-hash label is replaced by a release-identity label.
The label is diagnostic; the image ID comparison remains authoritative.

### Apple Container

The Apple Container backend pulls the same multi-architecture OCI references.
Because Apple Container's accepted syntax for pull, tag, inspect, and run is
not identical to Docker's, one utility owns normalization from a digest
reference to the local runnable name. Integration coverage on the macOS runner
guards that adapter.

Production orchestrator launch removes the build-root bind mount, `PYTHONPATH`,
and `BOT_BOTTLE_SOURCE_HASH`. It runs only the code baked into the pulled
image. Currentness is based on the packaged release identity/image ID.

### Firecracker

The publishing tool may continue computing content-derived artifact versions;
that is a producer concern. Runtime launch instead reads the version and
expected compressed checksum from the release manifest.

The artifact cache validates the packaged expected checksum even when a
previous `.verified` marker exists. The marker records the checksum it
validated, rather than an unqualified `ok`, so changing package metadata cannot
adopt an artifact verified against a different expectation.

The existing candidate-directory and explicit local-build paths remain for CI
and development. Candidate metadata must match the packaged identity unless
the caller selected local development mode.

### Overrides and failure behavior

`BOT_BOTTLE_ORCHESTRATOR_IMAGE` and `BOT_BOTTLE_GATEWAY_IMAGE` remain useful
for testing but production accepts them only when they are digest-qualified.
Mutable tag overrides require explicit local-build mode.

Artifact acquisition errors identify the packaged reference and backend. No
backend catches an acquisition failure and rebuilds from local source.

## Testing strategy

- Unit-test manifest parsing, immutability validation, missing-manifest errors,
  package-data inclusion, and deterministic build-hook injection.
- Assert Docker and Apple production preparation pulls and never builds.
- Assert explicit local mode retains existing build behavior.
- Assert the Apple orchestrator command has no source bind mount or
  `PYTHONPATH`.
- Assert running-image mismatches recreate infrastructure while matching,
  healthy instances are retained.
- Assert Firecracker uses packaged versions/checksums and rejects mismatches in
  downloaded, cached, and candidate artifacts.
- Extend wheel-install coverage to inspect the installed manifest.
- Run the full unit and Docker integration gates; run advisory Firecracker and
  macOS integration before publishing a release.
- Test ref resolution, tag-shape and branch-reachability enforcement, bundle
  idempotency, partial-publication rejection, and per-SHA concurrency.
- Test installer channel, tag, and exact-commit selection; checksum rejection;
  mutually exclusive selectors; and snapshot warning behavior.
- Test that release qualification happens before publication and that a failed
  suite or clean-install smoke test cannot create a release or move a channel.
- Test promotion reuses the qualified bundle byte-for-byte and that monotonic
  channel movement rejects an accidental rollback.

## Rollout

Land runtime, installer, bundle publication, and release qualification support
together, but do not move the production installer away from its current
source-install path until an end-to-end staging release has passed on Docker,
Firecracker, and Apple Container.

Create and protect `staging` and `production`, then promote one commit from
`main` through both branches. Publish its commit bundle, qualify an RC tag,
perform clean installs through the staging channel, promote the same SHA,
qualify a stable tag, and only then make the production channel the installer's
default. The stable promotion must demonstrate that every production artifact
identity matches the staging-qualified bundle.

Existing source checkouts use explicit local mode. Existing mutable local
images are ignored by production acquisition and may be pruned independently.
Snapshot retention and rollback procedures must be documented before automatic
publication from every successful `main` commit is enabled.