feat(prd-0081): reprovision CA, git-gate, and egress tokens on gateway bring-up
prd-number-check / require-numbered-prds (pull_request) Successful in 13s
tracker-policy-pr / check-pr (pull_request) Successful in 7s
test / image-input-builds (pull_request) Successful in 41s
test / unit (pull_request) Successful in 51s
test / integration-docker (pull_request) Successful in 58s
test / coverage (pull_request) Successful in 41s
test / image-input-builds (push) Successful in 48s
test / unit (push) Successful in 56s
lint / lint (push) Successful in 1m2s
Update Quality Badges / update-badges (push) Successful in 1m4s
test / integration-docker (push) Failing after 2m53s
test / coverage (push) Has been skipped
prd-number-check / require-numbered-prds (pull_request) Successful in 13s
tracker-policy-pr / check-pr (pull_request) Successful in 7s
test / image-input-builds (pull_request) Successful in 41s
test / unit (pull_request) Successful in 51s
test / integration-docker (pull_request) Successful in 58s
test / coverage (pull_request) Successful in 41s
test / image-input-builds (push) Successful in 48s
test / unit (push) Successful in 56s
lint / lint (push) Successful in 1m2s
Update Quality Badges / update-badges (push) Successful in 1m4s
test / integration-docker (push) Failing after 2m53s
test / coverage (push) Has been skipped
On a Firecracker gateway cold boot, reconcile every live agent VM against the fresh gateway: push the new mitmproxy CA into each agent's trust store, re-provision git-gate repos/creds from the persisted upstreams snapshot, and restore egress tokens. Per-bottle failures are logged and skipped so one unreachable VM does not block the rest. New modules / changes: - backend/firecracker/reconcile.py: attach_bottled_agents_to_gateway, _push_ca, _reprovision_git_gate, _guest_ip_from_config - git_gate/provision.py: write upstreams.json after key provisioning so the bring-up reconcile can reconstruct the upstream table without the manifest - backend/firecracker/infra.py: call attach_bottled_agents_to_gateway in the cold-boot branch of ensure_running() - backend/base.py: no-op default on BottleBackend - backend/firecracker/consolidated_launch.py: remove superseded _reprovision_running_bottles / _guest_ip_from_config - orchestrator: OrchestratorCore.update_agent_secret + POST /bottles/<id>/secret + client.update_agent_secret (single-secret in-place update, the reusable primitive from the 2026-07-26 hotfix) - tests: 15 new tests in test_firecracker_reconcile.py; updated test_firecracker_infra.py cold-boot cases; stale FC reprovision tests removed from test_backend_secret_reprovision.py Closes #516.
This commit was merged in pull request #517.
This commit is contained in:
@@ -0,0 +1,117 @@
|
||||
# PRD 0081: Reprovision gateway-dependent state on gateway bring-up
|
||||
|
||||
- **Status:** Active
|
||||
- **Author:** claude
|
||||
- **Created:** 2026-07-26
|
||||
- **Issue:** #516
|
||||
|
||||
## Summary
|
||||
|
||||
When the gateway is (re)built, reconcile every already-running bottle against the
|
||||
fresh gateway instead of persisting the gateway's state. On a gateway cold boot,
|
||||
the gateway reconciles all live bottles in one flow: **replace** each agent's
|
||||
trusted CA with the freshly-minted gateway CA, **re-provision** each bottle's
|
||||
git-gate repos + creds onto the gateway, and **restore** each bottle's egress
|
||||
tokens. One mechanism across all three services; the CA rotates for free on every
|
||||
bring-up.
|
||||
|
||||
## Problem
|
||||
|
||||
A gateway rebuild/restart silently breaks every already-running bottle:
|
||||
|
||||
- **CA (#510).** The gateway's mitmproxy mints a new CA on a fresh rootfs; agents
|
||||
still trust the old one, so egress fails TLS verification (`SSL certificate
|
||||
verification failed`).
|
||||
- **git-gate (#512).** Per-bottle bare repos (`/git/<id>`) + deploy creds
|
||||
(`/git-gate/creds/<id>`) live in the gateway's ephemeral rootfs; a rebuild wipes
|
||||
them and the agent 404s on fetch/push.
|
||||
|
||||
Both are the same root cause: per-bottle gateway-dependent state is provisioned
|
||||
**once, at bottle launch**, and nothing restores it for already-running bottles
|
||||
when the gateway comes back. The existing launch-time reprovision
|
||||
(`reprovision_bottles`) restores **only** egress tokens, and only as a side effect
|
||||
of the *next* launch.
|
||||
|
||||
Persisting the state (a host bind-mount / a per-VM data volume per service) was
|
||||
prototyped and rejected: it differs per service (a volume for the CA, another for
|
||||
git-gate, reprovision for tokens), it pins the CA static forever (no rotation),
|
||||
and it adds volume surface that `docker volume prune` / a wiped cache can silently
|
||||
destroy (the original #450 failure mode).
|
||||
|
||||
## Goals / Success Criteria
|
||||
|
||||
- A gateway (re)boot restores **all** running bottles' gateway-dependent state
|
||||
with no manual step and no relaunch — the agent's next egress / fetch / push
|
||||
just works.
|
||||
- **One** reconcile flow covering CA, git-gate, and egress tokens, rather than a
|
||||
different mechanism per service.
|
||||
- The CA **rotates** on every gateway bring-up (no long-lived CA), distributed to
|
||||
running agents by the same reconcile.
|
||||
- Per-bottle failures are tolerated: one unreachable or malformed bottle does not
|
||||
block the others or the gateway coming up.
|
||||
- **All backends** (Firecracker, docker, macOS) reconcile through the *same*
|
||||
contract — an abstract method on the backend base class, so a new backend
|
||||
cannot forget to implement it and none drifts onto a bespoke mechanism.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Deliberate mid-session CA rotation *without* a gateway restart — `rotate_ca`
|
||||
stays for that operator action.
|
||||
- Changing source-IP attribution, `/resolve`, or the plane split (#469).
|
||||
|
||||
## Design
|
||||
|
||||
**The contract — `attach_bottled_agents_to_gateway()` on the backend ABC.**
|
||||
Reconciling running bottles against the current gateway is a backend
|
||||
responsibility (only the backend can enumerate its agents and reach them —
|
||||
firecracker over SSH, docker/macOS over `exec`/`cp`), so it is an
|
||||
`@abc.abstractmethod` on `BottleBackend` (`backend/base.py`). Every backend
|
||||
implements it; the host calls it whenever the gateway is (re)brought up. This is
|
||||
what makes the fix cross-backend by construction rather than a per-backend
|
||||
follow-up. It reprovisions **all** registered bottles' gateway-dependent state:
|
||||
CA, git-gate, and egress tokens.
|
||||
|
||||
**Trigger — the gateway bring-up path.** The host calls
|
||||
`attach_bottled_agents_to_gateway()` only when the gateway was actually
|
||||
(re)brought up — the cold-boot branch of the infra bring-up (e.g.
|
||||
`FirecrackerInfraService.ensure_running` after it boots a fresh pair), never on an
|
||||
adopt of a healthy, current gateway (state intact). So it fires exactly when the
|
||||
gateway was (re)booted — including orchestrator restarts, since the pair boots
|
||||
together — and there is no bare-restart path that bypasses bring-up.
|
||||
|
||||
**Per-backend implementation.** Each backend's `attach_bottled_agents_to_gateway`
|
||||
enumerates its live bottles and, for each, reconciles the three services against
|
||||
the current gateway. The firecracker implementation, once the gateway VM is up
|
||||
and its CA is available:
|
||||
|
||||
1. Map each live bottle's guest IP → `bottle_id` from the orchestrator registry
|
||||
(`list_bottles`).
|
||||
2. Install the **current** shared git-gate hooks (`git_gate_render_hook` /
|
||||
`git_gate_render_access_hook`) into the fresh gateway once — rendered from
|
||||
code, never from a bottle's possibly-stale state dir.
|
||||
3. For each live agent VM (enumerated from its run dir):
|
||||
- **CA:** SSH the current gateway CA into the agent's trust store and run
|
||||
`update-ca-certificates` (unconditional replace — there is one gateway, so no
|
||||
fingerprint match is needed).
|
||||
- **git-gate:** rebuild the bottle's upstreams from its persisted git-gate
|
||||
state dir (deploy key, known_hosts, upstream URL) and re-init its bare repos
|
||||
+ per-repo creds under `/git/<bottle_id>`.
|
||||
- **egress token:** read the agent's `ENV_VAR_SECRET` and feed
|
||||
`reprovision_bottles`, restoring the orchestrator's in-memory tokens.
|
||||
4. Every per-bottle step is wrapped so one failure is logged and skipped.
|
||||
|
||||
**Retire the persistence prototype.** No CA data volume, no git-gate data volume
|
||||
(the abandoned PRs #511 / #513). The launch-time `_reprovision_running_bottles`
|
||||
call folds into this bring-up reconcile, so egress tokens are restored on the same
|
||||
cold-boot trigger (an adopt needs no restore — the orchestrator never restarted).
|
||||
|
||||
**CA rotation.** Because the gateway rootfs is ephemeral, every cold boot mints a
|
||||
fresh CA; reconcile is what distributes it, so a routine gateway rebuild doubles
|
||||
as a CA rotation with zero extra machinery.
|
||||
|
||||
## Open questions
|
||||
|
||||
- Docker's existing host-bind-mounted CA (`host_gateway_ca_dir`): once docker's
|
||||
`attach_bottled_agents_to_gateway` pushes the CA to running agents on bring-up,
|
||||
the bind-mount is redundant — drop it (so docker rotates like firecracker) or
|
||||
keep it as belt-and-suspenders? Leaning drop, for one behaviour across backends.
|
||||
@@ -1,374 +0,0 @@
|
||||
# PRD 0083: Packaged infrastructure artifacts
|
||||
|
||||
- **Status:** Active
|
||||
- **Author:** Codex
|
||||
- **Created:** 2026-07-27
|
||||
- **Issue:** #536
|
||||
|
||||
## Summary
|
||||
|
||||
Publish the orchestrator and gateway infrastructure before packaging
|
||||
bot-bottle, record their immutable identities in the application package, and
|
||||
make every production backend acquire those exact artifacts instead of
|
||||
building infrastructure during bottle startup. Docker and Apple Container pull
|
||||
digest-pinned OCI images; Firecracker pulls checksum-verified rootfs artifacts
|
||||
selected by the same packaged release manifest.
|
||||
|
||||
## Problem
|
||||
|
||||
Starting a bottle currently invokes an infrastructure build on Docker and
|
||||
Apple Container even when a healthy orchestrator is already running. Both
|
||||
infra composers call `ensure_built()` before their health/currentness checks,
|
||||
and both implementations unconditionally invoke their OCI builder. Layer
|
||||
caching can make this less expensive, but startup still depends on a working
|
||||
builder, build context, base-image registry, and package installation network.
|
||||
|
||||
The Apple Container orchestrator also bind-mounts the installed source over the
|
||||
package baked into its image. A pinned image alone would therefore not pin the
|
||||
code that actually runs.
|
||||
|
||||
Firecracker already downloads fixed rootfs artifacts, but it derives their
|
||||
versions from the package contents at launch. There is no package-level record
|
||||
that says which already-published infrastructure release belongs to an
|
||||
installed bot-bottle release.
|
||||
|
||||
The result is three different delivery contracts:
|
||||
|
||||
- Docker and Apple Container build mutable `:latest` images on the launch host.
|
||||
- Firecracker computes an artifact version locally and downloads it.
|
||||
- An installed wheel contains Dockerfiles specifically so production startup
|
||||
can rebuild its own infrastructure.
|
||||
|
||||
That makes startup slower and less reliable, prevents a packaged application
|
||||
from identifying the infrastructure it was tested with, and weakens the
|
||||
supply-chain boundary between release production and runtime.
|
||||
|
||||
## Goals / Success criteria
|
||||
|
||||
- A packaged bot-bottle release contains one validated release manifest with
|
||||
immutable orchestrator and gateway identities.
|
||||
- Docker and Apple Container acquire and run the manifest's digest-pinned OCI
|
||||
images without invoking `docker build` or `container build`.
|
||||
- The shipped Claude, Codex, and Pi agent images are built, smoke-tested,
|
||||
published, and selected by digest as part of the same commit bundle.
|
||||
- Firecracker acquires the manifest's versioned, checksum-verified
|
||||
orchestrator and gateway rootfs artifacts rather than deriving versions at
|
||||
launch.
|
||||
- All three backends expose the same backend-neutral `ensure_available`
|
||||
lifecycle contract.
|
||||
- A healthy running orchestrator is retained only when its recorded release
|
||||
identity matches the installed package.
|
||||
- Apple Container runs the package baked into the orchestrator image; no host
|
||||
source overlays production code.
|
||||
- Missing, mutable, malformed, or internally inconsistent release metadata
|
||||
fails closed before infrastructure starts.
|
||||
- Source-checkout development retains an explicit local-build mode.
|
||||
- Publishing produces the infrastructure first and the installable Python
|
||||
package last, so the package cannot name artifacts that were not published.
|
||||
- Any branch, tag, or commit can be resolved once to a full source SHA and
|
||||
published as one immutable, commit-addressed artifact bundle.
|
||||
- The installer defaults to the production channel, can select staging or an
|
||||
exact commit, and warns for commit snapshots that have not been qualified as
|
||||
a release.
|
||||
- Staging and production releases promote the same tested artifact bytes;
|
||||
promotion never rebuilds them.
|
||||
- A release is published only after the complete pre-release suite and a clean
|
||||
install from the published bundle succeed.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Prebuilding user-supplied Dockerfiles. Custom agent images continue to build
|
||||
through the backend-specific agent-image paths.
|
||||
- Installing Docker, Apple Container, Firecracker, or other host prerequisites.
|
||||
- Replacing Gitea's OCI or generic-package registries.
|
||||
- Adding image signing or a transparency log. Immutable digests and existing
|
||||
Firecracker SHA-256 verification are the integrity boundary for this version.
|
||||
- Forcing a network download when an immutable, verified artifact is already
|
||||
cached locally. "Pull" means acquire the packaged identity, with safe cache
|
||||
reuse.
|
||||
- Supporting live source overlays in packaged production installs.
|
||||
- Defining the project's version-numbering policy beyond stable
|
||||
`vX.Y.Z` and staging `vX.Y.Z-rc.N` tag shapes.
|
||||
|
||||
## Design
|
||||
|
||||
### Release manifest
|
||||
|
||||
`bot_bottle/release-manifest.json` is generated during package production and
|
||||
shipped as package data:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema": 1,
|
||||
"source_commit": "<40 lowercase hex>",
|
||||
"oci": {
|
||||
"orchestrator": "gitea.dideric.is/didericis/bot-bottle-orchestrator@sha256:<64 hex>",
|
||||
"gateway": "gitea.dideric.is/didericis/bot-bottle-gateway@sha256:<64 hex>",
|
||||
"agent_claude": "gitea.dideric.is/didericis/bot-bottle-claude@sha256:<64 hex>",
|
||||
"agent_codex": "gitea.dideric.is/didericis/bot-bottle-codex@sha256:<64 hex>",
|
||||
"agent_pi": "gitea.dideric.is/didericis/bot-bottle-pi@sha256:<64 hex>"
|
||||
},
|
||||
"firecracker": {
|
||||
"orchestrator": {
|
||||
"version": "<artifact version>",
|
||||
"sha256": "<rootfs.ext4.gz sha256>"
|
||||
},
|
||||
"gateway": {
|
||||
"version": "<artifact version>",
|
||||
"sha256": "<rootfs.ext4.gz sha256>"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The runtime loader accepts only schema 1, digest-qualified OCI references,
|
||||
non-empty Firecracker versions, 64-character lowercase SHA-256 values, and a
|
||||
full source commit. It returns immutable value objects rather than passing raw
|
||||
dictionaries through backend code.
|
||||
|
||||
The repository carries no pretend production pins. A source checkout or ad-hoc
|
||||
Git wheel carries a `development: true` marker and selects local infrastructure
|
||||
builds, preserving the contributor and current `install.sh` paths. The release
|
||||
packaging workflow must replace that marker with the validated schema above;
|
||||
it refuses mutable or incomplete release inputs.
|
||||
|
||||
### Packaging order
|
||||
|
||||
The release job operates on one tested commit:
|
||||
|
||||
1. Build and smoke-test the multi-architecture orchestrator, gateway, Claude,
|
||||
Codex, and Pi OCI images.
|
||||
2. Push them and resolve their registry manifest digests.
|
||||
3. Build Firecracker's orchestrator and gateway rootfs artifacts from the same
|
||||
source and pinned OCI bases.
|
||||
4. Publish both generic-package artifacts and retain their versions and
|
||||
compressed-file checksums.
|
||||
5. Generate `release-manifest.json` from those published identities.
|
||||
6. Build the wheel with that manifest. Do not produce a source distribution:
|
||||
rebuilding an sdist outside this release boundary could replace the pins
|
||||
with the development manifest.
|
||||
7. Install the wheel in a clean environment and assert that every manifest
|
||||
identity is valid and retrievable.
|
||||
8. Publish only that verified wheel.
|
||||
|
||||
The build hook receives the manifest as a file input. It must not query a
|
||||
registry or choose versions itself: package builds stay deterministic and
|
||||
network-independent once their release inputs exist.
|
||||
|
||||
### Commit-addressed bundle index
|
||||
|
||||
Every publication resolves its requested branch, tag, or commit to one
|
||||
40-character source SHA before doing any work. The publisher writes an
|
||||
immutable external bundle index under that SHA. The index contains:
|
||||
|
||||
- schema version and source SHA;
|
||||
- wheel URL and SHA-256;
|
||||
- orchestrator and gateway OCI digest references;
|
||||
- Claude, Codex, and Pi agent OCI digest references;
|
||||
- Firecracker package versions and compressed-file SHA-256 values;
|
||||
- producing workflow/run identity and publication timestamp; and
|
||||
- producing workflow provenance. Qualification is recorded separately in an
|
||||
immutable tag pointer so the commit bundle itself never changes.
|
||||
|
||||
The embedded wheel manifest contains the runtime subset of the same identities.
|
||||
Publication fails if the embedded manifest and external index differ.
|
||||
|
||||
Bundle coordinates are immutable. Re-running publication for a SHA verifies
|
||||
and reuses a byte-identical complete bundle; it never replaces one artifact or
|
||||
mixes outputs from different runs. A partial existing bundle is an error and
|
||||
requires an explicit administrative cleanup before retrying.
|
||||
|
||||
Commit bundles are durable release inputs, not expiring CI artifacts. The
|
||||
project may garbage-collect unqualified snapshots only under a documented
|
||||
retention policy; qualified staging and production bundles are retained.
|
||||
|
||||
### Branches, tags, and promotion
|
||||
|
||||
The protected promotion topology is:
|
||||
|
||||
```text
|
||||
feature -> main -> staging -> production
|
||||
| |
|
||||
vX.Y.Z-rc.N vX.Y.Z
|
||||
```
|
||||
|
||||
Changes reach `staging` and `production` through promotion pull requests; those
|
||||
branches do not accept direct development commits. A staging tag must point to
|
||||
a commit reachable from `staging` and uses `vX.Y.Z-rc.N`. A production tag must
|
||||
point to a commit reachable from `production` and uses `vX.Y.Z`.
|
||||
|
||||
Channels are small, mutable pointers to immutable qualified bundles:
|
||||
|
||||
- `production` points to the newest promoted stable tag;
|
||||
- `staging` points to the newest promoted release-candidate tag; and
|
||||
- `main` is not a release channel; its commits are installable snapshots.
|
||||
|
||||
Channel updates are serialized and monotonic. Moving a channel to an older
|
||||
release requires a distinct, explicit rollback operation that records the
|
||||
reason and target. Deleting or moving a Git tag does not mutate a published
|
||||
bundle or channel silently.
|
||||
|
||||
Promotion reuses the exact bundle selected by source SHA. It does not rebuild
|
||||
the wheel, OCI images, or Firecracker artifacts. Thus production runs the same
|
||||
bytes qualified in staging.
|
||||
|
||||
### Publication workflows
|
||||
|
||||
`publish-artifacts` is manually dispatchable for any branch, tag, or commit.
|
||||
It resolves the input to a source SHA, checks out that SHA, builds and publishes
|
||||
all infrastructure artifacts, generates the bundle index and embedded
|
||||
manifest, builds the wheel, installs the wheel in a clean environment, and
|
||||
publishes the verified wheel. It does not create a release or update staging or
|
||||
production. Its output is an unqualified snapshot and publication is
|
||||
serialized per source SHA.
|
||||
|
||||
After required CI succeeds, `main` may invoke the same publication operation
|
||||
automatically. Manual publication remains available so a branch can be tested
|
||||
before merge. Automatic and manual runs converge on the same idempotent bundle.
|
||||
|
||||
The release workflow is triggered by an eligible staging or production tag:
|
||||
|
||||
1. Resolve and validate the tag, source SHA, tag shape, and branch reachability.
|
||||
2. Run the complete pre-release suite against that SHA, including Docker,
|
||||
Firecracker, and Apple Container.
|
||||
3. After the suite passes, ensure the immutable commit bundle exists. Build it
|
||||
then if absent; otherwise verify every existing identity and checksum.
|
||||
4. Install using the published installer and bundle index on clean supported
|
||||
hosts, and run the post-install smoke test against the acquired artifacts.
|
||||
5. Publish an immutable qualified tag pointer, create the Gitea release, and
|
||||
advance the matching channel pointer.
|
||||
|
||||
No release artifact or channel update occurs before qualification succeeds.
|
||||
Jobs use per-SHA and per-channel concurrency locks so concurrent dispatches
|
||||
cannot publish twice or race a promotion.
|
||||
|
||||
### Installer selection and warnings
|
||||
|
||||
The installer installs the latest `production` channel by default. It accepts:
|
||||
|
||||
- `BOT_BOTTLE_CHANNEL=production` or `staging` to select the latest release in
|
||||
that channel;
|
||||
- `BOT_BOTTLE_REF=<40-character-sha>` to install an exact commit bundle; and
|
||||
- `BOT_BOTTLE_VERSION=<tag>` to install an exact qualified release.
|
||||
|
||||
Selectors are mutually exclusive. Branch names are accepted by the publication
|
||||
workflow but not by the installer because they are mutable; callers resolve
|
||||
and publish them first, then install the reported SHA.
|
||||
|
||||
An exact-commit install prints a prominent warning that the snapshot may not
|
||||
have passed release qualification, even if the same SHA is later associated
|
||||
with a tag. A tag or channel install suppresses that warning only when its
|
||||
validated pointer is marked qualified and selects the same immutable bundle
|
||||
SHA.
|
||||
|
||||
The installer downloads the wheel named by the external index, verifies its
|
||||
SHA-256 before invoking pipx or pip, and reports the selected channel/tag,
|
||||
source SHA, and bundle identity. It never builds a Git checkout for a packaged
|
||||
install and never trusts a mutable channel response without resolving it to an
|
||||
immutable bundle index.
|
||||
|
||||
### Backend-neutral lifecycle
|
||||
|
||||
Rename the host-side control-plane lifecycle operation from `ensure_built()` to
|
||||
`ensure_available()`. The production meaning is "make the package-selected
|
||||
artifact locally available and verified." Infrastructure composers call it
|
||||
before starting either plane.
|
||||
|
||||
The gateway receives the same contract so startup does not continue to rebuild
|
||||
the adjacent half of the fixed infrastructure pair.
|
||||
|
||||
A source checkout is itself a development boundary and selects the existing
|
||||
builders. `BOT_BOTTLE_INFRA_BUILD=local` provides the same explicit override
|
||||
for release debugging. Both paths deliberately bypass release metadata.
|
||||
Packaged production startup never silently falls back from a failed pull to a
|
||||
local build.
|
||||
|
||||
### Docker
|
||||
|
||||
The Docker backend runs `docker pull` with each digest-qualified manifest
|
||||
reference. Docker may reuse its content-addressed local store. Containers are
|
||||
created from the immutable reference, and the running container's image ID is
|
||||
compared with the locally resolved ID for currentness.
|
||||
|
||||
The orchestrator source-hash label is replaced by a release-identity label.
|
||||
The label is diagnostic; the image ID comparison remains authoritative.
|
||||
|
||||
### Apple Container
|
||||
|
||||
The Apple Container backend pulls the same multi-architecture OCI references.
|
||||
Because Apple Container's accepted syntax for pull, tag, inspect, and run is
|
||||
not identical to Docker's, one utility owns normalization from a digest
|
||||
reference to the local runnable name. Integration coverage on the macOS runner
|
||||
guards that adapter.
|
||||
|
||||
Production orchestrator launch removes the build-root bind mount, `PYTHONPATH`,
|
||||
and `BOT_BOTTLE_SOURCE_HASH`. It runs only the code baked into the pulled
|
||||
image. Currentness is based on the packaged release identity/image ID.
|
||||
|
||||
### Firecracker
|
||||
|
||||
The publishing tool may continue computing content-derived artifact versions;
|
||||
that is a producer concern. Runtime launch instead reads the version and
|
||||
expected compressed checksum from the release manifest.
|
||||
|
||||
The artifact cache validates the packaged expected checksum even when a
|
||||
previous `.verified` marker exists. The marker records the checksum it
|
||||
validated, rather than an unqualified `ok`, so changing package metadata cannot
|
||||
adopt an artifact verified against a different expectation.
|
||||
|
||||
The existing candidate-directory and explicit local-build paths remain for CI
|
||||
and development. Candidate metadata must match the packaged identity unless
|
||||
the caller selected local development mode.
|
||||
|
||||
### Overrides and failure behavior
|
||||
|
||||
`BOT_BOTTLE_ORCHESTRATOR_IMAGE` and `BOT_BOTTLE_GATEWAY_IMAGE` remain useful
|
||||
for testing but production accepts them only when they are digest-qualified.
|
||||
Mutable tag overrides require explicit local-build mode.
|
||||
|
||||
Artifact acquisition errors identify the packaged reference and backend. No
|
||||
backend catches an acquisition failure and rebuilds from local source.
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- Unit-test manifest parsing, immutability validation, missing-manifest errors,
|
||||
package-data inclusion, and deterministic build-hook injection.
|
||||
- Assert Docker and Apple production preparation pulls and never builds.
|
||||
- Assert explicit local mode retains existing build behavior.
|
||||
- Assert the Apple orchestrator command has no source bind mount or
|
||||
`PYTHONPATH`.
|
||||
- Assert running-image mismatches recreate infrastructure while matching,
|
||||
healthy instances are retained.
|
||||
- Assert Firecracker uses packaged versions/checksums and rejects mismatches in
|
||||
downloaded, cached, and candidate artifacts.
|
||||
- Extend wheel-install coverage to inspect the installed manifest.
|
||||
- Run the full unit and Docker integration gates; run advisory Firecracker and
|
||||
macOS integration before publishing a release.
|
||||
- Test ref resolution, tag-shape and branch-reachability enforcement, bundle
|
||||
idempotency, partial-publication rejection, and per-SHA concurrency.
|
||||
- Test installer channel, tag, and exact-commit selection; checksum rejection;
|
||||
mutually exclusive selectors; and snapshot warning behavior.
|
||||
- Test that release qualification happens before publication and that a failed
|
||||
suite or clean-install smoke test cannot create a release or move a channel.
|
||||
- Test promotion reuses the qualified bundle byte-for-byte and that monotonic
|
||||
channel movement rejects an accidental rollback.
|
||||
|
||||
## Rollout
|
||||
|
||||
Land runtime, installer, bundle publication, and release qualification support
|
||||
together, but do not move the production installer away from its current
|
||||
source-install path until an end-to-end staging release has passed on Docker,
|
||||
Firecracker, and Apple Container.
|
||||
|
||||
Create and protect `staging` and `production`, then promote one commit from
|
||||
`main` through both branches. Publish its commit bundle, qualify an RC tag,
|
||||
perform clean installs through the staging channel, promote the same SHA, and
|
||||
qualify a stable tag. Until that first stable qualification exists, the
|
||||
installer's production default fails closed because there is no production
|
||||
pointer. The stable promotion must demonstrate that every production artifact
|
||||
identity matches the staging-qualified bundle.
|
||||
|
||||
Existing source checkouts use explicit local mode. Existing mutable local
|
||||
images are ignored by production acquisition and may be pruned independently.
|
||||
Snapshot retention and rollback procedures must be documented before automatic
|
||||
publication from every successful `main` commit is enabled.
|
||||
Reference in New Issue
Block a user