didericis/bot-bottle
1
0
Fork 0
Code Issues 33 Pull Requests 9 Actions Packages Releases Wiki Activity

Reprovision gateway-dependent state on gateway bring-up #516

New Issue
Closed
opened 2026-07-26 18:48:18 -04:00 by didericis-claude · 0 comments
didericis-claude commented 2026-07-26 18:48:18 -04:00
Collaborator
Copy Link
Copy Source

Umbrella issue: when the gateway is (re)built, reconcile every already-running bottle against the fresh gateway, rather than persisting gateway state on volumes.

Folds in #510 (CA) and #512 (git-gate), which are two symptoms of one root cause. Supersedes the abandoned persistent-volume PRs (#511 CA-volume, #513 git-gate-volume).

The root cause

Gateway-dependent state is provisioned into the gateway once, at bottle launch, and lives only in the gateway's ephemeral rootfs. Rebuilding or restarting the gateway wipes it, and nothing re-provisions the bottles that were already running. orchestrator/reprovision.py restores only egress tokens (reprovision_from_secret) — not the CA, not git-gate declarations — and the registry does not store the git upstreams or keys.

Symptom 1 — CA (was #510)

Rebuilding the Firecracker gateway mints a brand-new mitmproxy CA, breaking every already-running bottle with SSL certificate verification failed on egress.

The #450 fix ("the gateway CA must survive recreation") was implemented only for the Docker backend, via the persistent host_gateway_ca_dir() bind-mount. The Firecracker gateway had no equivalent: backend/firecracker/gateway.py booted the VM through infra_vm.boot_vm(...) with no data_drive — contrast the orchestrator VM, which gets data_drive=self._ensure_registry_volume() (persistent /dev/vdb). The CA lives at /home/mitmproxy/.mitmproxy/ inside the rootfs, which boot_vm re-materializes on every boot. Each bottle only captures a point-in-time CA snapshot at launch (launch.py → egress/gateway-ca/gateway-ca.pem), never re-synced when the gateway rotates.

Observed:

CA SHA-256 fingerprint Captured
Live gateway (rebuilt) 66:63:6F:06:94:… after rebuild
claude-agent-1 (relaunched with gateway) 66:63:6F:06:94:… ✅ after rebuild
claude-agent-3 (running since prior day) C6:B1:80:C0:22:… ❌ day before

agent-3 → rebuilt gateway bumps TLS with a leaf signed by 66:63:6F… → agent-3's trust store only has C6:B1:80… → handshake fails.

Symptom 2 — git-gate (was #512)

Restarting the gateway drops every already-running bottle's git-gate state, so its agent gets HTTP 404 from the git-gate smart-HTTP server (port 9420) on info/refs — fetch and push both fail.

Per-bottle repos and creds are provisioned at launch (consolidated_launch step 3 → provision_gateway / git_gate/render) under /git/<bottle_id> and /git-gate/creds/<bottle_id>. On firecracker these live in the per-boot rootfs; on docker only the CA is bind-mounted, not /git or /git-gate/creds. Either way a gateway restart wipes them.

Observed (firecracker): after a gateway rebuild the live gateway VM held exactly ONE bottle's repo — /git/<id>/bot-bottle.git, created at gateway boot — the bottle relaunched together with the gateway. Bottles predating the restart had no /git/<id> and no /git-gate/creds/<id>, so their agents 404'd. mount showed only /dev/vda (rootfs); nothing persistent.

Approach — reconcile, don't persist

One flow across all services, run when the gateway comes up:

  • CA — replace each running agent's trusted CA with the freshly-minted gateway CA. Agents never need to match a prior CA; there is one gateway. This lets the CA rotate for free on every bring-up.
  • git-gate — re-provision each bottle's bare repos + creds onto the gateway.
  • egress tokens — restore each bottle's egress tokens.

The trigger lives in the gateway startup flow (FirecrackerGateway.connect_to_orchestrator, the cold-boot path), so it fires exactly when the gateway was (re)booted and never on an adopt.

Reconciling also avoids the persistent-volume approach's one-time break: seeding a fresh CA volume still mints one final CA on the first restart after the fix, breaking running bottles once. Teardown already rm -rfs /git/<id> + creds (deprovision_git_gate), so there is no accumulating state to manage either.

Status / building blocks

  • PRD drafted
  • OrchestratorCore.update_agent_secret + POST /bottles/<id>/secret (single-secret in-place update — the reusable primitive from the 2026-07-26 hotfix)
  • Bring-up reconcile: CA push to running agents
  • Bring-up reconcile: git-gate re-provision
  • Wire the reconcile into connect_to_orchestrator

Out of scope

  • Refreshing short-lived host credentials (e.g. the Codex access token) as part of the bring-up reconcile — deliberately not a criterion here.

  • Docker bot-bottle-orch-gateway is attached to a stale CI CA volume (bot-bottle-ci-ca-<run>) because CI reuses the production gateway singleton name — separate latent issue.

  • orchestrator/rotate_ca.py is docker-only.

Umbrella issue: when the gateway is (re)built, **reconcile** every already-running bottle against the fresh gateway, rather than persisting gateway state on volumes. Folds in #510 (CA) and #512 (git-gate), which are two symptoms of one root cause. Supersedes the abandoned persistent-volume PRs (#511 CA-volume, #513 git-gate-volume). ## The root cause Gateway-dependent state is provisioned into the gateway **once, at bottle launch**, and lives only in the gateway's ephemeral rootfs. Rebuilding or restarting the gateway wipes it, and nothing re-provisions the bottles that were already running. `orchestrator/reprovision.py` restores only egress tokens (`reprovision_from_secret`) — not the CA, not git-gate declarations — and the registry does not store the git upstreams or keys. ## Symptom 1 — CA (was #510) Rebuilding the Firecracker gateway mints a brand-new mitmproxy CA, breaking every already-running bottle with `SSL certificate verification failed` on egress. The #450 fix ("the gateway CA must survive recreation") was implemented only for the **Docker** backend, via the persistent `host_gateway_ca_dir()` bind-mount. The Firecracker gateway had no equivalent: `backend/firecracker/gateway.py` booted the VM through `infra_vm.boot_vm(...)` with **no `data_drive`** — contrast the orchestrator VM, which gets `data_drive=self._ensure_registry_volume()` (persistent `/dev/vdb`). The CA lives at `/home/mitmproxy/.mitmproxy/` inside the rootfs, which `boot_vm` re-materializes on every boot. Each bottle only captures a point-in-time CA snapshot at launch (`launch.py` → `egress/gateway-ca/gateway-ca.pem`), never re-synced when the gateway rotates. Observed: | CA | SHA-256 fingerprint | Captured | |---|---|---| | Live gateway (rebuilt) | `66:63:6F:06:94:…` | after rebuild | | claude-agent-1 (relaunched with gateway) | `66:63:6F:06:94:…` ✅ | after rebuild | | claude-agent-3 (running since prior day) | `C6:B1:80:C0:22:…` ❌ | day before | agent-3 → rebuilt gateway bumps TLS with a leaf signed by `66:63:6F…` → agent-3's trust store only has `C6:B1:80…` → handshake fails. ## Symptom 2 — git-gate (was #512) Restarting the gateway drops every already-running bottle's git-gate state, so its agent gets `HTTP 404` from the git-gate smart-HTTP server (port 9420) on `info/refs` — fetch and push both fail. Per-bottle repos and creds are provisioned at launch (`consolidated_launch` step 3 → `provision_gateway` / `git_gate/render`) under `/git/<bottle_id>` and `/git-gate/creds/<bottle_id>`. On firecracker these live in the per-boot rootfs; on docker only the CA is bind-mounted, not `/git` or `/git-gate/creds`. Either way a gateway restart wipes them. Observed (firecracker): after a gateway rebuild the live gateway VM held exactly ONE bottle's repo — `/git/<id>/bot-bottle.git`, created at gateway boot — the bottle relaunched together with the gateway. Bottles predating the restart had no `/git/<id>` and no `/git-gate/creds/<id>`, so their agents 404'd. `mount` showed only `/dev/vda` (rootfs); nothing persistent. ## Approach — reconcile, don't persist One flow across all services, run when the gateway comes up: - **CA** — replace each running agent's trusted CA with the freshly-minted gateway CA. Agents never need to match a prior CA; there is one gateway. This lets the CA rotate for free on every bring-up. - **git-gate** — re-provision each bottle's bare repos + creds onto the gateway. - **egress tokens** — restore each bottle's egress tokens. The trigger lives in the gateway startup flow (`FirecrackerGateway.connect_to_orchestrator`, the cold-boot path), so it fires exactly when the gateway was (re)booted and never on an adopt. Reconciling also avoids the persistent-volume approach's one-time break: seeding a fresh CA volume still mints one final CA on the first restart after the fix, breaking running bottles once. Teardown already `rm -rf`s `/git/<id>` + creds (`deprovision_git_gate`), so there is no accumulating state to manage either. ## Status / building blocks - [x] PRD drafted - [x] `OrchestratorCore.update_agent_secret` + `POST /bottles/<id>/secret` (single-secret in-place update — the reusable primitive from the 2026-07-26 hotfix) - [x] Bring-up reconcile: CA push to running agents - [x] Bring-up reconcile: git-gate re-provision - [x] Wire the reconcile into `connect_to_orchestrator` ## Out of scope - Refreshing short-lived **host credentials** (e.g. the Codex access token) as part of the bring-up reconcile — deliberately not a criterion here. - Docker `bot-bottle-orch-gateway` is attached to a stale CI CA volume (`bot-bottle-ci-ca-<run>`) because CI reuses the production gateway singleton name — separate latent issue. - `orchestrator/rotate_ca.py` is docker-only.
gitea-actions bot added the Status/Needs Triage label 2026-07-26 18:48:26 -04:00
didericis-claude changed title from Reprovision gateway-dependent state on gateway bring-up (PRD 0081) to Reprovision gateway-dependent state on gateway bring-up 2026-07-27 11:26:21 -04:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label Status/Needs Triage
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#516
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?