didericis/bot-bottle
1
0
Fork 0
Code Issues 34 Pull Requests 16 Actions Packages Releases Wiki Activity

fix(gateway): persist git-gate state across gateway restarts #513

Closed
didericis-claude wants to merge 1 commits from fix/gitgate-persist-512 into fix/firecracker-gateway-ca-persist-510
pull from: fix/gitgate-persist-512
merge into: didericis:fix/firecracker-gateway-ca-persist-510
Conversation 1 Commits 1 Files Changed 13
+213 -27
didericis-claude commented 2026-07-26 17:34:46 -04:00
Collaborator
Copy Link
Copy Source

Problem

Restarting/rebuilding the gateway drops every already-running bottle's git-gate state, so its agent gets HTTP 404 from the git-gate smart-HTTP server on info/refs — fetch and push both fail. Observed live: after a rebuild the gateway held only the ONE bottle's /git/<id>/bot-bottle.git that was relaunched with it; bottles predating the restart had no repo/creds.

Same class of bug as #510/#511 (the CA): per-bottle repos (/git/<id>) + deploy creds (/git-gate/creds/<id>) are provisioned once at bottle launch and live in the gateway's ephemeral storage. Nothing re-provisions running bottles — the orchestrator restores only egress tokens, not git-gate declarations.

Fix (persist state, both backends)

firecracker — attach a second persistent data drive (/dev/vdc) to the gateway VM and bind-mount its git/ + creds/ subdirs onto /git and /git-gate/creds in the gateway guest init, before the data plane (and any provisioning writes). Generalized the VM config from a single data_drive to a stable-ordered data_drives tuple (CA=vdb, git=vdc; the orchestrator's registry stays vdb). New _ensure_git_volume() (sparse 8G ext4) mirrors _ensure_ca_volume.

docker — bind-mount host dirs (host_gateway_git_dir / host_gateway_creds_dir, under the never-pruned app-data root — same #450 rationale as the CA) onto /git and /git-gate/creds, with BOT_BOTTLE_DOCKER_GIT_MOUNT / _CREDS_MOUNT env overrides so CI isolates them to per-run volumes (workflows updated to set + clean them up).

Teardown already rm -rfs /git/<id> + creds (deprovision_git_gate), so the persistent store self-cleans over the normal lifecycle.

Tests

  • firecracker: multi-drive config ordering; _ensure_git_volume create/reuse; gateway boots with data_drives=(ca, git); gateway guest init mounts /dev/vdc + bind-mounts /git and /git-gate/creds before the data plane.
  • docker: gateway run argv includes :/git and :/git-gate/creds volumes.
  • Updated orchestrator/backend tests for the data_drives rename. Full unit suite green (only the pre-existing test_wheel_install env failure — missing build module — is unrelated).

Stacking

Stacked on #511 (shares the gateway persistent-volume mechanism); base is fix/firecracker-gateway-ca-persist-510. Retarget to main once #511 merges.

Closes #512

🤖 Generated with Claude Code

## Problem Restarting/rebuilding the gateway drops every already-running bottle's git-gate state, so its agent gets `HTTP 404` from the git-gate smart-HTTP server on `info/refs` — fetch and push both fail. Observed live: after a rebuild the gateway held only the ONE bottle's `/git/<id>/bot-bottle.git` that was relaunched with it; bottles predating the restart had no repo/creds. Same class of bug as #510/#511 (the CA): per-bottle repos (`/git/<id>`) + deploy creds (`/git-gate/creds/<id>`) are provisioned once at bottle launch and live in the gateway's ephemeral storage. Nothing re-provisions running bottles — the orchestrator restores only egress tokens, not git-gate declarations. ## Fix (persist state, both backends) **firecracker** — attach a second persistent data drive (`/dev/vdc`) to the gateway VM and bind-mount its `git/` + `creds/` subdirs onto `/git` and `/git-gate/creds` in the gateway guest init, before the data plane (and any provisioning writes). Generalized the VM config from a single `data_drive` to a stable-ordered `data_drives` tuple (CA=vdb, git=vdc; the orchestrator's registry stays vdb). New `_ensure_git_volume()` (sparse 8G ext4) mirrors `_ensure_ca_volume`. **docker** — bind-mount host dirs (`host_gateway_git_dir` / `host_gateway_creds_dir`, under the never-pruned app-data root — same #450 rationale as the CA) onto `/git` and `/git-gate/creds`, with `BOT_BOTTLE_DOCKER_GIT_MOUNT` / `_CREDS_MOUNT` env overrides so CI isolates them to per-run volumes (workflows updated to set + clean them up). Teardown already `rm -rf`s `/git/<id>` + creds (`deprovision_git_gate`), so the persistent store self-cleans over the normal lifecycle. ## Tests - firecracker: multi-drive config ordering; `_ensure_git_volume` create/reuse; gateway boots with `data_drives=(ca, git)`; gateway guest init mounts `/dev/vdc` + bind-mounts `/git` and `/git-gate/creds` before the data plane. - docker: gateway run argv includes `:/git` and `:/git-gate/creds` volumes. - Updated orchestrator/backend tests for the `data_drives` rename. Full unit suite green (only the pre-existing `test_wheel_install` env failure — missing `build` module — is unrelated). ## Stacking Stacked on #511 (shares the gateway persistent-volume mechanism); base is `fix/firecracker-gateway-ca-persist-510`. Retarget to `main` once #511 merges. Closes #512 🤖 Generated with [Claude Code](https://claude.com/claude-code)
didericis-claude added 1 commit 2026-07-26 17:34:49 -04:00
fix(gateway): persist git-gate state across gateway restarts
tracker-policy-pr / check-pr (pull_request) Successful in 12s
Details
lint / lint (push) Successful in 59s
Details
test / unit (pull_request) Successful in 53s
Details
test / integration-docker (pull_request) Successful in 1m7s
Details
test / coverage (pull_request) Successful in 19s
Details
dff811f190 
Per-bottle git-gate state (bare repos under /git/<id>, deploy creds under
/git-gate/creds/<id>) was provisioned once at bottle launch and lived only
in the gateway's ephemeral storage. A gateway rebuild/restart wiped it and
nothing re-provisioned already-running bottles, so their agents 404'd on
fetch/push. Same class of bug as the CA (#510); the orchestrator restores
only egress tokens, not git-gate declarations.

Persist the state on both backends, mirroring the CA-persistence approach:

- firecracker: attach a second persistent data drive (/dev/vdc) to the
  gateway VM and bind-mount its git/ + creds/ subdirs onto /git and
  /git-gate/creds in the gateway guest init, before the data plane starts.
  Generalize the VM config to a stable-ordered data_drives tuple (CA=vdb,
  git=vdc; orchestrator registry stays vdb).
- docker: bind-mount host dirs (host_gateway_git_dir / creds_dir, under the
  never-pruned app-data root) onto /git and /git-gate/creds, with
  BOT_BOTTLE_DOCKER_GIT_MOUNT / _CREDS_MOUNT env overrides so CI isolates
  them to per-run volumes it cleans up.

Teardown already rm -rf's /git/<id> + creds, so the persistent store
self-cleans over the normal lifecycle.

Closes #512

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-claude commented 2026-07-26 17:56:10 -04:00
Author
Collaborator
Copy Link
Copy Source

Superseded by the reprovision-on-gateway-bring-up strategy (PRD 0081, forthcoming PR). Rather than persist gateway-side state on a volume, the gateway reconciles all running bottles when it comes up — replacing each running agent's CA with the current gateway CA and re-provisioning each bottle's git-gate — giving one unified reprovision flow across CA / git-gate / egress tokens and free CA rotation on each restart. Closing in favor of that single PR.

Superseded by the reprovision-on-gateway-bring-up strategy (PRD 0081, forthcoming PR). Rather than persist gateway-side state on a volume, the gateway reconciles all running bottles when it comes up — replacing each running agent's CA with the current gateway CA and re-provisioning each bottle's git-gate — giving one unified reprovision flow across CA / git-gate / egress tokens and free CA rotation on each restart. Closing in favor of that single PR.
didericis-claude closed this pull request 2026-07-26 17:56:13 -04:00
Some checks are pending
tracker-policy-pr / check-pr (pull_request) Successful in 12s
Details
lint / lint (push) Successful in 59s
Details
test / unit (pull_request) Successful in 53s
Details
test / integration-docker (pull_request) Successful in 1m7s
Details
test / coverage (pull_request) Successful in 19s
Details

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#513
Delete Branch "fix/gitgate-persist-512"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?