git-gate per-bottle state does not survive a gateway restart — running bottles 404 on fetch/push #512

New Issue

2026-07-26T17:24:49-04:00

didericis-claude commented

2026-07-26 17:24:49 -04:00

Summary

Restarting/rebuilding the gateway drops every already-running bottle's git-gate state, so its agent gets HTTP 404 from the git-gate smart-HTTP server (port 9420) on info/refs — fetch and push both fail.

Same class of bug as #510 (the CA): per-bottle git-gate state lives only in the gateway's ephemeral rootfs and is provisioned once, at bottle launch.

Evidence (observed, firecracker)

After a gateway rebuild, the live gateway VM held exactly ONE bottle's repo — /git/<id>/bot-bottle.git, created at gateway boot — the bottle that was relaunched together with the gateway. Bottles that predated the restart had no /git/<id> and no /git-gate/creds/<id>, so their agents 404'd. mount showed only /dev/vda (rootfs); nothing persistent.

Root cause

Per-bottle repos + creds are provisioned into the gateway at launch (consolidated_launch step 3 → provision_gateway / git_gate/render), under /git/<bottle_id> and /git-gate/creds/<bottle_id>.
On firecracker these live in the per-boot rootfs (no persistent volume); on docker only the CA is bind-mounted, not /git or /git-gate/creds. Either way a gateway restart wipes them.
Nothing re-provisions already-running bottles: orchestrator/reprovision.py restores only egress tokens (reprovision_from_secret), not git-gate declarations, and the registry does not store the git upstreams/keys.

Fix

Persist git-gate state (/git + /git-gate/creds) across gateway restarts, mirroring the #510 CA-persistence approach, on BOTH backends:

firecracker: a persistent data drive mounted (via bind mounts) at /git and /git-gate/creds.
docker: host bind-mount dirs under the app-data root (never pruned — same rationale as host_gateway_ca_dir).

Teardown already rm -rfs /git/<id> + creds (deprovision_git_gate), so the persistent store self-cleans over the normal lifecycle.

Stacked on #510/#511 (shares the gateway persistent-volume mechanism).

## Summary Restarting/rebuilding the gateway drops every already-running bottle's git-gate state, so its agent gets `HTTP 404` from the git-gate smart-HTTP server (port 9420) on `info/refs` — fetch and push both fail. Same class of bug as #510 (the CA): per-bottle git-gate state lives only in the gateway's ephemeral rootfs and is provisioned once, at bottle launch. ## Evidence (observed, firecracker) After a gateway rebuild, the live gateway VM held exactly ONE bottle's repo — `/git/<id>/bot-bottle.git`, created at gateway boot — the bottle that was relaunched together with the gateway. Bottles that predated the restart had no `/git/<id>` and no `/git-gate/creds/<id>`, so their agents 404'd. `mount` showed only `/dev/vda` (rootfs); nothing persistent. ## Root cause - Per-bottle repos + creds are provisioned into the gateway at launch (`consolidated_launch` step 3 → `provision_gateway` / `git_gate/render`), under `/git/<bottle_id>` and `/git-gate/creds/<bottle_id>`. - On firecracker these live in the per-boot rootfs (no persistent volume); on docker only the CA is bind-mounted, not `/git` or `/git-gate/creds`. Either way a gateway restart wipes them. - Nothing re-provisions already-running bottles: `orchestrator/reprovision.py` restores only egress tokens (`reprovision_from_secret`), not git-gate declarations, and the registry does not store the git upstreams/keys. ## Fix Persist git-gate state (`/git` + `/git-gate/creds`) across gateway restarts, mirroring the #510 CA-persistence approach, on BOTH backends: - firecracker: a persistent data drive mounted (via bind mounts) at `/git` and `/git-gate/creds`. - docker: host bind-mount dirs under the app-data root (never pruned — same rationale as `host_gateway_ca_dir`). Teardown already `rm -rf`s `/git/<id>` + creds (`deprovision_git_gate`), so the persistent store self-cleans over the normal lifecycle. Stacked on #510/#511 (shares the gateway persistent-volume mechanism).

gitea-actions bot added the Status/Needs Triage label 2026-07-26 17:24:55 -04:00

didericis referenced this issue from a commit

2026-07-26 17:34:26 -04:00

fix(gateway): persist git-gate state across gateway restarts

didericis-claude referenced a pull request that will close this issue

2026-07-26 17:34:47 -04:00

fix(gateway): persist git-gate state across gateway restarts #513

didericis-claude referenced this issue

2026-07-26 18:48:18 -04:00

Reprovision gateway-dependent state on gateway bring-up (PRD 0081) #516

didericis referenced this issue from a commit

2026-07-26 18:48:32 -04:00