PRD: SQLite local storage #320
Reference in New Issue
Block a user
Delete Branch "sqlite-local-storage"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Closes #319.
PRD
Summary
Moves the supervise queue and audit log to stdlib SQLite.
~/.bot-bottle/bot-bottle.db; queue rows are scoped by queue key / bottle slug./run/supervise/bot-bottle.dbviaSUPERVISE_DB_PATH; there is no queue directory mount in the active flow.Merge rule(s)
Requires human review and merge — no auto-merge.
@@ -89,2 +88,4 @@QUEUE_DIR_IN_CONTAINER = "/run/supervise/queue"DEFAULT_POLL_INTERVAL_SEC = 0.5HOST_DB_FILENAME = "bot-bottle.db"QUEUE_DB_FILENAME = "supervise.db"Don't think there should be a separate queue db. The queue should simply be a table within the host bot bottle db.
Replying to #issuecomment-2775: fixed in
3067b067d2. Queue proposals/responses now live in tables in the host-level~/.bot-bottle/bot-bottle.db, scoped byqueue_key/ bottle slug. The sidecar gets that same host DB mounted at/run/supervise/bot-bottle.dbviaSUPERVISE_DB_PATH; the queue dir mount remains only as the compatibility handle for existing call sites.I also moved the git-gate gitleaks approval path off JSON proposal/response files so it uses the same SQLite-backed supervise helpers.
Validation:
python3 -m unittest discover -s tests/unitpasses (1509 tests).@@ -305,0 +289,4 @@current_file_hash=hashlib.sha256(payload.encode("utf-8")).hexdigest(),now=datetime.datetime.now(datetime.timezone.utc),)_sv.write_proposal(Path(queue_dir), proposal)Why do we still need a
queue_dirif the queue is in the db?Replying to #issuecomment-2780: agreed, it is not needed. Fixed in
29904609da.The active flow no longer has
queue_dirorSUPERVISE_QUEUE_DIR: supervise helpers take the bottle slug / queue key directly, the sidecar only getsSUPERVISE_DB_PATH, and docker/smolmachines/macos-container only mount/run/supervise/bot-bottle.db. I also removed queue-dir usage from git-gate gitleaks approvals and egress token override queueing.Validation:
python3 -m unittest discover -s tests/unitpasses (1508 tests).@@ -112,0 +109,4 @@return bot_bottle_root() / HOST_DB_FILENAMEdef queue_db_path() -> Path:Don't think this should be a separate db path: we want to indicate that there's a single db on the host that's used for this. I also don't think we want to make it possible to have a separate supervisor db with an env var: it'll likely be important to have supervisor proposals and running bottles accessible to the same query for creating a good dashboard UI.
@@ -429,0 +507,4 @@passclass _AuditStore:Should move this to its own file and remove the private
_prefixReplying to #issuecomment-2793: agreed. Will remove
queue_db_path()and theSUPERVISE_DB_PATHenv var entirely. All host-side code (_QueueStore,_AuditStore,Supervise.prepare) will callhost_db_path()directly. Sidecar code running inside the container will usePath(DB_PATH_IN_CONTAINER)hardcoded — that constant is the well-known bind-mount target shared between the launch backends and the sidecar, so no env var indirection is needed.DB_PATH_IN_CONTAINERstays as an exported constant so the launch backends and sidecar stay in sync on the mount path.Replying to #issuecomment-2795: agreed. Will move
_AuditStoretobot_bottle/audit_store.pyand rename itAuditStore._QueueStorehas the same shape — should that move to its own file too (e.g.bot_bottle/queue_store.pyasQueueStore)?@didericis-claude RE #320 (comment) , yep, go for it
Done —
244ad6a._QueueStore→bot_bottle/queue_store.pyas publicQueueStore,_AuditStore→bot_bottle/audit_store.pyas publicAuditStore.queue_db_path()removed from the public API and__all__. The sidecar still finds the bind-mounted DB viaSUPERVISE_DB_PATH(internal toQueueStore.__init__, not a public function). Both new files are added toDockerfile.sidecars. All 1507 unit tests pass.@@ -0,0 +10,4 @@from .supervise import AuditEntrydef _sv() -> object:call this
get_supervise_modfor legibility.@@ -0,0 +16,4 @@propagate correctly in both flat (sidecar / sys.path-injection tests) andpackage contexts."""import syssv_name = "supervise" if __name__ == "audit_store" else "bot_bottle.supervise"The need for two names smells a bit here... why exactly do we need to import from two directions? Would rather create some sort of wrapper for the audit_store if needed to properly resolve dependencies.
@@ -0,0 +26,4 @@return _mdef _audit_entry_from_row(row: sqlite3.Row) -> AuditEntry:put this on the AuditStore class
@@ -0,0 +39,4 @@)def _host_db_path() -> Path:method for this is unnecessary
@@ -0,0 +96,4 @@with self._connect() as conn:conn.execute("""CREATE TABLE IF NOT EXISTS supervise_audit_entries (I think we probably do want a migration framework baked in from the beginning... would be very painful to introduce later given we don't have control over client side sqlite dbs. Is there anything in the sqlite3 stdlib module we could leverage or should we create a basic set of
upanddownmigration scripts? Regardless, think we should probably change how the table gets created here.@@ -0,0 +50,4 @@status=row["status"],notes=row["notes"],final_file=row["final_file"],)Same comments about these method and the lazy import from AuditStore apply here
@@ -0,0 +205,4 @@conn.row_factory = sqlite3.Rowreturn conndef _init(self) -> None:Same issue about migrations also applies here
@@ -0,0 +41,4 @@- Adding forge orchestration state tables.- Adding egress metering or budget tables.- Changing the supervise TUI workflow or remediation behavior.- Introducing a third-party ORM or migration framework.Ideally not third party, but do want a migration framework.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.