didericis/bot-bottle
1
0
Fork 0
Code Issues 25 Pull Requests 9 Actions Packages Projects Releases Wiki Activity

docs(prd): 0070 per-host orchestrator service #352

Closed
didericis-claude wants to merge 12 commits from prd-orchestrator into main
pull from: prd-orchestrator
merge into: didericis:main
Conversation 25 Commits 12 Files Changed 33
+27
1 changed files with 27 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Showing only changes of commit ed7307e0e3 - Show all commits
docs/prds/0070-per-host-orchestrator.md
+27
View File
1
 @@ -279,6 +279,33 @@ SQLite is right for the runtime tier (mutable, concurrent, queried) and
wrong for the other two (Nix can't read it at eval time; it fights the
declarative manifest trust model). Keep the tiers separate.
**The DB file is host-resident, not owned inside the orchestrator unit**
(decided in review). Two reasons:
- **Durability across restarts.** Re-adoption sweeps the registry after an
orchestrator restart, so the state *must* outlive the orchestrator
instance's lifecycle. A host-resident file (the slice-1 default under
`~/.cache/bot-bottle/orchestrator/`) is the simplest durable store; the
orchestrator VM/container mounts it rather than carrying it.
- **Integrity via access-scoping, not location.** Agents can't touch the
DB directly regardless of where it lives — they're network-isolated in
their bottles and only ever speak to the control/data plane. The real
risk is a *compromised agent-facing data-plane service* (egress/git-gate,
which parse hostile bytes) writing the registry and forging attribution.
The control is a **read/write split**: writes (register/deregister) come
only from the **control plane**; the **data plane gets the DB read-only**
for attribution lookups. A host-resident file makes that split
enforceable (mount `ro` into the data plane, `rw` into the control
plane) — which owning it "entirely inside" a monolithic orchestrator
would not.
Implementation note for the VM slices: SQLite **WAL** over a guest share
(virtiofs/9p) is finicky (the `-shm`/`-wal` files need real mmap/locking),
so the durable-DB-on-host may want a small **host-side owner** the
didericis marked this conversation as resolved Outdated
didericis commented 2026-07-13 12:19:18 -04:00
Outdated
Review
Copy Link
Copy Source

It’s worth it, should be a minimal surface for a malicious agent to exploit/we should be able to harden it and keep it secure.

Also spec-ed out more mitigations via short lived vault tokens (link the relevant issue in reply to this comment)

It’s worth it, should be a minimal surface for a malicious agent to exploit/we should be able to harden it and keep it secure. Also spec-ed out more mitigations via short lived vault tokens (link the relevant issue in reply to this comment)
orchestrator reaches over the control-plane RPC, rather than a shared
mount. Decide this at the Firecracker slice; `sqlite3` itself is stdlib, so
didericis marked this conversation as resolved Outdated
didericis commented 2026-07-13 12:20:35 -04:00
Outdated
Review
Copy Link
Copy Source

Whatever is most universal/reliable on every host machine (probably http?)

Whatever is most universal/reliable on every host machine (probably http?)
"the host needs SQLite" is a non-cost.
## Sequencing
didericis marked this conversation as resolved Outdated
didericis commented 2026-07-13 12:26:37 -04:00
Outdated
Review
Copy Link
Copy Source

Can you elaborate racing in flight launches? You mean possible parallel orchestrator relaunches on the same host, or an or orchestrator possibly restarting an agent before readopting it?

Regardless (unless I’m missing something), I think the procedure should be fairly straightforward:

  1. every orchestrator launch requires there to be no preexisting, healthy orchestrator launch. If a healthy or non healthy orchestrator is found, it must be fully removed/shut down first
  2. readoption requires waiting for the new orchestrator to be healthy
  3. once the new orchestrator is healthy, the orchestrator must search to find all agents needing adoption (via both sqlite state and process inspection) before handling any other requests
Can you elaborate racing in flight launches? You mean possible parallel orchestrator relaunches on the same host, or an or orchestrator possibly restarting an agent before readopting it? Regardless (unless I’m missing something), I think the procedure should be fairly straightforward: 1) every orchestrator launch requires there to be no preexisting, healthy orchestrator launch. If a healthy or non healthy orchestrator is found, it must be fully removed/shut down first 2) readoption requires waiting for the new orchestrator to be healthy 3) once the new orchestrator is healthy, the orchestrator must search to find all agents needing adoption (via both sqlite state and process inspection) before handling any other requests
Jump straight to the **virtualized** end state (not a host-daemon stepping
didericis marked this conversation as resolved Outdated
didericis commented 2026-07-13 12:28:01 -04:00
Outdated
Review
Copy Link
Copy Source

Guessing this will have to vary per backend, but think we’ve solved this already? Had spikes related to putting sidecars in vms where I think we figured out how to do this

Guessing this will have to vary per backend, but think we’ve solved this already? Had spikes related to putting sidecars in vms where I think we figured out how to do this
1