docs(prd): 0070 per-host orchestrator service #352
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: didericis/bot-bottle#352
Reference in New Issue
Block a user
Delete Branch "prd-orchestrator"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Per-host orchestrator PRD (#351): fold the per-bottle sidecar bundle into one persistent per-host service that runs the sidecar functions, coordinates with the console, and brokers agent launches. Virtualized from the start, backend-native (fc VM / apple ctr / docker ctr), behind a single backend-agnostic contract; per-backend variation on
BottleBackend(singleOrchestrator, no parallel hierarchy).Leads with the security review — secret concentration, shared fate, the launch-broker-as-new-privileged-core, and the source-IP attribution invariant each backend must enforce. Proposes one SQLite DB (orchestrator-owned) for runtime state, kept distinct from build-time constants and declarative user config.
Supersedes 0069's Stage-1/4 sidecar framing (banner added to 0069); depends on 0069's nix-built fixed images.
🤖 Generated with Claude Code
adc03e0e26to095896817c@@ -0,0 +301,4 @@- **Egress sharing tradeoff:** is the secret-concentration blast radius ofone shared mitmproxy worth the resource win, or share only supervise(near-zero secrets) and keep egress + git-gate per-bottle initially?It’s worth it, should be a minimal surface for a malicious agent to exploit/we should be able to harden it and keep it secure.
Also spec-ed out more mitigations via short lived vault tokens (link the relevant issue in reply to this comment)
@@ -0,0 +303,4 @@one shared mitmproxy worth the resource win, or share only supervise(near-zero secrets) and keep egress + git-gate per-bottle initially?- **Control-plane shape:** RPC transport (unix socket / vsock / HTTP overthe TAP) and the live-reload protocol for per-bottle policy.Whatever is most universal/reliable on every host machine (probably http?)
@@ -0,0 +306,4 @@the TAP) and the live-reload protocol for per-bottle policy.- **State re-adoption:** exact scheme for an orchestrator restart tore-adopt running agent VMs from the SQLite registry without racingin-flight launches.Can you elaborate racing in flight launches? You mean possible parallel orchestrator relaunches on the same host, or an or orchestrator possibly restarting an agent before readopting it?
Regardless (unless I’m missing something), I think the procedure should be fairly straightforward:
@@ -0,0 +308,4 @@re-adopt running agent VMs from the SQLite registry without racingin-flight launches.- **VM-to-VM routing:** the nft forward rules + addressing for aper-host orchestrator VM on its own TAP.Guessing this will have to vary per backend, but think we’ve solved this already? Had spikes related to putting sidecars in vms where I think we figured out how to do this
@@ -0,0 +310,4 @@- **VM-to-VM routing:** the nft forward rules + addressing for aper-host orchestrator VM on its own TAP.- **Broker request schema:** the exact structured contract that staysauditable and can't be coerced into launching arbitrary payloads.human readable json with static flags and ids makes sense for all communication with the side, I think. Might be missing cases where that doesn’t work, but we’ll tackle them as we go/make that the ideal. Would also be worth making it a signed jwt for provenance
@@ -0,0 +105,4 @@responsibility** and part of the contract:> **Invariant:** a packet's source address, as seen by the orchestrator,> *provably* identifies the originating bottle.Given all the different ways to do ip spoofing, would it also be worth adding an outbound proxy to the agent that injects a token to every outbound request to the sidecar so an agent can prove who it is? I think we’ve already added that proxy/should be trivial to add a token, but let me know if that didn’t land
Responses to the inline comments — all folded into the PRD in commit
73a7582(new Decisions (review 2026-07-13) section + inline edits). Per-line threading isn't reliable through the API here, so consolidating:1. Egress sharing tradeoff — Agreed, consolidating egress; recorded the decision as "worth it, minimal hardened surface" and paired it with the identity token + the short-lived-vault-token mitigations, which are tracked in #355 (generic
SecretProvider) and written up in this PRD's Secret handling section.2. Control-plane transport — Agreed: HTTP as the wire contract (universal/reliable everywhere; unix socket left as an optional local optimization). Folded into the contract section.
3. State re-adoption / "racing in-flight launches" — I meant the second thing you named: not parallel orchestrators (your step 1 rules that out), but a launch that had started (VM booting / slot claimed) yet hadn't committed to SQLite when the old orchestrator died — a SQLite-only sweep would miss it. Your 3-step procedure is exactly right and I've adopted it verbatim; step 3's "SQLite and process inspection" is precisely what catches that not-yet-committed launch. Added one thing: launches write an intent record before committing resources so the sweep can reconcile intent vs. actual.
4. VM-to-VM routing — Right, it's per-backend — in the design it's literally
BottleBackend.wire()(DNAT+forward for fc, shared-net for docker/apple), not orchestrator logic. If the earlier "sidecars in VMs" spike already solved the fc forward rules, this drops from open-question to "implement per the spike." Can you point me at that spike (branch/notes)? I'll reference it and close the question.5. Broker request schema — Agreed and adopted: human-readable JSON of static flags + ids only (no free-form paths/argv — that's what keeps it un-coercible into arbitrary launches), wrapped as a signed JWT so the broker verifies provenance (request came from the real orchestrator, not a forged one from a compromised co-located component). Same shape for all orchestrator↔broker/sidecar comms.
6. Identity token vs IP spoofing — Good defense-in-depth; added it as an app-layer layer on the attribution invariant. Honest nuance captured in the PRD: on Firecracker the
/31+ nft make source IP unspoofable by construction, so there it's belt-and-suspenders (cheap insurance vs. a misconfigured invariant); on weaker backends (Docker) it's genuinely load-bearing. Requirements noted: per-bottle, unguessable, non-cross-leakable (a bottle can only ever prove it's itself), provisioned at launch, required by the sidecar. One caveat it does not fix: it's app-layer, so a compromised orchestrator still sees every token — that's the concentration problem, handled under Secret handling, not here. And yes — the agent already egresses through the sidecar proxy, so adding the header is trivial, as you said.@didericis-claude RE 4, don’t know where/when we did it, can search when implementing. But should be possible/not a blocker, even if it requires modifying the host a bit (like it does on nixos).
Can start implementing now. Can do it in this branch/PR.
RE 4 — sounds good. Marked it not a blocker in the PRD (commit
8d54fc3): resolvable at implementation time, host tweak acceptable (same as the pool setup already needs on NixOS), spike showed feasibility. No need to hunt it down now.On starting implementation — agreed it's unblocked. One suggestion on where: land this PRD PR as-is (docs-only) so the design is a stable reference, then implement in a dedicated branch — keeps "approve the design" separate from "review the code" and avoids one giant mixed diff. But I'm happy to build right here in this branch if you'd rather iterate design + code together; your call.
Per the PRD sequencing, the natural first slice is:
BottleBackendseam (the sidecar bundle is already containers; collapse N into one persistent container).Firecracker (the shim +
wire()) comes after that's proven. Want me to start on slice 1? If so, just confirm: land the PRD first + new branch, or build in this branch.@didericis-claude do in this branch incase the prd changes as we implement it
Sounds good — implementing in this branch so the PRD and code evolve together.
Starting with slice 1: the orchestrator service as a plain-process dev-harness — the consolidation core that's independent of any VM packaging:
bot_bottle/orchestrator/registry.py— SQLite (WAL) bottle registry on the existingDbStore/TableMigrationsbase; the runtime-state tier from the PRD (source IP + per-bottle identity token), with fail-closed attribution (source IP and identity token must both match one active record).bot_bottle/orchestrator/control_plane.py— the HTTP control plane (register/deregister/list/attribute/health); routing is a puredispatch()so it's socket-free testable, with a thin stdlib handler/server adapter. register/deregister are the live-reload path.bot_bottle/orchestrator/__main__.py—python -m bot_bottle.orchestratorrunnable harness.Launch/teardown + the broker + the actual egress/git/supervise data plane come in later slices once this core is proven. Pushing shortly.
First implementation slice of PRD 0070, the backend-neutral consolidation core as a plain-process dev-harness (no VM packaging yet): * orchestrator/registry.py — SQLite (WAL) runtime-state store on the existing DbStore/TableMigrations base. Live bottle registry keyed by source IP + per-bottle identity token, with fail-closed attribution: a request resolves to a bottle only when its source IP AND identity token both match exactly one active record (unknown/ambiguous IP, empty token, or token mismatch all deny). Tokens are 256-bit urandom. * orchestrator/control_plane.py — the HTTP control plane (the universal transport chosen in 0070): register / deregister / list / attribute / health. Routing is a pure dispatch() so it is socket-free testable; Handler/ControlPlaneServer/make_server are a thin stdlib adapter. register/deregister are the live-reload path; listing redacts tokens. * orchestrator/__main__.py — `python -m bot_bottle.orchestrator` harness. Launch/teardown, the launch broker, and the egress/git/supervise data plane come in later slices. 24 unit tests (attribution matrix, persistence, dispatch, one real-socket round-trip). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCckSlice 1 pushed in
1702664— CI green (lint incl. pyright-strict, unit, integration, coverage).Landed:
bot_bottle/orchestrator/{registry,control_plane,__main__}.py— the SQLite (WAL) registry + fail-closed attribution (source IP and identity token), the HTTP control plane (register/deregister/list/attribute/health, puredispatch()+ thin stdlib adapter), and thepython -m bot_bottle.orchestratordev-harness. 24 unit tests (attribution matrix, persistence, dispatch, one real-socket round-trip).Next slices: launch/teardown + the signed-JWT broker, then wiring it as the Docker orchestrator (cheapest backend to prove the
BottleBackendseam).@@ -0,0 +3,4 @@The backend-agnostic control-plane RPC (CLI / console -> orchestrator) over**HTTP** — the universal transport chosen in 0070 (works on every host; novsock / unix-socket portability caveats). Endpoints mutate the liveregistry, so register/deregister are the *live-reload* path — no restart:Debating whether orchestrator should own db entirely so the host doesn’t need sqlite… am thinking this is probably better, as that potentially opens up agents to mutate the db, and sqlite is universal enough that keeping it on the host is fine. Keeping it on the host also means it survives orchestrator reboots.
RE the DB-ownership comment on
control_plane.py— agree with keeping it host-resident, and it's already the slice-1 default (default_db_path()→~/.cache/bot-bottle/orchestrator/registry.db, on the host). Folded the decision into the PRD's State section (commited7307e). Two refinements though:Durability is actually a hard requirement, not just a nice-to-have. The restart re-adoption we specced sweeps the registry to re-adopt running bottles, so the state must outlive the orchestrator instance's lifecycle. Host-resident file is the simplest way; the orchestrator VM/container mounts it rather than carrying it.
On "opens agents up to mutating the db" — the real control is access-scoping, not location. Agents can't touch the DB directly wherever it lives — they're network-isolated in their bottles and only ever speak to the control/data plane. The risk is a compromised agent-facing data-plane service (egress/git-gate, which parse hostile bytes) writing the registry and forging attribution. So the fix is a read/write split: writes (register/deregister) only from the control plane; the data plane gets the DB read-only for attribution lookups. A host-resident file is what makes that split enforceable (mount
rointo the data plane,rwinto the control plane) — so location helps because it enables the split + durability, more than because it blocks agents (they were already blocked).One implementation wrinkle for the VM slices: SQLite WAL over a guest share (virtiofs/9p) is finicky — the
-wal/-shmfiles need real mmap/locking. So the durable-DB-on-host may want a small host-side owner the orchestrator reaches over the control-plane RPC, rather than a shared mount into the VM. No need to decide now (slice 1 is a host process, so the file is local); flagged it for the Firecracker slice. And yes —sqlite3is stdlib, so "the host needs sqlite" is a non-cost.@@ -0,0 +45,4 @@def default_db_path() -> Path:"""Host location for the orchestrator's runtime-state DB. Runtime state(not config), so it lives under the cache dir like the pool state."""return Path.home() / ".cache" / "bot-bottle" / "orchestrator" / "registry.db"@didericis-claude Think we should use the same “bot-bottle.db” for all bot-bottle state, including the registry. Makes querying and future console related integration easier. Thoughts?
Agreed — done in
b174442.default_db_path()now returnshost_db_path(), so the registry co-tenants the existing sharedbot-bottle.db. This is architecturally clean: theDbStore/TableMigrationsbase already namespaces each store byschema_key, so slot leases / supervise queue / registry share one file with separate migration tracks — one place to query and point a console at, as you said.Two things I want to flag though, because they change how we integrate the console and enforce security:
1. The console should read state via the orchestrator's RPC, not the DB file directly. One shared DB makes ad-hoc querying easy for us (debugging/analytics), but if the console integrates by opening
bot-bottle.dbit bypasses the policy the orchestrator enforces and can mutate state. Cleaner: the console talks to the control-plane API (which we're building), and the shared DB is the convenience for offline inspection. So "one DB" helps ops/debugging more than it should drive the console integration.2. Unifying actually strengthens the attribution-integrity rule — but only if the orchestrator is the sole DB opener. Last thread we said "data plane gets the registry read-only." With one shared file, coarse
ro/rwmount-splitting can't isolate just the registry, so the rule gets simpler and stronger: only the orchestrator (control plane) opensbot-bottle.db; the data plane reaches attribution + writes proposals via RPC, never a file handle. No agent-facing component touches the file, so none can forge attribution.Concrete caveat I hit while wiring this up:
bot-bottle.dbis today rw-bind-mounted into the per-bottle supervise sidecar (so it can write proposals — seehost_db_path()'s "share this one file with a sidecar" note). That's an agent-facing component with write access to the file the registry now lives in — i.e. exactly the forge-attribution risk. It's fine because it's the pattern the orchestrator removes (supervise consolidates into the orchestrator; sidecar writes become RPC calls), but until that lands we must not expose the unified DB through a data-plane-writable mount. Captured all of this in the PRD State section.Also dropped the WAL pragma I'd added — flipping the shared
bot-bottle.dbto WAL would affect supervise/audit and is finicky over guest shares, so it's a deliberate future change, not something to sneak in here; kept abusy_timeoutfor contention. Tests still green.@@ -0,0 +37,4 @@from ..db_store import DbStorefrom ..migrations import TableMigrationsfrom ..supervise_types import host_db_paththis should be moved to db_store
Done in
b4b4e08.host_db_path+HOST_DB_FILENAMEnow live canonically indb_store(alongsideDbStore), and the registry imports from there.Two things I preserved carefully so nothing broke:
db_store.host_db_pathresolvesbot_bottle_rootfromsupervise_typesvia a lazy import inside the function, sodb_storedoesn't importsupervise_typesat module load (which would cycle, sincesupervise_typesnow importsdb_store).supervise_types.bot_bottle_rootstill propagates to the DB path (verified).supervise_typesre-exports both names for the historical import path, soqueue_store/audit_storeare unchanged.Full unit suite is behavior-preserving (only the pre-existing
/bin/sleepsidecar-init errors remain). Also dropped the now-unusedimport sysfromsupervise_types.One thing I left alone:
supervise.pystill has its ownhost_db_pathduplicate (a comment there literally says "must stay in sync" with the other copy) — it usessupervise.bot_bottle_root's patch namespace, which several tests rely on, so consolidating it too is a separate, riskier change. Happy to tackle that dedup separately if you want it gone.@didericis-claude looks good, let's do slice 2. Ok to do in the same PR.
Great — slice 2 in this PR it is. (Also stripped the vestigial
bot_bottle_root/host_db_pathre-exports first, in6847fdf—pathsis now the sole home; nosupervise.bot_bottle_root/.host_db_pathreferences remain.)Slice 2 — launch lifecycle + the launch-broker contract (still backend-neutral / dev-harness, so it stays fast to test before the Docker wiring):
launch_bottle/teardown_bottle— extend the HTTP RPC beyond register/deregister:launch_bottlemints the bottle (registry: source IP + identity token), issues a broker request, and returns the bottle handle + token;teardown_bottlederegisters + issues the teardown request.LaunchBrokerinterface plus the signed-JWT request envelope we agreed on: static flags + ids only (no free-form paths/argv), signed so the broker verifies provenance (came from the real orchestrator). Build + verify + a schema guard, with a stub/local broker for the harness (records requests; no real launch yet).Slice 3 is then the real Docker orchestrator — a concrete
LaunchBroker/BottleBackend.launch_unitthat actually starts the sidecar-bundle container, proving the seam on the cheapest backend.Starting on slice 2.
@didericis-claude Actually let's branch off this PR for slice 2. Go for it.
Updated slice roadmap (now includes the consolidated sidecar)
Also written into the PRD's Sequencing section (
8e567ed). Built as a stack of small PRs off this one:LaunchBrokerrequest (HS256 JWT, static ids/flags only, provenance-verified, fail-closed); stub broker + registry rollback. (#356)docker run/rmper bottle from static fields; proves the backend seam. (#357)Stack merge order: #352 → #356 → #357 → #358.
Roadmap update — consolidated-sidecar arc expanded (
32e9bb9)The "make the consolidated sidecar real" work is genuinely a multi-slice arc; split it out in the PRD Sequencing:
bot-bottle-sidecarsfromDockerfile.sidecarsand launch that singleton (#360)Then 8. Firecracker and 9. macOS. Slices 1–5 implemented (#352/#356/#357/#358/#360).
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.