didericis/bot-bottle
1
0
Fork 0
Code Issues 33 Pull Requests 3 Actions Packages Releases Wiki Activity

Host-side control server: broker bottle launches and own host-durable state #468

New Issue
Open
opened 2026-07-23 20:40:43 -04:00 by didericis · 3 comments
didericis commented 2026-07-23 20:40:43 -04:00
Owner
Copy Link
Copy Source

Currently container launches are done directly via the CLI. We want to move this so both the CLI and the orchestrator communicate with a host-side control server, which becomes the single privileged component on the host: it brokers launches, owns orchestrator lifecycle, and is the sole writer of host-durable state.

Launch flows

Web console

web console -(iroh)-> orchestrator -(http)-> host_controller -> launch

CLI

cli -(http)-> orchestrator -(http)-> host_controller -> launch

Bootstrap / recovery

The host controller owns orchestrator lifecycle, so orchestrator startup cannot route through the orchestrator. The CLI needs a direct channel for bootstrap and recovery even though all bottle operations funnel through the orchestrator:

cli -(http)-> host_controller   # start / restart / status of the orchestrator itself

This is the path #391 (backend-agnostic orchestrator restart) would target.

The prize is that after this, the CLI no longer needs the Docker socket. That is what lets a dedicated Gitea runner user drop the root-equivalent docker group (called out in PRD 0070 under "Relationship to other work").

Existing scaffold

orchestrator/broker.py has a well-formed contract — LaunchRequest (ids + static flags only, never argv), HS256 sign/verify, fixed-schema validation, fail-closed BrokerAuthError, and a StubBroker that exercises the full sign → verify → act path in-process. DockerBroker exists but is on no production path; every backend starts the orchestrator with --broker stub.

Four gaps between that and a real host service:

  1. No transport. LaunchBroker.submit(token) is an in-process method call from Orchestrator.launch_bottle (service.py L115). Needs a BrokerClient that POSTs the signed token and a host-side HTTP server that verifies and acts.
  2. The signing secret is ephemeral and self-generated. orchestrator/__main__.py L58 does secrets.token_bytes(32) and hands the same value to signer and verifier — only viable because they share a process. A separate daemon needs it provisioned out of band and durable across orchestrator restarts.
  3. Replay protection is claimed but not enforced. sign_request emits jti and iat; verify_request reads neither. No expiry window, no jti cache. Harmless in-process; over a wire a captured launch token replays indefinitely.
  4. The op vocabulary is launch / teardown only. Everything else host-privileged is still in the CLI (see below), so the schema has to grow — carefully, since PRD 0070's security argument rests on "structured requests only, static flags + ids."

What moves off the CLI

Host-privileged operations currently scattered across backend/*/consolidated_launch.py and driven by a short-lived CLI process:

  • OrchestratorService.ensure_running() — starting the infra container
  • the agent docker run / VM boot itself
  • IP allocation (next_free_ip, _network_container_ips)
  • provision_git_gate — docker cp/exec of per-bottle deploy keys into the gateway
  • _reprovision_running_bottles — docker exec printenv ENV_VAR_SECRET against live agents
  • live-bottle enumeration for reconcile
  • agent image builds

Orchestrator.reconcile is the tell: it takes live_source_ips as a parameter only because the orchestrator cannot see the backend. With a host service that enumeration becomes internal and the parameter goes away.

Database / state ownership

We originally wanted a single unified DB for all state mounted on the host. That isn't possible — SQLite locking is not coherent across guest kernels over a share, which is why the macOS backend already uses a container-only volume (INFRA_DB_VOLUME = "bot-bottle-mac-db"). Rather than working around it with a mounted volume for one shared DB, split the state by owner and lifetime. Depends on #469, which gets the DB off the data plane first.

Orchestrator

Owns all operational state on a volume nothing else mounts (generalizing the existing macOS design to every backend). Durable across orchestrator restarts so re-adoption works and running bottles are not bricked by a restart:

  • orchestrator_bottles — registry: bottle_id, source_ip, identity_token, metadata, policy
  • bottled_agent_secrets — encrypted per-bottle egress tokens
  • supervise_proposals / supervise_responses — the approval queue

Stays SQLite: mutable, transactional, queried. The orchestrator is sole mounter as well as sole writer.

Host

Owns the historical record, written only by the host controller over the authenticated channel:

  • supervise audit entries
  • egress traffic log — currently written to sys.stderr (the container log) by egress_addon.py; both halves of "the audit record" should land in one place
  • host-side config (bot_bottle_config, orchestrator_config), or moved out of the DB entirely per PRD 0070's three-tier table

Append-only, never updated, never transactionally queried — so this should be a JSONL log, not SQLite. O_APPEND writes are atomic and there is no locking protocol to get wrong, and hash-chaining for tamper-evidence becomes cheap. It also survives orchestrator destruction and container-runtime volume pruning (the #450 lesson), and stays readable without the orchestrator running.

Note what host-location does not buy: integrity against a live compromised orchestrator, which makes the decisions being audited and can forge or omit entries wherever the file lives. An off-box copy is the answer to that, separately.

Gateway

None. After #469 the data plane holds no database state at all and reaches everything over RPC.

Open decisions

  1. Is the audit writer the same process as the broker? PRD 0070 wants the broker small enough to audit line-by-line; an audit-append endpoint adds parsing surface inside the privileged launcher. Proposal: one daemon for install simplicity, but structurally separate handlers with no shared parsing, and the launch op requiring the signed JWT while audit-append takes a plain bearer token. That does not help against orchestrator compromise (it holds both creds) but stops a bug in the boring path from reaching the privileged one.
  2. Symmetric or asymmetric signing? PRD 0070 specifies asymmetric; the code is HS256. Proposal: stay symmetric. Asymmetric matters when the verifier is less privileged than the signer, and here it is the reverse — a broker that can forge orchestrator requests gains nothing, since it is already the component that launches. Also keeps the no-runtime-deps policy (stdlib has no Ed25519).
  3. How wide does the request schema get? Each op moved off the CLI widens the privileged surface. Needs an explicit rule for what stays expressible as ids + static flags.

Cost

This converts on-demand privilege into standing privilege: today the Docker socket is exercised by a CLI the user invokes, afterwards a privileged daemon runs continuously under launchd/systemd. PRD 0070's argument is that the broker's privilege is narrower (structured requests vs. a raw socket), which holds — but "always running" is a new property, and the install/upgrade story grows. Precedent exists in the Firecracker one-time privileged pool setup.

Related

  • PRD 0070 — the contract, the launch broker, and the state tiers this implements
  • #469 — get bot-bottle.db off the data plane (lands underneath this)
  • #391 — backend-agnostic orchestrator restart (targets the bootstrap path)
  • #386 — prebuilt images from the Gitea OCI registry (the fixed image set the broker validates against)
  • #355 — generic SecretProvider

Implementation will need a PRD.

Currently container launches are done directly via the CLI. We want to move this so both the CLI and the orchestrator communicate with a host-side control server, which becomes the single privileged component on the host: it brokers launches, owns orchestrator lifecycle, and is the sole writer of host-durable state. ## Launch flows ### Web console ``` web console -(iroh)-> orchestrator -(http)-> host_controller -> launch ``` ### CLI ``` cli -(http)-> orchestrator -(http)-> host_controller -> launch ``` ### Bootstrap / recovery The host controller owns orchestrator lifecycle, so orchestrator startup cannot route *through* the orchestrator. The CLI needs a direct channel for bootstrap and recovery even though all bottle operations funnel through the orchestrator: ``` cli -(http)-> host_controller # start / restart / status of the orchestrator itself ``` This is the path #391 (backend-agnostic orchestrator restart) would target. The prize is that after this, **the CLI no longer needs the Docker socket**. That is what lets a dedicated Gitea runner user drop the root-equivalent `docker` group (called out in PRD 0070 under "Relationship to other work"). ## Existing scaffold [`orchestrator/broker.py`](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/bot_bottle/orchestrator/broker.py) has a well-formed *contract* — `LaunchRequest` (ids + static flags only, never argv), HS256 sign/verify, fixed-schema validation, fail-closed `BrokerAuthError`, and a `StubBroker` that exercises the full sign → verify → act path in-process. `DockerBroker` exists but is on no production path; every backend starts the orchestrator with `--broker stub`. Four gaps between that and a real host service: 1. **No transport.** `LaunchBroker.submit(token)` is an in-process method call from `Orchestrator.launch_bottle` ([`service.py`](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/bot_bottle/orchestrator/service.py) L115). Needs a `BrokerClient` that POSTs the signed token and a host-side HTTP server that verifies and acts. 2. **The signing secret is ephemeral and self-generated.** [`orchestrator/__main__.py`](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/bot_bottle/orchestrator/__main__.py) L58 does `secrets.token_bytes(32)` and hands the same value to signer and verifier — only viable because they share a process. A separate daemon needs it provisioned out of band and durable across orchestrator restarts. 3. **Replay protection is claimed but not enforced.** `sign_request` emits `jti` and `iat`; `verify_request` reads neither. No expiry window, no `jti` cache. Harmless in-process; over a wire a captured launch token replays indefinitely. 4. **The op vocabulary is `launch` / `teardown` only.** Everything else host-privileged is still in the CLI (see below), so the schema has to grow — carefully, since PRD 0070's security argument rests on "structured requests only, static flags + ids." ## What moves off the CLI Host-privileged operations currently scattered across `backend/*/consolidated_launch.py` and driven by a short-lived CLI process: - `OrchestratorService.ensure_running()` — starting the infra container - the agent `docker run` / VM boot itself - IP allocation (`next_free_ip`, `_network_container_ips`) - `provision_git_gate` — docker `cp`/`exec` of per-bottle deploy keys into the gateway - `_reprovision_running_bottles` — `docker exec printenv ENV_VAR_SECRET` against live agents - live-bottle enumeration for reconcile - agent image builds `Orchestrator.reconcile` is the tell: it takes `live_source_ips` as a parameter *only* because the orchestrator cannot see the backend. With a host service that enumeration becomes internal and the parameter goes away. ## Database / state ownership We originally wanted a single unified DB for all state mounted on the host. That isn't possible — SQLite locking is not coherent across guest kernels over a share, which is why the macOS backend already uses a container-only volume (`INFRA_DB_VOLUME = "bot-bottle-mac-db"`). Rather than working around it with a mounted volume for one shared DB, split the state by owner and lifetime. Depends on #469, which gets the DB off the data plane first. ### Orchestrator Owns all **operational** state on a volume nothing else mounts (generalizing the existing macOS design to every backend). Durable across orchestrator restarts so re-adoption works and running bottles are not bricked by a restart: - `orchestrator_bottles` — registry: `bottle_id`, `source_ip`, `identity_token`, `metadata`, `policy` - `bottled_agent_secrets` — encrypted per-bottle egress tokens - `supervise_proposals` / `supervise_responses` — the approval queue Stays SQLite: mutable, transactional, queried. The orchestrator is sole mounter as well as sole writer. ### Host Owns the **historical record**, written only by the host controller over the authenticated channel: - supervise audit entries - egress traffic log — currently written to `sys.stderr` (the container log) by [`egress_addon.py`](https://gitea.dideric.is/didericis/bot-bottle/src/branch/main/bot_bottle/egress_addon.py); both halves of "the audit record" should land in one place - host-side config (`bot_bottle_config`, `orchestrator_config`), or moved out of the DB entirely per PRD 0070's three-tier table Append-only, never updated, never transactionally queried — so this should be a **JSONL log, not SQLite**. `O_APPEND` writes are atomic and there is no locking protocol to get wrong, and hash-chaining for tamper-evidence becomes cheap. It also survives orchestrator destruction and container-runtime volume pruning (the #450 lesson), and stays readable without the orchestrator running. Note what host-location does *not* buy: integrity against a live compromised orchestrator, which makes the decisions being audited and can forge or omit entries wherever the file lives. An off-box copy is the answer to that, separately. ### Gateway None. After #469 the data plane holds no database state at all and reaches everything over RPC. ## Open decisions 1. **Is the audit writer the same process as the broker?** PRD 0070 wants the broker small enough to audit line-by-line; an audit-append endpoint adds parsing surface inside the privileged launcher. Proposal: one daemon for install simplicity, but structurally separate handlers with no shared parsing, and the launch op requiring the signed JWT while audit-append takes a plain bearer token. That does not help against orchestrator compromise (it holds both creds) but stops a bug in the boring path from reaching the privileged one. 2. **Symmetric or asymmetric signing?** PRD 0070 specifies asymmetric; the code is HS256. Proposal: stay symmetric. Asymmetric matters when the verifier is less privileged than the signer, and here it is the reverse — a broker that can forge orchestrator requests gains nothing, since it is already the component that launches. Also keeps the no-runtime-deps policy (stdlib has no Ed25519). 3. **How wide does the request schema get?** Each op moved off the CLI widens the privileged surface. Needs an explicit rule for what stays expressible as ids + static flags. ## Cost This converts on-demand privilege into standing privilege: today the Docker socket is exercised by a CLI the user invokes, afterwards a privileged daemon runs continuously under launchd/systemd. PRD 0070's argument is that the broker's privilege is *narrower* (structured requests vs. a raw socket), which holds — but "always running" is a new property, and the install/upgrade story grows. Precedent exists in the Firecracker one-time privileged pool setup. ## Related - PRD 0070 — the contract, the launch broker, and the state tiers this implements - #469 — get `bot-bottle.db` off the data plane (lands underneath this) - #391 — backend-agnostic orchestrator restart (targets the bootstrap path) - #386 — prebuilt images from the Gitea OCI registry (the fixed image set the broker validates against) - #355 — generic `SecretProvider` Implementation will need a PRD.
gitea-actions bot added the Status/Needs Triage label 2026-07-23 20:40:57 -04:00
didericis-claude changed title from WIP - Create host side control server to prepare for console controlled launch to Host-side control server: broker bottle launches and own host-durable state 2026-07-23 21:21:33 -04:00
didericis-claude added Kind/Feature
Priority
High
2
 and removed Status/Needs Triage labels 2026-07-23 21:21:43 -04:00
gitea-actions bot added the Status/Needs Triage label 2026-07-23 21:21:51 -04:00
didericis removed the Status/Needs Triage label 2026-07-23 22:57:29 -04:00
didericis commented 2026-07-23 23:02:48 -04:00
Author
Owner
Copy Link
Copy Source
  1. structurally separate handlers is a good idea. One server/one process also makes sense.
  2. symmetric is fine
  3. should be fairly wide (we don't want low level commands exposed, only high level ones like "provision-bottle", "ensure-running", etc) @didericis-codex take a stab at identifying a minimum set of high level endpoints we could provide to keep the existing cli functionality, will review and go into the code to verify. Will do a first pass best guess target, but will probably require more direction that becomes clearer during implementation.
1. structurally separate handlers is a good idea. One server/one process also makes sense. 2. symmetric is fine 3. should be fairly wide (we don't want low level commands exposed, only high level ones like "provision-bottle", "ensure-running", etc) @didericis-codex take a stab at identifying a minimum set of high level endpoints we could provide to keep the existing cli functionality, will review and go into the code to verify. Will do a first pass best guess target, but will probably require more direction that becomes clearer during implementation.
didericis-codex commented 2026-07-23 23:10:38 -04:00
Collaborator
Copy Link
Copy Source

I did a code pass over the current CLI and backend interfaces. My first-pass minimum is two deliberately separate HTTP surfaces on the same process.

CLI/bootstrap surface (host bearer auth)

  • GET /v1/status
    • Controller version, selected/default backend readiness, orchestrator state, and enough diagnostic detail to implement backend status.
  • POST /v1/orchestrator/ensure-running
    • Idempotent. Implements the bootstrap path needed by start and supervise.
  • POST /v1/orchestrator/restart
    • The explicit recovery path for #391. I would not make this generic process control.
  • GET /v1/bottles
    • Active bottles across all backends. Implements list active and supplies the selection data for commit/cleanup.
  • POST /v1/bottles/{identity}/attach-agent
    • Bidirectional streamed stdio/PTY attachment to the provider entrypoint stored with the bottle. No caller-supplied executable or argv; the controller derives the command from the provisioned bottle plus a narrow mode such as interactive or headless.
  • POST /v1/bottles/{identity}/snapshot
    • High-level equivalent of commit; the controller selects the backend freezer and records the resulting resumable artifact.
  • GET /v1/cleanup-plan
    • Read-only enumeration of orphaned resources, returning an opaque plan id/digest so the CLI can preserve its current preview + confirmation.
  • POST /v1/cleanup
    • Takes only that opaque plan id/digest and executes the previously displayed plan. This avoids turning cleanup into arbitrary resource deletion.

I would keep backend setup/backend teardown out of the daemon API for v1. They install or remove the daemon's own host prerequisites and are better treated as install/uninstall operations. init, edit, info, list available, and login are also non-privileged/local and do not need controller endpoints. The existing supervise proposal/response API can remain on the orchestrator.

Orchestrator/broker surface (signed request auth)

  • POST /v1/broker/operations

One transport endpoint, but a closed tagged union rather than a generic command:

  • provision-bottle
  • teardown-bottle
  • reconcile-host

provision-bottle should carry a versioned, fully declarative launch spec: bottle/identity ids, backend, approved image reference, network allocation intent, policy/metadata references, and static feature flags. It must not accept an executable, argv, shell text, arbitrary runtime flags, arbitrary mounts, arbitrary devices, or caller-chosen container names. resume is the same operation with an existing identity; image/snapshot selection stays controller-owned. Image building can initially be an internal step of provision-bottle, then become only fixed-ref validation once #386 lands.

teardown-bottle is idempotent and identity-scoped. reconcile-host takes no caller-supplied live-IP list: the controller enumerates the runtimes itself, reconciles against orchestrator records, and returns the result.

Separately, the audit handler should be POST /v1/audit/events with its plain bearer credential, as proposed in the issue. It should share neither request parsing nor dispatch with /v1/broker/operations.

That maps the existing commands as follows:

  • start = local manifest/UI resolution → ensure orchestrator → provision → attach → teardown
  • resume = provision existing identity → attach → teardown
  • list active = list bottles
  • commit = snapshot
  • cleanup = cleanup plan → confirm → execute
  • supervise = ensure orchestrator, then existing orchestrator API

One likely implementation clarification: the attach endpoint is required to actually remove Docker/VM control from the CLI. Leaving docker exec -it (or its Firecracker/macOS equivalents) in the CLI would retain the privilege we are trying to eliminate. The transport can be settled during the PRD, but the capability boundary belongs in v1.

I did a code pass over the current CLI and backend interfaces. My first-pass minimum is two deliberately separate HTTP surfaces on the same process. ### CLI/bootstrap surface (host bearer auth) - `GET /v1/status` - Controller version, selected/default backend readiness, orchestrator state, and enough diagnostic detail to implement `backend status`. - `POST /v1/orchestrator/ensure-running` - Idempotent. Implements the bootstrap path needed by `start` and `supervise`. - `POST /v1/orchestrator/restart` - The explicit recovery path for #391. I would not make this generic process control. - `GET /v1/bottles` - Active bottles across all backends. Implements `list active` and supplies the selection data for `commit`/cleanup. - `POST /v1/bottles/{identity}/attach-agent` - Bidirectional streamed stdio/PTY attachment to the provider entrypoint stored with the bottle. No caller-supplied executable or argv; the controller derives the command from the provisioned bottle plus a narrow mode such as `interactive` or `headless`. - `POST /v1/bottles/{identity}/snapshot` - High-level equivalent of `commit`; the controller selects the backend freezer and records the resulting resumable artifact. - `GET /v1/cleanup-plan` - Read-only enumeration of orphaned resources, returning an opaque plan id/digest so the CLI can preserve its current preview + confirmation. - `POST /v1/cleanup` - Takes only that opaque plan id/digest and executes the previously displayed plan. This avoids turning cleanup into arbitrary resource deletion. I would keep `backend setup`/`backend teardown` out of the daemon API for v1. They install or remove the daemon's own host prerequisites and are better treated as install/uninstall operations. `init`, `edit`, `info`, `list available`, and `login` are also non-privileged/local and do not need controller endpoints. The existing supervise proposal/response API can remain on the orchestrator. ### Orchestrator/broker surface (signed request auth) - `POST /v1/broker/operations` One transport endpoint, but a closed tagged union rather than a generic command: - `provision-bottle` - `teardown-bottle` - `reconcile-host` `provision-bottle` should carry a versioned, fully declarative launch spec: bottle/identity ids, backend, approved image reference, network allocation intent, policy/metadata references, and static feature flags. It must not accept an executable, argv, shell text, arbitrary runtime flags, arbitrary mounts, arbitrary devices, or caller-chosen container names. `resume` is the same operation with an existing identity; image/snapshot selection stays controller-owned. Image building can initially be an internal step of `provision-bottle`, then become only fixed-ref validation once #386 lands. `teardown-bottle` is idempotent and identity-scoped. `reconcile-host` takes no caller-supplied live-IP list: the controller enumerates the runtimes itself, reconciles against orchestrator records, and returns the result. Separately, the audit handler should be `POST /v1/audit/events` with its plain bearer credential, as proposed in the issue. It should share neither request parsing nor dispatch with `/v1/broker/operations`. That maps the existing commands as follows: - `start` = local manifest/UI resolution → ensure orchestrator → provision → attach → teardown - `resume` = provision existing identity → attach → teardown - `list active` = list bottles - `commit` = snapshot - `cleanup` = cleanup plan → confirm → execute - `supervise` = ensure orchestrator, then existing orchestrator API One likely implementation clarification: the attach endpoint is required to actually remove Docker/VM control from the CLI. Leaving `docker exec -it` (or its Firecracker/macOS equivalents) in the CLI would retain the privilege we are trying to eliminate. The transport can be settled during the PRD, but the capability boundary belongs in v1.
didericis commented 2026-07-23 23:38:42 -04:00
Author
Owner
Copy Link
Copy Source

@didericis-claude should cleanup the cli a little bit before this:

  • remove info
  • change list active to active
  • change list available to list

File a new issue and implement a new PR to clean up the cli, and do this.

@didericis-claude should cleanup the cli a little bit before this: - remove `info` - change `list active` to `active` - change `list available` to `list` File a new issue and implement a new PR to clean up the cli, and do this.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label Kind/Feature
Priority
High
2
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#468
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?