didericis/bot-bottle
1
0
Fork 0
Code Issues 33 Pull Requests 5 Actions 2 Packages Releases Wiki Activity

fix(supervise): get bot-bottle.db off the data plane (supervise + egress proposals over RPC) #471

Open
didericis-claude wants to merge 6 commits from fix/db-off-data-plane-469 into main
pull from: fix/db-off-data-plane-469
merge into: didericis:main
Conversation 9 Commits 6 Files Changed 33
+1743 -646
didericis-claude commented 2026-07-23 22:08:35 -04:00
Collaborator
Copy Link
Copy Source

Closes #469.

What

Enforces PRD 0070's rule — only the orchestrator opens bot-bottle.db; the data plane reaches state through the control-plane RPC, never a direct file handle. Three data-plane daemons held a direct read-write handle on the shared SQLite file: the supervise MCP server, the egress DLP addon (the most attack-exposed process — TLS-bumping and body-scanning hostile traffic), and the git-gate pre-receive hook (the third writer the issue didn't enumerate; it runs in the same gateway container, so the DB mount couldn't be dropped without migrating it too).

Control plane — agent half of the supervise flow

Two new endpoints on orchestrator/control_plane.py, both attributed by (source_ip, identity_token) exactly like /resolve (never a caller-supplied slug), so a bottle can only ever queue or read its own proposals even if the data plane is compromised:

  • POST /supervise/propose → 201 {proposal_id}
  • POST /supervise/poll → 200 {status, notes?, final_file?} (pending | unknown | terminal decision). A decided poll archives server-side, preserving the archive-after-read contract.

Backed by Orchestrator.supervise_queue_proposal / supervise_poll_response, and reached through new PolicyResolver.propose_supervise / poll_supervise client methods.

Data plane — no more DB handle

  • supervise_server.py: queues via RPC and keeps its ~30s grace window by polling the RPC (no wait_for_response/write_proposal/read_response/archive_proposal).
  • egress_addon.py: token-allow block queues/polls over RPC; safelist still keyed by the resolved bottle.
  • git_gate_render.py: the gitleaks-allow hook queues/polls over RPC using (source_ip, identity_token) threaded into the CGI env by git_http_backend.py.

Packaging

Drop the DB bind-mount and SUPERVISE_DB_PATH from every data-plane container/VM (docker gateway + infra, macOS infra, firecracker infra). Each consolidated backend's SUPERVISE_DB_PATH equaled host_db_path(), so the orchestrator falls back to the same file via BOT_BOTTLE_ROOT and remains the sole opener.

Docs

PRD 0070: the rule is now in force; transitional caveat removed.

Tests

Rewrote the supervise-server and egress supervise tests around a fake resolver backed by the real queue (contracts preserved through the RPC seam); added propose/poll coverage for the control plane, service, and PolicyResolver, including cross-bottle isolation. Full unit suite green except 10 pre-existing test_cli_start_headless errors (environment: sandbox stdin isn't a PTY; identical on the base branch).

🤖 Generated with Claude Code

Closes #469. ## What Enforces PRD 0070's rule — *only the orchestrator opens `bot-bottle.db`; the data plane reaches state through the control-plane RPC, never a direct file handle*. Three data-plane daemons held a direct read-write handle on the shared SQLite file: the supervise MCP server, the **egress DLP addon** (the most attack-exposed process — TLS-bumping and body-scanning hostile traffic), and the **git-gate pre-receive hook** (the third writer the issue didn't enumerate; it runs in the same gateway container, so the DB mount couldn't be dropped without migrating it too). ## Control plane — agent half of the supervise flow Two new endpoints on `orchestrator/control_plane.py`, both attributed by `(source_ip, identity_token)` exactly like `/resolve` (never a caller-supplied slug), so a bottle can only ever queue or read **its own** proposals even if the data plane is compromised: - `POST /supervise/propose` → `201 {proposal_id}` - `POST /supervise/poll` → `200 {status, notes?, final_file?}` (`pending` | `unknown` | terminal decision). A decided poll archives server-side, preserving the archive-after-read contract. Backed by `Orchestrator.supervise_queue_proposal` / `supervise_poll_response`, and reached through new `PolicyResolver.propose_supervise` / `poll_supervise` client methods. ## Data plane — no more DB handle - `supervise_server.py`: queues via RPC and keeps its ~30s grace window by polling the RPC (no `wait_for_response`/`write_proposal`/`read_response`/`archive_proposal`). - `egress_addon.py`: token-allow block queues/polls over RPC; safelist still keyed by the resolved bottle. - `git_gate_render.py`: the gitleaks-allow hook queues/polls over RPC using `(source_ip, identity_token)` threaded into the CGI env by `git_http_backend.py`. ## Packaging Drop the DB bind-mount and `SUPERVISE_DB_PATH` from every data-plane container/VM (docker gateway + infra, macOS infra, firecracker infra). Each consolidated backend's `SUPERVISE_DB_PATH` equaled `host_db_path()`, so the orchestrator falls back to the same file via `BOT_BOTTLE_ROOT` and remains the sole opener. ## Docs PRD 0070: the rule is now in force; transitional caveat removed. ## Tests Rewrote the supervise-server and egress supervise tests around a fake resolver backed by the real queue (contracts preserved through the RPC seam); added propose/poll coverage for the control plane, service, and `PolicyResolver`, including cross-bottle isolation. Full unit suite green except 10 pre-existing `test_cli_start_headless` errors (environment: sandbox stdin isn't a PTY; identical on the base branch). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
didericis added 1 commit 2026-07-23 23:00:52 -04:00
fix(supervise): reach the queue over RPC, get bot-bottle.db off the data plane 2c496dc3d0 
PRD 0070's rule — only the orchestrator opens bot-bottle.db; the data
plane reaches state through the control-plane RPC — was not in force.
Three data-plane daemons held a direct read-write handle on the shared
SQLite file: the supervise MCP server, the egress DLP addon (the most
attack-exposed process, TLS-bumping hostile traffic), and the git-gate
pre-receive hook. An RCE in any of them could read every bottle's
plaintext identity_token and forge attribution fleet-wide (issue #469).

Add the agent half of the supervise flow to the control plane:

  POST /supervise/propose  -> queue a proposal, 201 {proposal_id}
  POST /supervise/poll     -> non-blocking decision poll, 200 {status,...}

Both attribute the caller by (source_ip, identity_token) exactly like
/resolve — never a caller-supplied slug — so a bottle can only ever queue
or read its own proposals even if the data plane is compromised. A decided
poll archives server-side, preserving the archive-after-read contract.

Data plane: the supervise server, egress addon, and git-gate hook now
queue/poll through PolicyResolver.propose_supervise / poll_supervise
instead of opening the DB. supervise_server keeps its ~30s grace window
by polling the RPC; egress keeps its safelist keyed by resolved bottle;
the git-gate hook gets (source_ip, identity_token) from the CGI env.

Packaging: drop the DB bind-mount and SUPERVISE_DB_PATH from the
data-plane containers/VMs (docker gateway + infra, macOS infra,
firecracker infra). The orchestrator remains the sole opener of the one
file via BOT_BOTTLE_ROOT / host_db_path().

Update PRD 0070: the rule is now in force; remove the transitional caveat.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis force-pushed fix/db-off-data-plane-469 from 8a82861459 to 2acaa082e4 2026-07-23 23:00:52 -04:00 Compare
didericis force-pushed fix/db-off-data-plane-469 from 2acaa082e4 to 385cca72a7 2026-07-23 23:05:01 -04:00 Compare
didericis force-pushed fix/db-off-data-plane-469 from 385cca72a7 to 30d74cbf2b 2026-07-23 23:24:23 -04:00 Compare
didericis added 1 commit 2026-07-23 23:24:40 -04:00
test: annotations + coverage for the supervise RPC seam
tracker-policy-pr / check-pr (pull_request) Successful in 6s
Details
test / integration-docker (pull_request) Successful in 15s
Details
test / unit (pull_request) Successful in 39s
Details
test / integration-firecracker (pull_request) Successful in 3m20s
Details
test / coverage (pull_request) Successful in 44s
Details
test / publish-infra (pull_request) Has been skipped
Details
72fdb1d14b 
Add pyright-strict parameter/return annotations to the fake resolvers and
test helpers, and cover the new control-plane validation/403 branches
(/supervise/propose + /supervise/poll) plus the supervise-server and egress
poll-error fail-closed paths, so the diff-coverage gate stays above 90%.
No production behavior change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis force-pushed fix/db-off-data-plane-469 from 30d74cbf2b to 72fdb1d14b 2026-07-23 23:24:40 -04:00 Compare
didericis-codex requested changes 2026-07-24 00:24:31 -04:00
Dismissed
didericis-codex left a comment
Collaborator
Copy Link
Copy Source

Found two blocking issues:

  • High — data-plane compromise can bypass supervision entirely. The gateway receives BOT_BOTTLE_CONTROL_PLANE_TOKEN, and PolicyResolver sends it with the new supervise RPCs. That same token authorizes every control-plane route, including /supervise/respond. A compromised egress or git-gate process can queue a proposal and approve it directly—or modify policies and inspect other proposals—without operator involvement. The new (source_ip, identity_token) checks do not protect trusted operator routes. Use a separate, least-privilege data-plane credential restricted to /resolve, /supervise/propose, and /supervise/poll. See bot_bottle/policy_resolver.py:69-77 and bot_bottle/orchestrator/control_plane.py:100-114.

  • Medium — polling can permanently lose an operator decision. supervise_poll_response() archives the proposal before the HTTP response is successfully delivered. If the connection drops after archiving, the retry receives unknown; egress, supervise, and git-gate eventually fail or time out despite approval. The old flow archived only after the caller received the response. Keep terminal decisions available for idempotent polls, or add an explicit acknowledgement before archiving. See bot_bottle/orchestrator/service.py:225-250.

I would request changes before merging.

Found two blocking issues: - **High — data-plane compromise can bypass supervision entirely.** The gateway receives `BOT_BOTTLE_CONTROL_PLANE_TOKEN`, and `PolicyResolver` sends it with the new supervise RPCs. That same token authorizes every control-plane route, including `/supervise/respond`. A compromised egress or git-gate process can queue a proposal and approve it directly—or modify policies and inspect other proposals—without operator involvement. The new `(source_ip, identity_token)` checks do not protect trusted operator routes. Use a separate, least-privilege data-plane credential restricted to `/resolve`, `/supervise/propose`, and `/supervise/poll`. See `bot_bottle/policy_resolver.py:69-77` and `bot_bottle/orchestrator/control_plane.py:100-114`. - **Medium — polling can permanently lose an operator decision.** `supervise_poll_response()` archives the proposal before the HTTP response is successfully delivered. If the connection drops after archiving, the retry receives `unknown`; egress, supervise, and git-gate eventually fail or time out despite approval. The old flow archived only after the caller received the response. Keep terminal decisions available for idempotent polls, or add an explicit acknowledgement before archiving. See `bot_bottle/orchestrator/service.py:225-250`. I would request changes before merging.
didericis commented 2026-07-24 00:42:04 -04:00
Owner
Copy Link
Copy Source

@didericis-claude RE first issue @didericis-codex mentioned, implement roles embedded into JWT tokens the orchestrator issues and decrypts. Roles should be based on service type (gateway, cli, etc)

@didericis-claude RE first issue @didericis-codex mentioned, implement roles embedded into JWT tokens the orchestrator issues and decrypts. Roles should be based on service type (`gateway`, `cli`, etc)
didericis added 1 commit 2026-07-24 01:23:01 -04:00
feat(control-plane): role-scoped signed tokens so the gateway can't drive operator routes
test / integration-docker (pull_request) Successful in 17s
Details
tracker-policy-pr / check-pr (pull_request) Successful in 16s
Details
test / unit (pull_request) Successful in 39s
Details
lint / lint (push) Successful in 56s
Details
test / integration-firecracker (pull_request) Successful in 3m21s
Details
test / coverage (pull_request) Successful in 19s
Details
test / publish-infra (pull_request) Has been skipped
Details
1db2a9eb67 
Review follow-up on #469: the data plane held the same control-plane secret
that authorizes every route, so a compromised egress/git-gate could queue a
supervise proposal AND approve it (or rewrite policy, read injected tokens) —
the (source_ip, identity_token) checks attribute the *bottle*, not the caller.

Replace the single shared bearer secret with role-scoped, HMAC-signed tokens
(compact HS256 JWTs, stdlib-only — no new dependency):

  * new `control_auth` mints/verifies `{role}` tokens; roles are `gateway`
    (data plane) and `cli` (host operator/launcher).
  * the orchestrator holds only the signing *key* and verifies; `dispatch`
    gates each route by role — `gateway` reaches /resolve + /supervise/
    {propose,poll}, everything else is `cli`-only (401 unauthenticated,
    403 wrong role).
  * the gateway is handed a pre-minted `gateway` token it cannot rewrite into
    `cli`; the host CLI mints its own `cli` token from the host key.
  * `gateway_init` scopes the signing key to the orchestrator process and the
    gateway token to the data-plane daemons, so even in the combined infra
    container a compromised data-plane daemon never sees the key.

Launchers (docker gateway + infra, macOS infra) inject the minted token(s);
Firecracker stays open behind its nft boundary. Open mode (no key) still grants
full `cli` access — the fail-visible fallback for tests.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-claude commented 2026-07-24 01:23:32 -04:00
Author
Collaborator
Copy Link
Copy Source

@didericis @didericis-codex — done. Pushed feat(control-plane): role-scoped signed tokens addressing codex's High finding via role JWTs.

What changed

  • New bot_bottle/control_auth.py: stdlib HS256 (no dependency) mint/verify of compact {"role": ...} tokens. Roles: gateway (data plane) and cli (host operator/launcher).
  • The orchestrator holds only the signing key and verifies; dispatch now gates each route by role — gateway reaches /resolve + /supervise/{propose,poll}, everything else is cli-only. Wrong role → 403, missing token → 401.
  • The gateway is handed a pre-minted gateway token (new BOT_BOTTLE_CONTROL_AUTH_JWT) it can't rewrite into cli; the host CLI mints its own cli token from the host key. So a compromised egress/git-gate can no longer approve its own proposals, rewrite policy, or read injected upstream tokens.
  • gateway_init scopes the signing key to the orchestrator process and the gateway token to the data-plane daemons, so even in the combined infra container a compromised data-plane daemon never sees the key. Standalone gateway containers never get the key at all.
  • Launchers (docker gateway + infra, macOS infra) inject the minted token(s); Firecracker stays open behind its nft boundary; open mode (no key) still grants full cli access as the fail-visible test fallback.

Decisions (happy to revisit): tokens are signed, not encrypted (the role isn't a secret) and long-lived (parity with the static token they replace — the win is the unforgeable role claim, not rotation). cli is a superset that reaches every route; gateway is strictly the three data-plane lookups.

Residual: in the combined infra container the signing key still lives in the orchestrator process's env, so a root-level compromise of a co-located data-plane daemon could read /proc/<orch>/environ. That's the "run daemons as distinct non-root UIDs" hardening already noted as a follow-up in #469; the standalone gateway (the attack-exposed egress path) is fully isolated.

On the Medium finding (poll archives the decision before the HTTP response is delivered): per @didericis's direction I scoped this PR to the JWT roles only and left that one for a follow-up — it's a real idempotency bug, and the fix is small (stop archiving on supervise_poll_response and return the decision idempotently; decided proposals already drop off the operator's pending list). Say the word and I'll fold it in here instead.

New tests: test_control_auth (sign/verify, tamper, alg-confusion, alg=none), role gating in test_orchestrator_control_plane + the docker auth integration test, per-process env scoping in test_gateway_init, and the CLI/resolver token wiring. Lint + pyright clean locally; CI running.

@didericis @didericis-codex — done. Pushed `feat(control-plane): role-scoped signed tokens` addressing codex's **High** finding via role JWTs. **What changed** - New `bot_bottle/control_auth.py`: stdlib HS256 (no dependency) `mint`/`verify` of compact `{"role": ...}` tokens. Roles: `gateway` (data plane) and `cli` (host operator/launcher). - The orchestrator holds only the **signing key** and verifies; `dispatch` now gates each route by role — `gateway` reaches `/resolve` + `/supervise/{propose,poll}`, everything else is `cli`-only. Wrong role → **403**, missing token → **401**. - The gateway is handed a **pre-minted `gateway` token** (new `BOT_BOTTLE_CONTROL_AUTH_JWT`) it can't rewrite into `cli`; the host CLI mints its own `cli` token from the host key. So a compromised egress/git-gate can no longer approve its own proposals, rewrite policy, or read injected upstream tokens. - `gateway_init` scopes the signing key to the orchestrator process and the gateway token to the data-plane daemons, so even in the combined infra container a compromised data-plane daemon never sees the key. Standalone gateway containers never get the key at all. - Launchers (docker gateway + infra, macOS infra) inject the minted token(s); Firecracker stays open behind its nft boundary; open mode (no key) still grants full `cli` access as the fail-visible test fallback. **Decisions** (happy to revisit): tokens are **signed, not encrypted** (the role isn't a secret) and **long-lived** (parity with the static token they replace — the win is the unforgeable role claim, not rotation). `cli` is a superset that reaches every route; `gateway` is strictly the three data-plane lookups. **Residual**: in the *combined* infra container the signing key still lives in the orchestrator process's env, so a root-level compromise of a co-located data-plane daemon could read `/proc/<orch>/environ`. That's the "run daemons as distinct non-root UIDs" hardening already noted as a follow-up in #469; the standalone gateway (the attack-exposed egress path) is fully isolated. **On the Medium finding** (poll archives the decision before the HTTP response is delivered): per @didericis's direction I scoped this PR to the JWT roles only and left that one **for a follow-up** — it's a real idempotency bug, and the fix is small (stop archiving on `supervise_poll_response` and return the decision idempotently; decided proposals already drop off the operator's pending list). Say the word and I'll fold it in here instead. New tests: `test_control_auth` (sign/verify, tamper, alg-confusion, alg=none), role gating in `test_orchestrator_control_plane` + the docker auth integration test, per-process env scoping in `test_gateway_init`, and the CLI/resolver token wiring. Lint + pyright clean locally; CI running.
didericis commented 2026-07-24 02:12:33 -04:00
Owner
Copy Link
Copy Source

@didericis-claude make the poll idempotent and raise the interval to 250 ms

@didericis-claude make the poll idempotent and raise the interval to 250 ms
didericis added 1 commit 2026-07-24 02:26:12 -04:00
fix(supervise): make the response poll idempotent and slow it to 250ms
tracker-policy-pr / check-pr (pull_request) Successful in 15s
Details
test / integration-docker (pull_request) Successful in 42s
Details
test / unit (pull_request) Successful in 46s
Details
lint / lint (push) Successful in 57s
Details
test / integration-firecracker (pull_request) Successful in 3m22s
Details
test / coverage (pull_request) Successful in 23s
Details
test / publish-infra (pull_request) Has been skipped
Details
d8b61b3658 
The control-plane supervise poll archived the proposal on the same call
that returned the decision, so a dropped connection lost the operator's
decision (the retry saw `unknown`). Poll no longer archives — a re-poll
returns the same decision. Decided proposals still drop off the operator's
pending list (a response row exists); their rows are reaped when the
bottle is torn down or reconciled.

Also raise the grace-window poll interval from 50ms to 250ms now that each
poll is an HTTP round trip rather than a local file read.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-claude commented 2026-07-24 02:26:22 -04:00
Author
Collaborator
Copy Link
Copy Source

Done — poll is idempotent (no archive on read; a re-poll returns the same decision) and the interval is now 250ms. Decided proposals still drop off the pending list, and the rows are reaped on teardown/reconcile.

Done — poll is idempotent (no archive on read; a re-poll returns the same decision) and the interval is now 250ms. Decided proposals still drop off the pending list, and the rows are reaped on teardown/reconcile.
didericis-codex requested changes 2026-07-24 02:38:15 -04:00
Dismissed
didericis-codex left a comment
Collaborator
Copy Link
Copy Source

Blocking finding:

  • High — Firecracker still lets a compromised data-plane daemon bypass supervision. The Firecracker init script starts the orchestrator without BOT_BOTTLE_CONTROL_PLANE_TOKEN and starts the gateway daemons with no gateway JWT (bot_bottle/backend/firecracker/infra_vm.py:488-501). Consequently ControlPlaneServer enters open mode and grants every unauthenticated request the cli role (bot_bottle/orchestrator/control_plane.py:399-430). The nft boundary only prevents an agent from reaching the port; it does not constrain egress, supervise, or git-http, which run in the same infra VM and already reach the service at 127.0.0.1. If one of those data-plane daemons is compromised, it can still call /supervise/respond, policy mutation, and the other operator routes directly. Please provision a signing key and a pre-minted gateway JWT for the Firecracker processes too, with the same per-process scoping used by the other combined launcher.

The previous polling/idempotency finding is resolved, and the unit suite is passing in my re-review run.

Blocking finding: - **High — Firecracker still lets a compromised data-plane daemon bypass supervision.** The Firecracker init script starts the orchestrator without `BOT_BOTTLE_CONTROL_PLANE_TOKEN` and starts the gateway daemons with no gateway JWT (`bot_bottle/backend/firecracker/infra_vm.py:488-501`). Consequently `ControlPlaneServer` enters open mode and grants every unauthenticated request the `cli` role (`bot_bottle/orchestrator/control_plane.py:399-430`). The nft boundary only prevents an agent from reaching the port; it does not constrain egress, supervise, or git-http, which run in the same infra VM and already reach the service at `127.0.0.1`. If one of those data-plane daemons is compromised, it can still call `/supervise/respond`, policy mutation, and the other operator routes directly. Please provision a signing key and a pre-minted `gateway` JWT for the Firecracker processes too, with the same per-process scoping used by the other combined launcher. The previous polling/idempotency finding is resolved, and the unit suite is passing in my re-review run.
didericis added 1 commit 2026-07-24 03:03:10 -04:00
fix(firecracker): provision role-scoped control-plane auth in the infra VM
tracker-policy-pr / check-pr (pull_request) Successful in 9s
Details
test / integration-docker (pull_request) Successful in 16s
Details
lint / lint (push) Successful in 58s
Details
test / unit (pull_request) Successful in 2m4s
Details
test / integration-firecracker (pull_request) Successful in 3m21s
Details
test / coverage (pull_request) Successful in 16s
Details
test / publish-infra (pull_request) Has been skipped
Details
aac27d8a40 
The Firecracker infra VM started the control plane with no signing key, so it
ran open and granted every unauthenticated caller the `cli` role. The nft
boundary only fences off the separate agent VM — the egress / supervise /
git-http daemons run in this same VM and reach the orchestrator over 127.0.0.1,
so a compromised data-plane daemon could still drive the operator routes
(approve its own supervise proposals, rewrite policy, read injected tokens).

Generate the signing key on the persistent volume, hand it only to the
orchestrator process, mint a `gateway` JWT for the data-plane daemons, and
mirror the key back to the host token file so the CLI signs `cli` tokens the VM
verifies — the same per-process scoping the docker / macOS launchers use.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis-claude commented 2026-07-24 03:03:20 -04:00
Author
Collaborator
Copy Link
Copy Source

Good catch — you're right, open mode reopened the exact hole for the daemons co-located in the infra VM (the nft boundary only fences off the separate agent VM). Fixed in aac27d8: the infra VM now generates the control-plane signing key on its persistent volume, hands it only to the orchestrator process, mints a gateway JWT for the data-plane daemons, and the host mirrors the key back into its token file so the CLI signs cli tokens the VM verifies. Same per-process scoping the docker/macOS launchers use — no more open mode. Watching the firecracker integration job.

Good catch — you're right, open mode reopened the exact hole for the daemons co-located in the infra VM (the nft boundary only fences off the separate agent VM). Fixed in `aac27d8`: the infra VM now generates the control-plane signing key on its persistent volume, hands it **only** to the orchestrator process, mints a `gateway` JWT for the data-plane daemons, and the host mirrors the key back into its token file so the CLI signs `cli` tokens the VM verifies. Same per-process scoping the docker/macOS launchers use — no more open mode. Watching the firecracker integration job.
didericis-codex requested changes 2026-07-24 11:51:17 -04:00
didericis-codex left a comment
Collaborator
Copy Link
Copy Source

The original Firecracker role-bypass finding is fixed, but the fix introduces a new blocking regression:

  • High — starting/adopting Firecracker invalidates authentication for already-running Docker/macOS control planes. Firecracker now generates its own signing key on the guest volume and _with_signing_key() unconditionally overwrites the single host-wide control-plane-token with that guest key (bot_bottle/backend/firecracker/infra_vm.py:193-213). Docker and macOS orchestrators are launched with the value of that same host file and retain it in their process environment, while every new OrchestratorClient mints its cli JWT from the file’s current value. If another backend is already running, ensure_running() for Firecracker therefore changes the key used by all subsequent host clients without rotating the other live orchestrators; supervise, teardown, policy updates, and cleanup against those backends begin returning 401. Cross-backend operation is supported elsewhere in the CLI, so the signing key needs to remain host-canonical/shared, or token selection/storage must be scoped per control-plane endpoint rather than clobbering the global file.

Also, _with_signing_key() treats failure to retrieve the key as best-effort and returns a usable-looking infra handle (infra_vm.py:193-207), even though every authenticated operation in the immediately following launch will then fail with the stale/missing host key. Once key synchronization is required for correctness, this path should retry or fail explicitly rather than proceeding.

Targeted authentication, Firecracker-infra, gateway-init, and supervise tests pass; they do not cover coexistence with an already-running control plane using the previous host key.

The original Firecracker role-bypass finding is fixed, but the fix introduces a new blocking regression: - **High — starting/adopting Firecracker invalidates authentication for already-running Docker/macOS control planes.** Firecracker now generates its own signing key on the guest volume and `_with_signing_key()` unconditionally overwrites the single host-wide `control-plane-token` with that guest key (`bot_bottle/backend/firecracker/infra_vm.py:193-213`). Docker and macOS orchestrators are launched with the value of that same host file and retain it in their process environment, while every new `OrchestratorClient` mints its `cli` JWT from the file’s current value. If another backend is already running, `ensure_running()` for Firecracker therefore changes the key used by all subsequent host clients without rotating the other live orchestrators; supervise, teardown, policy updates, and cleanup against those backends begin returning 401. Cross-backend operation is supported elsewhere in the CLI, so the signing key needs to remain host-canonical/shared, or token selection/storage must be scoped per control-plane endpoint rather than clobbering the global file. Also, `_with_signing_key()` treats failure to retrieve the key as best-effort and returns a usable-looking infra handle (`infra_vm.py:193-207`), even though every authenticated operation in the immediately following launch will then fail with the stale/missing host key. Once key synchronization is required for correctness, this path should retry or fail explicitly rather than proceeding. Targeted authentication, Firecracker-infra, gateway-init, and supervise tests pass; they do not cover coexistence with an already-running control plane using the previous host key.
didericis-claude commented 2026-07-24 12:24:50 -04:00
Author
Collaborator
Copy Link
Copy Source

You're right — the mirror-into-the-shared-host-file approach was wrong; it clobbers the host-canonical key and 401s a co-running Docker/macOS control plane. Fixed in 66409c7 by keeping the host token file as the single source of truth: the launcher now pushes the host key into the freshly booted infra VM over SSH (atomic write, same transport as persist_env_var_secret), and the VM's init waits for it and refuses to start the control plane — rather than run open — if it never arrives. Nothing writes back to the host file, so other backends are untouched, and the best-effort silent path is gone (both the push and the guest fail loudly). Watching the firecracker integration job.

You're right — the mirror-into-the-shared-host-file approach was wrong; it clobbers the host-canonical key and 401s a co-running Docker/macOS control plane. Fixed in `66409c7` by keeping the host token file as the single source of truth: the launcher now **pushes** the host key into the freshly booted infra VM over SSH (atomic write, same transport as `persist_env_var_secret`), and the VM's init **waits** for it and **refuses to start the control plane** — rather than run open — if it never arrives. Nothing writes back to the host file, so other backends are untouched, and the best-effort silent path is gone (both the push and the guest fail loudly). Watching the firecracker integration job.
didericis added 1 commit 2026-07-24 12:38:13 -04:00
fix(firecracker): keep the host key canonical instead of clobbering the host token file
tracker-policy-pr / check-pr (pull_request) Successful in 9s
Details
test / integration-docker (pull_request) Successful in 17s
Details
test / unit (pull_request) Successful in 46s
Details
test / integration-firecracker (pull_request) Successful in 3m27s
Details
test / coverage (pull_request) Successful in 26s
Details
test / publish-infra (pull_request) Has been skipped
Details
4a83f45d9d 
The previous firecracker fix let the VM generate its own signing key on the
guest volume and had the host overwrite the single host-wide control-plane-token
with it. That breaks a co-running Docker/macOS control plane: their orchestrators
still verify with the old key while new host clients start signing `cli` tokens
with the guest key, so supervise/teardown/policy calls against those backends
begin returning 401 (cross-backend operation is supported).

Keep the host token file the single source of truth. The launcher now pushes the
host-canonical key into the freshly booted infra VM over SSH (atomic write,
mirroring persist_env_var_secret); the VM's init waits for it and refuses to
start the control plane — rather than run open — if it never arrives. Nothing
ever writes back to the host file, so other backends are untouched, and the
best-effort silent path is gone (both the push and the guest fail loudly).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis force-pushed fix/db-off-data-plane-469 from 66409c770b to 4a83f45d9d 2026-07-24 12:38:13 -04:00 Compare
All checks were successful
tracker-policy-pr / check-pr (pull_request) Successful in 9s
Required
Details
test / integration-docker (pull_request) Successful in 17s
Details
test / unit (pull_request) Successful in 46s
Details
test / integration-firecracker (pull_request) Successful in 3m27s
Details
test / coverage (pull_request) Successful in 26s
Details
test / publish-infra (pull_request) Has been skipped
Details
This pull request has changes requested by an official reviewer.
This branch is out-of-date with the base branch
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin fix/db-off-data-plane-469:fix/db-off-data-plane-469
git checkout fix/db-off-data-plane-469
Sign in to join this conversation.
Reviewers
No Reviewers
didericis-codex
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#471
Delete Branch "fix/db-off-data-plane-469"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?