Reevaluated against the actual nft ruleset: the firecracker split's isolation
is nearly free, not "the real work." Agent VMs are already dropped except the
DNAT'd gateway ports, so re-pointing that single DNAT rule at the gateway VM
(dnat to $(gw_guest)) isolates agents from a separate orchestrator VM with zero
new agent rules; the only added nft is a mirrored gateway link + one
gateway->orchestrator forward rule. The effort is the mechanical second-VM
lifecycle, and it's validatable on a Firecracker host.
Sharpened the Access bullet accordingly and added a note to Sequencing
decoupling the original consolidation's "real work" (the broker shim) from the
split, whose difficulty actually runs the other way (docker/macOS do the real
work — new control network, dual-homed gateway, the token-only->L3 upgrade;
firecracker is nearly free).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Remote terminal is an orchestrator operation, not a data-plane one: the
agent PTY is exec_agent-sourced (ssh -t / container exec --tty / docker
exec), which the orchestrator owns — the gateway never holds a PTY. A
--remote flag publishes a running agent's session; local + remote clients
share it via a session multiplexer (one PTY, many clients),
operator-role-gated, so the split's "one authenticated remote door" holds
for terminals too. Detailed design (multiplexer location, input
arbitration, per-client rendering, always-on vs --remote) tracked in #478.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
v1 decision: agent-image builds run in the orchestrator (the control-plane
side already owns launches and hosts builds in today's combined unit; the
data-plane gateway must not build). Consequence recorded in the memory
section: the orchestrator keeps the ~4 GB buildah ceiling and the gateway
is the slim unit (~1 GB), so the split is ~memory-neutral-to-+1 GB for now
— the ~1.5 GB win waits on a later move to a dedicated slim build unit
(#468-adjacent). Also flags the tension of co-locating buildah with the
control plane. Removed the corresponding open question.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Now that #469 got the DB off the data plane (removing the cross-kernel
SQLite-coherence reason the planes were consolidated), separating them
into distinct runtimes is possible again. Adds a Design subsection to
0070 covering:
* Access — the only hop that changes is gateway->orchestrator (loopback
-> network). A dedicated `bot-bottle-orchestrator` --internal control
network with a dual-homed gateway gives Docker/macOS the L3 agent
isolation that only Firecracker's nft currently has (both are
token-only today); Firecracker gets it nearly free (agents already
can't reach non-DNAT'd ports).
* Memory — firecracker's mem_size_mib is a fixed ceiling (no reclaim
without a balloon); the 4 GB combined VM is really buildah headroom.
Right-sized: ~512 MiB orchestrator + ~1 GB gateway < today's 4 GB,
if agent-image builds move off the gateway.
* iroh belongs in the orchestrator (control-plane remote entry point);
keep the data-plane gateway slim and P2P-free.
Plus open questions: where agent-image builds run post-split, and whether
agent-PTY-to-mobile relays through the orchestrator's iroh endpoint.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
PRD 0070's rule — only the orchestrator opens bot-bottle.db; the data
plane reaches state through the control-plane RPC — was not in force.
Three data-plane daemons held a direct read-write handle on the shared
SQLite file: the supervise MCP server, the egress DLP addon (the most
attack-exposed process, TLS-bumping hostile traffic), and the git-gate
pre-receive hook. An RCE in any of them could read every bottle's
plaintext identity_token and forge attribution fleet-wide (issue #469).
Add the agent half of the supervise flow to the control plane:
POST /supervise/propose -> queue a proposal, 201 {proposal_id}
POST /supervise/poll -> non-blocking decision poll, 200 {status,...}
Both attribute the caller by (source_ip, identity_token) exactly like
/resolve — never a caller-supplied slug — so a bottle can only ever queue
or read its own proposals even if the data plane is compromised. A decided
poll archives server-side, preserving the archive-after-read contract.
Data plane: the supervise server, egress addon, and git-gate hook now
queue/poll through PolicyResolver.propose_supervise / poll_supervise
instead of opening the DB. supervise_server keeps its ~30s grace window
by polling the RPC; egress keeps its safelist keyed by resolved bottle;
the git-gate hook gets (source_ip, identity_token) from the CGI env.
Packaging: drop the DB bind-mount and SUPERVISE_DB_PATH from the
data-plane containers/VMs (docker gateway + infra, macOS infra,
firecracker infra). The orchestrator remains the sole opener of the one
file via BOT_BOTTLE_ROOT / host_db_path().
Update PRD 0070: the rule is now in force; remove the transitional caveat.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The shared gateway self-generates a mitmproxy CA that every bottle installs
to trust its TLS interception. It was persisted on a Docker named volume,
which survives `docker rm` but is silently wiped by `docker volume prune` /
`docker system prune --volumes` during routine host maintenance. When that
happens the gateway mints a fresh CA on restart, and every already-running
bottle fails the TLS handshake even after it re-resolves and reconnects to
the moved gateway — a re-attachment blocker distinct from #443/#445.
Move CA persistence to a host bind-mount under the app-data root
(`bot_bottle_root()/gateway-ca`, via `host_gateway_ca_dir()`), mirroring how
the shared DB and control-plane token already live on the host. Docker never
prunes a path under the root, and it stays inspectable + rotatable from the
host. mitmproxy already adopts an existing CA and generates one only on first
run, so the bind-mount gives adopt-existing/generate-on-first-run for free.
Add an explicit rollover path: `rotate_gateway_ca()` clears the persisted CA
so the next start remints it, and `python -m bot_bottle.orchestrator.rotate_ca`
wires that together with dropping the running gateway container (whose
mitmproxy still holds the old CA in memory). Rotation stays an operator action
— it doesn't auto-re-provision running bottles, which re-attach to pick up the
new anchor.
Scope: the Docker infra/gateway path (the "infra container" in the report).
The macOS (`container`-only volume) and Firecracker (VM-attached ext4) backends
persist the CA differently and are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Retire "sidecar" for the consolidated per-host path (PRD 0070 naming
decision): the orchestrator is the umbrella/control plane, and the
egress/git/supervise data-plane unit it runs is the "gateway".
- git mv sidecar.py -> gateway.py and the two integration + one unit test
files; DockerSidecar->DockerGateway, Sidecar->Gateway,
SidecarError->GatewayError, SIDECAR_*->GATEWAY_*, ensure_sidecar->
ensure_gateway, sidecar_status->gateway_status, container name
bot-bottle-orch-sidecar->bot-bottle-orch-gateway.
- Prose rename across broker/registry/egress/policy_resolver + PRD 0070.
- Preserved: the image name bot-bottle-sidecars, the
BOT_BOTTLE_SIDECAR_IMAGE env var, Dockerfile.sidecars, and PRD 0069's
own stage-name cross-references (that doc still uses "sidecar").
No behavior change. Full unit suite green (1679 tests; the 13
test_sidecar_init /bin/sleep errors are pre-existing NixOS-local noise).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Per review: use one shared bot-bottle.db for all runtime state including
the registry (DbStore namespaces by schema_key), so it's one queryable
file for backup/console. default_db_path() -> host_db_path(). Drop the
unilateral WAL flip — WAL on the shared DB affects supervise/audit and is
finicky over guest shares, so it's a deliberate future change; keep a
busy_timeout for lock contention.
PRD State section updated: integrity now by SOLE ownership (only the
orchestrator opens bot-bottle.db; data plane + console reach state via the
control-plane RPC, never a file handle) rather than ro/rw mount-splitting,
which one shared file can't do. Notes the transitional caveat that the
supervise sidecar currently rw-mounts bot-bottle.db.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Per review: the SQLite runtime-state DB lives on the host, not owned
inside the orchestrator unit. Durability (re-adoption must survive an
orchestrator restart) + integrity via access-scoping (control-plane rw,
data-plane ro) rather than location. Note the WAL-over-guest-share wrinkle
for the VM slices (may want a host-side DB owner reached over the RPC).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Per review: resolvable at implementation time (host tweak acceptable, as
the pool setup already needs on NixOS); the earlier sidecars-in-VMs spike
showed feasibility. No need to hunt the spike now.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Add a "Secret handling — FUTURE pattern (not v1)" subsection: vault as a
separate trust domain holding long-lived roots, deriving short-lived
scoped creds where the upstream allows (with the honest limit that a
compromised proxy can still abuse currently-authorized access). The
mechanism is a generic, user-extensible SecretProvider that generalizes
PRD 0048's DeployKeyProvisioner and drops into the manifest wherever a raw
token is accepted — discovered from ~/.bot-bottle/contrib like user
AgentProviders. Marked explicitly as not required for the initial
orchestrator; tracked as #355.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
Fold the per-bottle sidecar bundle into a single persistent per-host
orchestrator: runs the sidecar functions (egress/git-gate/supervise),
coordinates with the console, and brokers agent launches. Virtualized
from the start with backend-native isolation (fc VM / apple ctr / docker
ctr), fronted by a single backend-agnostic contract; per-backend
variation lives on BottleBackend, not a parallel Orchestrator hierarchy.
Leads with the security review (secret concentration, shared fate, the
launch-broker-as-new-privileged-core, and the source-IP attribution
invariant each backend must enforce). Proposes one SQLite DB owned by the
orchestrator for runtime state (slot leases, approvals, registry) —
distinct from build-time constants (flat .env) and user config
(declarative ~/.bot-bottle). Sequences docker -> firecracker -> macos,
developing the service as a plain-process dev-harness before the VM.
Supersedes 0069's Stage-1/4 sidecar framing; depends on 0069's nix-built
fixed images. Tracking issue #351.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WBMWTEtQdJ4W5UrWuLHCck
didericis-codex
didericis
didericis-claude