refactor: repo reorganization for clearer separation of concerns #477

Merged

didericis merged 30 commits from refactor/repo-reorg into fix/db-off-data-plane-469 2026-07-24 19:24:56 -04:00

Conversation 0 Commits 30 Files Changed 169

+3120 -2953

didericis-claude commented 2026-07-24 12:54:46 -04:00

Collaborator

Copy Link

Copy Source

Part of #444

Umbrella branch for a broad, behavior-preserving reorganization of the repo. The goal is to make the tree easier to parse and to make separation of concerns something you can see in the layout and enforce, rather than infer from scattered modules.

Stacked on #471. Base is fix/db-off-data-plane-469, not main, so this shows a refactor-only diff. Retarget to main once #471 merges.

Why

The package root has accumulated loose modules that belong to distinct concerns (persistence, gateway data plane, orchestration), and some concerns have drifted into more than one shape. Recent review churn on #471 made the cost concrete: the control-plane auth provisioning was re-derived per backend and failed review three times (see #476). Before the substantive backend-uniformity work, group the code so each concern has one obvious home and cross-concern coupling is visible at the import site.

Landed so far

1. Gateway → backend layer. DockerGateway now lives in backend/docker/gateway.py — the shape the other backends' gateway classes will share — while orchestrator/gateway.py keeps only the backend-neutral Gateway lifecycle ABC + constants. Deleted the dead standalone-gateway path (--gateway, Orchestrator.ensure_gateway, the gateway ctor arg) that this class was the sole consumer of, and fixed a latent bug it surfaced: the shared-CA read targeted a container name (bot-bottle-orch-gateway) the consolidated flow never creates — now reads from bot-bottle-infra.
2. bot_bottle.store package. Consolidated the SQLite store family — the DbStore base, the migrations schema base, the concrete audit/config/queue stores, and StoreManager — out of the package root into bot_bottle/store/. Callers use direct submodule imports; the orchestrator's own stores stay in orchestrator/ and repoint to the moved base.

Planned (this branch)

• Backend gateway uniformity (#476): a shared control-plane-auth provisioning contract (host-canonical signing key → orchestrator, pre-minted gateway JWT → data-plane daemons) so docker / macOS / firecracker stop hand-rolling it, and real Gateway impls for the VM backends.
• Continue grouping loose root modules by concern so the layout enforces the boundaries.

No behavior change; full unit suite green (2251 tests).

🤖 Generated with Claude Code

Part of #444 Umbrella branch for a broad, **behavior-preserving** reorganization of the repo. The goal is to make the tree easier to parse and to make separation of concerns something you can *see* in the layout and enforce, rather than infer from scattered modules. > **Stacked on #471.** Base is `fix/db-off-data-plane-469`, not `main`, so this shows a refactor-only diff. Retarget to `main` once #471 merges. ## Why The package root has accumulated loose modules that belong to distinct concerns (persistence, gateway data plane, orchestration), and some concerns have drifted into more than one shape. Recent review churn on #471 made the cost concrete: the control-plane auth provisioning was re-derived per backend and failed review three times (see #476). Before the substantive backend-uniformity work, group the code so each concern has one obvious home and cross-concern coupling is visible at the import site. ## Landed so far 1. **Gateway → backend layer.** `DockerGateway` now lives in `backend/docker/gateway.py` — the shape the other backends' gateway classes will share — while `orchestrator/gateway.py` keeps only the backend-neutral `Gateway` lifecycle ABC + constants. Deleted the dead standalone-gateway path (`--gateway`, `Orchestrator.ensure_gateway`, the `gateway` ctor arg) that this class was the sole consumer of, and fixed a latent bug it surfaced: the shared-CA read targeted a container name (`bot-bottle-orch-gateway`) the consolidated flow never creates — now reads from `bot-bottle-infra`. 2. **`bot_bottle.store` package.** Consolidated the SQLite store family — the `DbStore` base, the `migrations` schema base, the concrete `audit`/`config`/`queue` stores, and `StoreManager` — out of the package root into `bot_bottle/store/`. Callers use direct submodule imports; the orchestrator's own stores stay in `orchestrator/` and repoint to the moved base. ## Planned (this branch) - **Backend gateway uniformity (#476):** a shared control-plane-auth provisioning contract (host-canonical signing key → orchestrator, pre-minted `gateway` JWT → data-plane daemons) so docker / macOS / firecracker stop hand-rolling it, and real `Gateway` impls for the VM backends. - Continue grouping loose root modules by concern so the layout enforces the boundaries. No behavior change; full unit suite green (2251 tests). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

didericis-claude added 2 commits 2026-07-24 12:54:47 -04:00

refactor(gateway): move DockerGateway to the backend layer, drop the dead standalone-gateway path ... dfce3d9505

The `DockerGateway` container-lifecycle impl now lives in
`backend/docker/gateway.py` (the shape backend gateway classes will share);
`orchestrator/gateway.py` keeps only the backend-neutral pieces (the `Gateway`
ABC, constants, `rotate_gateway_ca`).

Delete the standalone-gateway path it was the only consumer of. `--gateway` on
`python -m bot_bottle.orchestrator` was invoked nowhere — the production docker
flow runs the gateway data plane inside the combined `bot-bottle-infra`
container via `OrchestratorService`, never this class. Removing it takes with it
`Orchestrator.ensure_gateway()` and the `gateway` ctor arg; `gateway_status()`
becomes a stub reporting `configured: false` so the documented `GET /gateway`
control-plane route keeps its contract.

Fix a latent bug the move surfaced: `launch.py` read the shared gateway CA via
`docker exec bot-bottle-orch-gateway`, a container the consolidated flow never
creates — so the agent CA install was reaching a nonexistent name. Read it from
`bot-bottle-infra` (INFRA_NAME) instead.

Repoint the affected tests to the new module; drop the unit test + fake that
covered the deleted standalone wiring.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

refactor(store): consolidate the SQLite store family into bot_bottle.store ... d7a58e52fd

Move the DbStore base (db_store), the shared schema-migration base (migrations /
TableMigrations), the concrete stores (audit_store, config_store, queue_store),
and StoreManager out of the package root into a bot_bottle/store/ package, so
persistence lives in one place rather than scattered across the root.

Callers use direct submodule imports (from bot_bottle.store.store_manager
import StoreManager). Inside the moved modules the relative imports point back
up to their non-store siblings (..paths, ..supervise_types) while co-moved
siblings stay package-local; the flat-import fallbacks are untouched. The
orchestrator's own stores (config_store, secret_store, registry) stay in
orchestrator/ and repoint to the moved DbStore / TableMigrations.

No behavior change; full unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 13:04:45 -04:00

refactor(manifest): consolidate the manifest modules into a bot_bottle.manifest package ... a845cba925

Move the nine root-level manifest modules into a bot_bottle/manifest/ package,
dropping the redundant manifest_ prefix: the facade (manifest.py) becomes the
package __init__ and keeps re-exporting Manifest / ManifestIndex and the piece
types, while manifest_<x>.py become manifest/<x>.py (agent, bottle, egress,
extends, git, loader, schema, util).

Because the facade stays the package __init__, the ~13 callers that import
`from bot_bottle.manifest import ...` are unchanged; only direct-piece imports
move to `bot_bottle.manifest.<name>`. Inside the package, intra-manifest
imports drop the prefix and the non-manifest siblings (log, yaml_subset,
agent_provider) move to `..`.

No behavior change; full unit suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 13:25:14 -04:00

refactor(gateway): move the data-plane daemons into a bot_bottle.gateway package ... f77023db1d

Separate the gateway (data plane) from the orchestrator (control plane) at the
module level. The gateway runtime files move out of the package root — and the
backend-neutral Gateway lifecycle ABC + GATEWAY_* constants move out of
orchestrator/ — into a new bot_bottle/gateway/ package:

  gateway/__init__.py      (was orchestrator/gateway.py: Gateway ABC + consts
                            + rotate_gateway_ca)
  gateway/gateway_init.py  (the PID-1 daemon supervisor)
  gateway/egress_addon.py, egress_addon_core.py, egress_dlp_config.py,
          dlp_detectors.py            (the egress mitmproxy daemon)
  gateway/git_http_backend.py         (the git-http daemon)
  gateway/git_gate_render.py          (the git-gate pre-receive rendering)
  gateway/supervise_server.py         (the supervise MCP daemon)
  gateway/policy_resolver.py          (the data-plane control-plane RPC client)

orchestrator/ now holds only control-plane files. The shared plan/types/auth
layer (egress.py=EgressPlan, git_gate.py=GitGatePlan, supervise.py,
supervise_types.py, control_auth.py) and the launch-time git-gate provisioning
helpers stay at root, so orchestrator/ and backend/ still own them.

Because these daemons are invoked as `python3 -m bot_bottle.<name>`, loaded flat
by mitmproxy, and referenced in Dockerfile.gateway, the move updates more than
Python imports: the `-m` invocations (firecracker/macOS infra scripts), the
Dockerfile.gateway addon shim + ENTRYPOINT, gateway_init's _DAEMONS module
paths, and the git-gate CGI heredocs all now point at bot_bottle.gateway.*.

No behavior change; full unit suite green (2251).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 14:15:08 -04:00

refactor(supervise): split the supervise plane by tier; per-service store managers ... 44e2b5a897

Give each service its own store package + manager, and cut the supervise module
along the control/data-plane boundary so nothing in the shared layer reaches up
into the orchestrator.

Stores, by owner:
  - bot_bottle/store/ keeps only the shared base (DbStore, migrations) and the
    concrete stores that aren't service-owned (audit_store, config_store).
  - bot_bottle/orchestrator/store/ now houses the orchestrator-owned stores —
    queue_store (supervise queue), secret_store, config_store — plus a new
    orchestrator store_manager that migrates them (composing audit/config
    downward from the base). The old shared store_manager is gone.

Supervise plane, by tier:
  - bot_bottle/supervisor/ (NEUTRAL, importable by every tier including the
    gateway): types.py (the Proposal/Response/AuditEntry wire types + the tool/
    status/poll constants + the shared daemon constants moved out of
    supervise.py) and plan.py (SupervisePlan, a pure DTO).
  - bot_bottle/orchestrator/supervisor/ (orchestrator-only): queue.py (the
    queue/audit I/O wrappers + render_diff + sha256_hex) and supervise.py (the
    Supervise lifecycle that stages the DB via the store manager). Its __init__
    re-exports the neutral vocabulary so orchestrator-side callers import from
    one place.

The gateway now imports only bot_bottle.supervisor.types (never
bot_bottle.supervise), so the data plane holds no code dependency on the
orchestrator — it reaches the queue over the control-plane RPC. This removes
the circular import that moving queue_store under orchestrator introduced
(supervise -> orchestrator -> service -> supervise).

supervise_types.py -> supervisor/types.py; supervise.py deleted (split). Full
unit suite green (2251).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 14:24:10 -04:00

refactor(supervise): fold Supervise into the facade as Supervisor; drop dead flat-fallbacks ... 3e2cbcab88

Move the `Supervise` lifecycle out of its own `orchestrator/supervisor/
supervise.py` and into the package `__init__`, renaming the class to
`Supervisor`. Callers now import it from `bot_bottle.orchestrator.supervisor`
alongside the queue surface it belongs with.

Remove the dead `try/except ImportError` flat-import fallbacks from the
package-only store modules (db_store, audit_store, config_store, queue_store)
and image_cache. Those fallbacks existed for when the store files were
flat-copied into the gateway; post-PRD-0070 the data plane never opens the DB,
so these modules are only ever imported as part of the package. The two gateway
data-plane files that may still be loaded flat (egress_addon_core,
git_gate_render) keep their fallbacks.

Full unit suite green (2251).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 14:36:34 -04:00

refactor(supervise): drop obsolete queue helpers (archive_proposal, wait_for_response, audit_dir/path) ... 27a122e24b

Remove the supervise queue helpers that no live path uses anymore — they're
mechanisms the PRD-0070 / idempotent-poll migration superseded:

  - archive_proposal (+ QueueStore.archive_proposal): the poll stopped archiving
    on read (#469 review), so single-proposal archiving has no caller; decided
    proposals drop off the pending list via their response row and are reaped in
    bulk by archive_all_proposals on teardown/reconcile.
  - wait_for_response: the data plane polls the queue over the control-plane RPC
    now, so nothing block-waits on it.
  - audit_dir / audit_log_path: file-based audit paths, replaced by the SQLite
    AuditStore.

Also drops the tests that covered those obsolete helpers. Kept the prod-unused
read accessors list_pending_proposals and read_audit_entries: those aren't dead
mechanisms — they're the read side of live write paths that many tests use to
verify real queue/audit behavior.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 14:52:24 -04:00

refactor(supervise): make Supervisor a service class the orchestrator calls ... 8abccf7ffe

Turn the loose queue/audit functions in orchestrator/supervisor/queue.py into
methods on a concrete Supervisor service. The Orchestrator now owns an
injectable `self._supervisor` and calls `self._supervisor.write_proposal(...)`
etc., instead of module-level free functions — the supervise dependency is
explicit and mockable, and a Supervisor can be scoped to a `db_path` (defaults
to the host DB) so tests can point it at a temp database.

  - Supervisor (in the package __init__) drops the ABC and gains write_proposal,
    read_proposal, list_pending_proposals, list_all_pending_proposals,
    write_response, read_response, archive_all_proposals, write_audit_entry,
    read_audit_entries, plus the launch-time prepare().
  - render_diff / sha256_hex are pure and stateless, so they move to
    orchestrator/supervisor/util.py (re-exported from the facade) rather than
    becoming methods.
  - queue.py is deleted; service.py + the supervise tests call through a
    Supervisor instance.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 15:03:42 -04:00

refactor(supervise): hash the proposed file inside Proposal.new; sha256_hex → root util ... db6a151803

Move sha256_hex out of orchestrator/supervisor/util.py to the root
bot_bottle.util (it's a generic string hash, not supervise-specific), and have
Proposal.new take the proposed file contents and compute current_file_hash
itself instead of every caller passing sha256_hex(proposed_file).

Every call site set current_file_hash to the hash of the proposed file, and
nothing reads the field except the DB round-trip (schema + _row_to_proposal +
from_dict, all untouched), so folding the hash into the factory is
behavior-preserving and removes the boilerplate. Callers now pass just the
file; the orchestrator no longer imports sha256_hex at all.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 15:09:31 -04:00

refactor(util): make render_diff generic (caller supplies side titles); move to root util ... d41236c376

Generalize render_diff so it isn't supervise-flavored: the "(current)" /
"(proposed)" side titles are no longer baked in — callers pass `before_title`
and `after_label` (required). Move it from orchestrator/supervisor/util.py to
the base bot_bottle.util alongside sha256_hex, and delete the now-empty
supervisor/util.py.

The supervise callsite (Orchestrator.supervise_respond) assigns
before_title="current", after_label="proposed"; the facade no longer re-exports
render_diff (callers import from bot_bottle.util). Behavior at the supervise
callsite is unchanged.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 15:27:01 -04:00

refactor(docker): rename OrchestratorService -> DockerInfraService, move to backend/docker ... b96a8b44e0

OrchestratorService wasn't the orchestrator — it's the host-side lifecycle of
the docker infra *container* (the one that runs the Orchestrator). It read as
"the orchestrator as a service" and lived in orchestrator/lifecycle.py, while
its siblings (macOS MacosInfraService, Firecracker infra_vm) live under their
backend package. Rename it DockerInfraService and move it to
backend/docker/infra.py alongside the docker backend, with its docker-only
constants (INFRA_*/ORCHESTRATOR_* image + container names, daemon list, mount
paths).

orchestrator/lifecycle.py keeps only the backend-neutral pieces the other infra
services share — DEFAULT_PORT, DEFAULT_STARTUP_TIMEOUT_SECONDS,
OrchestratorStartError, source_hash — which macOS / firecracker / client still
import from there. backend/docker/infra.py imports those (backend -> orchestrator
is an allowed direction). Renamed the unit test to test_docker_infra.py.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 15:47:33 -04:00

refactor(backend): split the heavy backend/__init__ into base + selection, thin the package init ... 3e62f31d8b

backend/__init__.py eagerly imported the whole framework (manifest, egress,
git-gate, env, workspace, agent-provider), so importing ANY backend.* submodule
paid ~32 modules just to run the package init. Split it:

  - backend/base.py     — the abstract contract (BottleSpec, BottlePlan,
                          BottleCleanupPlan, ExecResult, ActiveAgent, Bottle,
                          BottleImages, BottleBackend). Carries the framework
                          imports; loaded only when a caller needs the contract.
  - backend/selection.py — backend registry / selection / enumeration
                          (get_bottle_backend, _get_backends, _auto_select_backend,
                          has_backend, known_backend_names, enumerate_active_agents).
                          Concrete backends still imported lazily inside.
  - backend/__init__.py  — thin: a __getattr__ that resolves the public names
                          from those submodules on first access (+ a TYPE_CHECKING
                          block for checkers). Existing `from bot_bottle.backend
                          import X` and patch.object call-sites keep working.

`import bot_bottle.backend` drops from 32 -> 2 bot_bottle modules. Repointed the
backend-selection test's patch targets to the `selection` module (the functions
reference each other there now; the FirecrackerBottleBackend class-method
patches are unchanged — same class object).

Note: backend.docker.util is still heavy because backend/docker/__init__ eagerly
imports DockerBottleBackend — slimming the three sub-package inits is the
follow-up that makes the leaves cheap.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 15:53:52 -04:00

refactor(backend): thin the docker/firecracker/macos-container package inits ... 923d44bc09

The three concrete-backend package inits eagerly imported their
`*BottleBackend` (which pulls the whole framework), so importing any leaf under
them (e.g. `backend.docker.util`) paid ~78 modules. Give each the same thin
`__getattr__` treatment as the top-level backend package: re-export the public
class(es) lazily on first access, with a TYPE_CHECKING block for checkers.

`import bot_bottle.backend.docker.util` drops from 78 -> 6 bot_bottle modules;
the docker/firecracker/macos-container packages themselves are ~3 each. All
`from bot_bottle.backend.docker import DockerBottleBackend` call-sites and the
lazy selection path keep working.

This is what makes backend leaves cheap — and unblocks moving docker_cmd into
backend/docker/util if we want.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 15:58:28 -04:00

refactor(docker): fold run_docker into backend/docker/util; drop the docker_cmd shim ... eab9d15130

docker_cmd.py existed as a top-level module solely so the orchestrator's docker
components could share run_docker without dragging the (then-heavy) backend
layer in. Now that backend/__init__ and backend/docker/__init__ are thin,
importing backend.docker.util costs 6 modules instead of 76, so the shim's whole
reason to exist is gone.

Move run_docker into backend/docker/util.py (where the other docker subprocess
primitives live) and delete docker_cmd.py. Backend siblings import it via
`from .util import run_docker`; the two docker-specific orchestrator modules
(docker_broker, rotate_ca) import `from ..backend.docker.util import run_docker`.
No import cycle (backend.docker.util pulls nothing from orchestrator);
orchestrator.__main__ stays lean at 28 modules. run_docker patch targets are
unchanged (tests patch it in the importing module).

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 16:04:19 -04:00

refactor(manifest): move Manifest/ManifestIndex to manifest.index; thin the package init ... 14c28946a7

manifest/__init__.py was a facade that eagerly imported every piece (agent,
bottle, egress, git, loader, schema, util) and defined the aggregate Manifest /
ManifestIndex, so importing a leaf like `manifest.util` paid 19 modules.

Move the aggregate model — Manifest, ManifestIndex, and the bottle-resolution
helpers — into manifest/index.py (which keeps the framework imports it needs),
and make __init__ a thin __getattr__ facade over the 12 __all__ names (mapping
each to its submodule), with a TYPE_CHECKING block. The package docstring is
preserved.

`import bot_bottle.manifest.util` drops 19 -> 3; the package itself is 2. All
`from bot_bottle.manifest import Manifest/ManifestIndex/ManifestError/…`
call-sites keep working via the lazy facade; no importer changes needed.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 16:28:02 -04:00

refactor(git-gate): make GitGate a service class in a git_gate package ... 450037b7e9

Turn the git_gate module into a package with GitGate as a concrete service class
(dropping the ABC), mirroring the Supervisor shape. The host-side git-gate
operations are now methods the backend drives:

  git_gate/
    __init__.py  — thin __getattr__ facade (keeps `from bot_bottle.git_gate
                   import …` working; render/hook names lazily forwarded to
                   gateway.git_gate_render)
    plan.py      — GitGatePlan (the launch DTO the backend contract references)
    service.py   — GitGate: prepare / provision_dynamic_keys /
                   revoke_provisioned_keys / preflight_host_keys
    provision.py — deploy-key lifecycle (was git_gate_provision.py)
    host_key.py  — host-key preflight (was git_gate_host_key.py)

The backend launch + base.py preflight now call GitGate() methods instead of the
free functions. The runtime rendering / in-gateway hook execution stay in
gateway/git_gate_render.py (data plane) — only the host-side service moved.

Because the facade forwards names lazily, the ~25 `from bot_bottle.git_gate
import GitGatePlan` call-sites are unchanged, and importing git_gate.plan (the
contract's dependency) no longer drags in the provisioning / forge-API code.

Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 16:44:06 -04:00

refactor(egress): make Egress a service class in an egress package ... a446551acb

Split the flat egress.py into an egress/ package: neutral DTOs
(EgressRoute, EgressPlan) in plan.py, and the concrete Egress service
class plus rendering/env helpers in service.py, behind a thin
__getattr__ facade so leaf imports stay cheap.

Egress drops ABC and becomes a concrete service the backends call:
resolve_token_values / agent_env_entries / prepare are now methods.
Backend launch paths (docker, firecracker, macos_container) call
Egress().<method>(...) instead of the module-level functions.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:00:54 -04:00

refactor(gateway): split data-plane files into egress/supervisor/git_gate services ... ce744a85c4

Group the gateway's data-plane modules into three service sub-packages
mirroring the host-side trio (bot_bottle.egress / .supervisor / .git_gate):

  gateway/egress/     addon_core, addon, dlp_config, dlp_detectors
  gateway/supervisor/ server            (was supervise_server)
  gateway/git_gate/   render, http_backend

Prefix-stripped filenames now that the package namespaces them; each
sub-package has a thin docstring __init__ (no eager imports, cheap leaf
loads). The two cross-cutting files stay at the gateway root:
policy_resolver (shared per-client lookup) and gateway_init, renamed to
bootstrap now that gateway/ already namespaces it.

Updated all importers (bot_bottle + tests), the in-VM/container `-m`
launch strings, the Dockerfile.gateway addon shim + ENTRYPOINT, and the
five gateway entries in scripts/critical-modules.txt. Full unit suite
green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:08:39 -04:00

refactor(gateway): rename bootstrap's _Supervisor to _DaemonManager ... 74dc984cf8

The PID-1 process manager in gateway/bootstrap.py was named _Supervisor,
which collided conceptually with the supervise permissions service
(Supervisor / gateway.supervisor). It manages the _DaemonSpec child set,
so _DaemonManager names what it does without the "supervise" echo.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:14:30 -04:00

refactor(gateway): move egress entrypoint into gateway/egress ... 82d02d2c4b

egress_entrypoint.sh is the mitmdump launcher for the egress data plane,
so it belongs with the gateway egress service, not at the package root.
Moved to bot_bottle/gateway/egress/entrypoint.sh (prefix stripped now
that the package namespaces it).

Only the Dockerfile.gateway COPY source changes; the runtime target
(/app/egress-entrypoint.sh, referenced by bootstrap's egress DaemonSpec)
is unchanged, and the infra images inherit it via FROM gateway. Updated
the test that resolves the script path and the infra-artifact docstring
example. Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:25:28 -04:00

refactor(cli): group subcommand handlers under cli/commands/ ... 50a67c04bd

Move the eleven per-command modules (backend, cleanup, commit, edit,
info, init, list, login, resume, start, supervise) into a new
bot_bottle/cli/commands/ package, leaving the dispatcher (__init__),
entrypoint (__main__), and shared helpers (_common, tui) at the cli
root. Makes the command surface obvious at a glance and separates
handlers from the plumbing that registers them.

Updated the dispatcher's COMMANDS imports to .commands.*, bumped the
moved files' relative-import depths (.._common, .. import tui, sibling
.start unchanged), and repointed test references to
bot_bottle.cli.commands.*. Full unit suite green (2243); CLI dispatch
verified via `python -m bot_bottle.cli --help`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:31:19 -04:00

refactor(cli): trim _common to a PROG constants leaf; inline os.getcwd() ... a21f2358c6

_common had rotted into a junk drawer. Trim it to what actually
justifies a shared leaf module and rename for legibility:

  * REPO_DIR: deleted (dead — nothing imported it).
  * read_tty_line: dropped the pointless re-export; cleanup/start/init
    now import it straight from bot_bottle.util.
  * USER_CWD: deleted. It captured os.getcwd() at import time, but
    nothing chdirs and no test patched it, so it was equivalent to a
    live os.getcwd() — inlined at the six call sites.
  * PROG: kept, now the sole member. It's still a leaf (both the
    dispatcher and the commands it imports need it) so it can't move
    into __init__ without a circular import.

_common.py -> constants.py. Full unit suite green (2243); CLI dispatch
verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:40:08 -04:00

refactor(cli): make help a first-class command; drop usage() from dispatcher ... 3ccd308613

Move the top-level usage/command-list text out of the dispatcher into
commands/help.py as cmd_help, registered in COMMANDS so `bb help` now
works as a real command. The dispatcher's -h/--help, no-args, and
unknown-command fallbacks call cmd_help() for the text while keeping
their exit codes (0 for help/-h, 2 for bare no-args, Die for unknown).

Added `help` to NO_MIGRATION_COMMANDS so it prints without a migrated
DB, and listed it in the command summary. Full unit suite green (2243);
dispatch + migration-gate tests pass, `bb help` verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:52:56 -04:00

refactor(cli): registry in commands/__init__, dispatcher main() in __main__ ... b276227cbb

Split the cli package's two responsibilities out of __init__:

  * commands/__init__.py now assembles the COMMANDS registry and
    NO_MIGRATION_COMMANDS from the per-command modules — the command
    surface lives entirely under commands/.
  * __main__.py now owns main() (dispatch + migration gate + exit-code
    mapping) alongside the runnable entry guard.

cli/__init__.py shrinks to a shim that re-exports main / COMMANDS /
NO_MIGRATION_COMMANDS, so bot_bottle.cli.main (repo-root cli.py entry)
and the tests' bot_bottle.cli.COMMANDS keep working unchanged. Verified
`python -m bot_bottle.cli` (runpy) and `cli.py` both dispatch; full unit
suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 17:57:58 -04:00

refactor(orchestrator): rename control_plane module to server ... 4166057abc

orchestrator/control_plane.py -> orchestrator/server.py. Within the
orchestrator package the "control_plane" filename stutters (the
orchestrator *is* the control plane), and `orchestrator.server` reads as
"the orchestrator's HTTP server", pairing with service.py (domain logic)
and matching gateway/supervisor/server.py.

Scope is the module name only. The class ControlPlaneServer, the
CONTROL_AUTH_HEADER constant, and the "control plane" architectural term
(CONTROL_PLANE_PORT, host_control_plane_token, control_plane_url, …) are
deliberately unchanged — that's the load-bearing control-plane/data-plane
distinction. Test file renamed to match; full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 18:17:52 -04:00

refactor: unify component naming — control_plane/control_auth -> orchestrator ... ca1d341d4f

The codebase used "control plane" both as an architectural role term AND
as an identifier alias for the orchestrator component, producing
duplicate names for one thing (control_plane_url vs orchestrator_url,
CONTROL_PLANE_PORT, host_control_plane_token, …). Going forward the
concrete component is always named for what it is — Gateway or
Orchestrator — and the plane vocabulary is reserved for prose (module
descriptions, the security argument).

Renamed (identifiers + the in-repo env/wire/file string values, all
setters/getters are in this repo so the change is atomic):

  ControlPlaneServer            -> OrchestratorServer
  control_plane_url             -> orchestrator_url
  probe_control_plane_url       -> probe_orchestrator_url
  host_control_plane_token      -> host_orchestrator_token
  CONTROL_PLANE_PORT            -> ORCHESTRATOR_PORT
  CONTROL_PLANE_TOKEN_ENV/FILE  -> ORCHESTRATOR_TOKEN_ENV/FILENAME
  BOT_BOTTLE_CONTROL_PLANE_TOKEN-> BOT_BOTTLE_ORCHESTRATOR_TOKEN
  control-plane-token (file)    -> orchestrator-token

  control_auth (module)         -> orchestrator_auth  (stays top-level;
                                   the gateway imports it and must not
                                   import the orchestrator/ package)
  CONTROL_AUTH_HEADER           -> ORCHESTRATOR_AUTH_HEADER
  x-bot-bottle-control-auth     -> x-bot-bottle-orchestrator-auth
  CONTROL_AUTH_JWT_ENV          -> ORCHESTRATOR_AUTH_JWT_ENV
  BOT_BOTTLE_CONTROL_AUTH_JWT   -> BOT_BOTTLE_ORCHESTRATOR_AUTH_JWT
  _control_auth_headers         -> _orchestrator_auth_headers

Prose plane-terms ("control plane", "data plane") are preserved,
including the test name test_data_plane_daemons_get_jwt_not_key (it
names the security invariant). Gateway and orchestrator verified to
agree on the renamed wire header; full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 2 commits 2026-07-24 18:24:24 -04:00

refactor(orchestrator): collapse supervisor package into supervisor.py ... cb3a74edb0

orchestrator/supervisor/ held only __init__.py, so the package no longer
earns its own directory. Moved it to orchestrator/supervisor.py and
removed the dir. External imports are unchanged (orchestrator.supervisor
resolves identically as a module); only the file's own relative-import
depths shift up one level. Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

refactor(orchestrator): move registry into store/registry_store ... 3efb014ace

registry.py is a SQLite-backed store (its class is already RegistryStore),
so it belongs with the other orchestrator-owned stores rather than at the
package root. Moved orchestrator/registry.py -> orchestrator/store/
registry_store.py, joining queue_store / secret_store / config_store.

Repointed the in-package importers (.registry -> .store.registry_store)
and the tests, bumped the moved file's relative-import depths, renamed
the test file to match, and listed it in the store package docstring.
Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis added 1 commit 2026-07-24 18:54:47 -04:00

refactor: make the two remaining heavy __init__ modules lazy ... e2740842a0

Audit of package __init__ import cost turned up two eager ones:

orchestrator/__init__.py (27 -> 2): was pure `from .X import Y`
re-exports (registry_store, broker, docker_broker, gateway, service,
server). Because importing any submodule runs the parent __init__, the
CLI migration gate (orchestrator.store.store_manager, hit on every
command) transitively dragged backend.docker, docker_broker, server, and
http.server. Converted to the same lazy __getattr__ + _LAZY facade used
by manifest/backend/egress/git_gate; the re-export API is unchanged and
the docker/http drag is gone.

cli/commands/__init__.py (78 -> 23): the registry imported all twelve
handlers to build COMMANDS, so importing one command pulled all of them
plus their deps. COMMANDS now maps each name to a thin lazy wrapper that
imports its handler's module on first dispatch. Values stay callable, so
the dispatcher and the patch.dict dispatch tests are unchanged; a CLI run
now loads only the one command it dispatches. The remaining 23 is the
dispatcher's own baseline (store_manager migration gate + help + log).

Full unit suite green (2243); `bb help` + dispatch/migration-gate tests
verified.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis-claude referenced this pull request 2026-07-24 19:06:21 -04:00

Raise code quality toward all-5 rubric through focused cleanup and guardrails #444

didericis added 1 commit 2026-07-24 19:11:40 -04:00

fix(types): satisfy pyright strict on the refactored facades ... bef45348f5

Two facade issues from the reorg:

  * orchestrator/__init__.py used `__all__ = list(_LAZY)`, which pyright
    strict can't analyze (reportUnsupportedDunderAll) — and because it
    isn't a static literal, the TYPE_CHECKING re-export imports read as
    unused. Replaced with the explicit literal list the other lazy
    facades already use.
  * manifest/index.py imported six piece types (ManifestAgentProvider,
    EGRESS_AUTH_SCHEMES, ManifestEgressConfig/Route, ManifestGitEntry,
    ManifestKeyConfig) it never referenced — the package facade re-exports
    those straight from their submodules, so index.py needn't import them.
    Dropped the dead imports.

pyright: 0 errors (was 24). Full unit suite green (2243).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

didericis merged commit bef45348f5 into fix/db-off-data-plane-469 2026-07-24 19:24:56 -04:00

didericis deleted branch refactor/repo-reorg 2026-07-24 19:24:57 -04:00

Sign in to join this conversation.

Reviewers

No Reviewers

Labels

Clear labels

Compat/Breaking

Breaking change that won't be backward compatible

Kind/Bug

Something is not working

Kind/Documentation

Documentation changes

Kind/Enhancement

Improve existing functionality

Kind/Feature

New functionality

Kind/Security

This is security issue

Kind/Testing

Issue or pull request related to testing

Priority

Critical

1

The priority is critical

Priority

High

2

The priority is high

Priority

Low

4

The priority is low

Priority

Medium

3

The priority is medium

Reviewed

Confirmed

1

Issue has been confirmed

Reviewed

Duplicate

2

This issue or pull request already exists

Reviewed

Invalid

3

Invalid issue

Reviewed

Won't Fix

3

This issue won't be fixed

Status

Abandoned

3

Somebody has started to work on this but abandoned work

Status

Blocked

1

Something is blocking this issue or pull request

Status

Need More Info

2

Feedback is required to reproduce issue or to continue work

Status/Needs Triage

Awaiting initial classification

No Label

Milestone

No items

No Milestone

Assignees

Clear assignees

didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis

No Assignees

2 Participants

Notifications

Subscribe

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#477