didericis/bot-bottle
1
0
Fork 0
Code Issues 37 Pull Requests 4 Actions Packages Releases Wiki Activity

Run containers inside a bottle (nested_containers) #456

Open
didericis-claude wants to merge 13 commits from prd-nested-containers into main
pull from: prd-nested-containers
merge into: didericis:main
Conversation 5 Commits 13 Files Changed 24
+1584 -9
didericis-claude commented 2026-07-21 15:39:34 -04:00
Collaborator
Copy Link
Copy Source

Closes #392.

PRD: https://gitea.dideric.is/didericis/bot-bottle/src/branch/prd-nested-containers/docs/prds/prd-new-nested-containers.md

Summary

Adds the nested_containers bottle flag. On the macos-container backend it starts a guest-local rootless podman service inside the bottle and exposes its Docker-compatible API socket, so the agent keeps typing docker and docker compose. No host Docker socket is mounted and the guest gains no capabilities.

This ports spike/rootless-docker-macos onto main and resolves the issue's open questions:

  • Keep the flag, rename it. Podman needs no privilege grant, but the flag still gates ~100MB of derived image, a resident service per bottle, and relaxed modes on /dev/fuse and /dev/net/tun. docker_access was the wrong name — it implies Docker and grants access to nothing on the host.
  • Backend scope: macOS only for now. docker and firecracker reject nested_containers: true in the shared prepare template rather than silently ignoring it. Mounting the host socket stays off the table on every backend.
  • Podman is no longer in the derived layer — every built-in agent image ships it since the trixie bump (#451), so the layer is just the Docker CLI, the compose plugin, fuse-overlayfs, slirp4netns, and uidmap.

Nested containers are not an isolation layer: the single-UID mapping means root inside one is the agent user outside it. Documented in the README, the PRD, and the module docstring.

Schema

nested_containers: true
egress:
  routes:
    - host: registry-1.docker.io
      preserve_auth: true
      dlp: { outbound_detectors: false, inbound_detectors: false }
    - host: auth.docker.io
      preserve_auth: true
    - host: ghcr.io
      preserve_auth: true
      dlp: { outbound_detectors: false, inbound_detectors: false }

preserve_auth (#453) is required for Docker Hub and GHCR: both use a client-fetched per-scope bearer token the proxy would otherwise strip. DLP off on registry routes sidesteps #455 (large-pull OOM) and costs nothing — the scanned bodies are compressed layer blobs.

Merge rule

nested_containers follows supervise: later bottle replaces earlier in both extends: chains and runtime multi-bottle composition.

Verification

Unit: 2106 pass, pylint 9.83, pyright clean.

Live acceptance criteria from the issue thread run on the macOS host and posted as a follow-up comment before merge.

Closes #392. PRD: https://gitea.dideric.is/didericis/bot-bottle/src/branch/prd-nested-containers/docs/prds/prd-new-nested-containers.md ## Summary Adds the `nested_containers` bottle flag. On the `macos-container` backend it starts a guest-local **rootless podman** service inside the bottle and exposes its Docker-compatible API socket, so the agent keeps typing `docker` and `docker compose`. No host Docker socket is mounted and the guest gains no capabilities. This ports `spike/rootless-docker-macos` onto main and resolves the issue's open questions: - **Keep the flag, rename it.** Podman needs no privilege grant, but the flag still gates ~100MB of derived image, a resident service per bottle, and relaxed modes on `/dev/fuse` and `/dev/net/tun`. `docker_access` was the wrong name — it implies Docker and grants access to nothing on the host. - **Backend scope: macOS only for now.** `docker` and `firecracker` reject `nested_containers: true` in the shared `prepare` template rather than silently ignoring it. Mounting the host socket stays off the table on every backend. - **Podman is no longer in the derived layer** — every built-in agent image ships it since the trixie bump (#451), so the layer is just the Docker CLI, the compose plugin, `fuse-overlayfs`, `slirp4netns`, and `uidmap`. Nested containers are **not** an isolation layer: the single-UID mapping means `root` inside one is the agent user outside it. Documented in the README, the PRD, and the module docstring. ## Schema ```yaml nested_containers: true egress: routes: - host: registry-1.docker.io preserve_auth: true dlp: { outbound_detectors: false, inbound_detectors: false } - host: auth.docker.io preserve_auth: true - host: ghcr.io preserve_auth: true dlp: { outbound_detectors: false, inbound_detectors: false } ``` `preserve_auth` (#453) is required for Docker Hub and GHCR: both use a client-fetched per-scope bearer token the proxy would otherwise strip. DLP off on registry routes sidesteps #455 (large-pull OOM) and costs nothing — the scanned bodies are compressed layer blobs. ## Merge rule `nested_containers` follows `supervise`: later bottle replaces earlier in both `extends:` chains and runtime multi-bottle composition. ## Verification Unit: 2106 pass, pylint 9.83, pyright clean. Live acceptance criteria from the issue thread run on the macOS host and posted as a follow-up comment before merge.
didericis-claude commented 2026-07-21 15:50:50 -04:00
Author
Collaborator
Copy Link
Copy Source

Live verification on the macOS host — mechanism works, acceptance criteria do not pass

Ran on macOS 26 / Apple Container 1.0.0 with nested_containers: true and the registry routes from the issue comment.

What passed

  • Derived image bot-bottle-claude:latest-nested-containers builds, including the docker:28-cli layer.
  • guest-local container engine is ready — the rootless podman bootstrap succeeds inside the bottle with zero added capabilities, on two independent runs.
  • Versions in-bottle: Docker client 28.5.2, compose v2.40.3, server Podman 5.4.2 on Debian 13 via the Docker-compat API (1.41). The trixie bump (#451) did land, so the bookworm 4.3.1 exit-code swallow is no longer in play.

What failed

Check Result
1. Engine versions OK
2. Registry reach All three failed
3. Exit-code passthrough Blocked — no image to run
4. DNS from a nested container Blocked — no image to run
5. Clone with submodules Partial fail — main repo OK, all 4 submodules failed
6. docker compose integration tests Failed — 0 tests ran

Registry reach is the blocker for 2/3/4.

  • quay.io/podman/hello:latest → egress: host 'cdn01.quay.io' is not in the bottle's egress.routes allowlist. The manifest host is routed; its blob CDN is not. Fixable by routing cdn0*.quay.io.
  • alpine:3.20 (Docker Hub) → unauthorized: authentication required
  • ghcr.io/linuxcontainers/alpine:latest → unauthorized

The last two are the important finding: preserve_auth: true on registry-1.docker.io + auth.docker.io + ghcr.io was not sufficient. The registry-auth strip called out in #392's "Blocked on" list is still blocking, and #453 alone does not close it. Needs a look at the gateway log (the run had log: 2) to see whether the route flag applied and where the bearer went.

Submodule clone (check 5) is an unrelated pre-existing gap. bot-bottle-platform's .gitmodules records ssh://git@gitea.dideric.is:30009/..., while the git-gate insteadOf rules written into ~/.gitconfig only cover the ssh://git@100.78.141.42:30009/... IP form. No rewrite fires, git tries a direct SSH connection, and DNS for gitea.dideric.is fails inside the bottle — correctly, since it isn't a routed host. Check 6 then fails downstream of it (empty console submodule → nothing to build).

Reading

The nested_containers mechanism in this PR is verified working. Everything that failed is in the surrounding plumbing — egress routes for registry CDNs, the registry-auth strip, and git-gate submodule URL rewrites — none of which this diff touches.

So: merging this is safe and it is genuinely useful for locally-built images, but it does not close #392 on its own. The issue's own acceptance list needs the registry-auth decision resolved first.

## Live verification on the macOS host — mechanism works, acceptance criteria do **not** pass Ran on macOS 26 / Apple Container 1.0.0 with `nested_containers: true` and the registry routes from the [issue comment](https://gitea.dideric.is/didericis/bot-bottle/issues/392#issuecomment-4886). ### What passed - Derived image `bot-bottle-claude:latest-nested-containers` builds, including the `docker:28-cli` layer. - `guest-local container engine is ready` — the rootless podman bootstrap succeeds inside the bottle with **zero added capabilities**, on two independent runs. - Versions in-bottle: Docker client **28.5.2**, compose **v2.40.3**, server **Podman 5.4.2** on Debian 13 via the Docker-compat API (1.41). The trixie bump (#451) did land, so the bookworm 4.3.1 exit-code swallow is no longer in play. ### What failed | Check | Result | |---|---| | 1. Engine versions | **OK** | | 2. Registry reach | **All three failed** | | 3. Exit-code passthrough | **Blocked** — no image to run | | 4. DNS from a nested container | **Blocked** — no image to run | | 5. Clone with submodules | **Partial fail** — main repo OK, all 4 submodules failed | | 6. `docker compose` integration tests | **Failed** — 0 tests ran | **Registry reach is the blocker for 2/3/4.** - `quay.io/podman/hello:latest` → `egress: host 'cdn01.quay.io' is not in the bottle's egress.routes allowlist`. The manifest host is routed; its **blob CDN is not**. Fixable by routing `cdn0*.quay.io`. - `alpine:3.20` (Docker Hub) → `unauthorized: authentication required` - `ghcr.io/linuxcontainers/alpine:latest` → `unauthorized` The last two are the important finding: **`preserve_auth: true` on `registry-1.docker.io` + `auth.docker.io` + `ghcr.io` was not sufficient.** The registry-auth strip called out in #392's "Blocked on" list is still blocking, and #453 alone does not close it. Needs a look at the gateway log (the run had `log: 2`) to see whether the route flag applied and where the bearer went. **Submodule clone (check 5) is an unrelated pre-existing gap.** `bot-bottle-platform`'s `.gitmodules` records `ssh://git@gitea.dideric.is:30009/...`, while the git-gate `insteadOf` rules written into `~/.gitconfig` only cover the `ssh://git@100.78.141.42:30009/...` IP form. No rewrite fires, git tries a direct SSH connection, and DNS for `gitea.dideric.is` fails inside the bottle — correctly, since it isn't a routed host. Check 6 then fails downstream of it (empty `console` submodule → nothing to build). ### Reading The `nested_containers` mechanism in this PR is verified working. Everything that failed is in the surrounding plumbing — egress routes for registry CDNs, the registry-auth strip, and git-gate submodule URL rewrites — none of which this diff touches. So: merging this is safe and it is genuinely useful for locally-built images, but it does **not** close #392 on its own. The issue's own acceptance list needs the registry-auth decision resolved first.
didericis-claude commented 2026-07-21 21:01:59 -04:00
Author
Collaborator
Copy Link
Copy Source

Live verification: green, on macOS 26 / Apple Container 1.0.0 / podman 5.4.2

Every check below ran with plain docker, no extra flags (no --add-host, -e, or -v).

Check Result
docker pull from Docker Hub PASS
docker pull from GHCR PASS
docker pull from quay.io PASS
docker run exit-code propagation PASS (exit=7)
docker compose up / logs / down PASS
Compose published port, curl'd from the bottle PASS (200)
curl https://quay.io from inside a nested container PASS (200)
curl https://pypi.org/simple/ from inside a nested container PASS (200)
curl https://example.com from inside a nested container PASS — 403, allowlist enforced
Clone bot-bottle-platform with submodules PASS (all 6, incl. nested)

The egress boundary applies to nested containers, not just the bottle — that 403 comes from the proxy, and the gateway log shows the request arriving there.

What the verification turned up

It took twelve live runs, because the spike was written against bookworm's podman 4.3.1 and #451's trixie bump moved it to podman 5 — a different networking stack. Each missing piece failed at a different layer while image pulls kept working, which made every one of them read as a compat-API bug:

  • passt missing — podman 5 defaults to pasta; nothing started at all.
  • nft / aardvark-dns missing — containers created but never started; then DNS inside them failed. --no-install-recommends on the base image's podman means none of the networking stack arrives with it. This is the likeliest explanation for the "DNS inside nested containers fails entirely" note in the issue — a missing package, not a design gap.
  • preserve_auth never serialized. _route_to_yaml_fields parsed the flag off the manifest and the addon honored it, but the renderer in between dropped it. Docker Hub fetched a valid bearer, the proxy stripped it, and the registry answered the anonymous request with a fresh challenge — indistinguishable from a credential problem. This means #453's flag has never worked end to end; it is not specific to nested containers.
  • Blob CDNs need their own routes. Auth succeeds, the manifest fetches, then the pull 403s on production.cloudfront.docker.com / pkg-containers.githubusercontent.com / cdn0*.quay.io.
  • Reaching the proxy took four attempts because podman applies proxy settings at several layers and the last writer wins. containers.conf env, http_proxy=false, and the service environment all looked right on disk and changed nothing inside a container. The decider is ~/.docker/config.json's proxies block — client-side, copied in by the Docker CLI, beating everything the service does. containers.conf hosts_file has the same blind spot: native podman run only.

Not fixed here

  • Alpine's BusyBox wget drops the connection after TLS interception ("error getting response"). The proxy logs the decrypted request and curl on the same host returns 200, so this is a BusyBox client limitation. Documented in the README.
  • #455 (large-pull OOM) is sidestepped by disabling DLP on registry routes, not solved.

DNS open question, resolved

Public DNS inside a nested container fails by design — everything egresses through the proxy. The real breakage was reaching the proxy: it is named bot-bottle-gateway, which resolves only through the bottle's /etc/hosts, and a nested container gets its own. The name is now resolved in the bottle and the address substituted into the proxy URL.

## Live verification: green, on macOS 26 / Apple Container 1.0.0 / podman 5.4.2 Every check below ran with plain `docker`, **no extra flags** (no `--add-host`, `-e`, or `-v`). | Check | Result | |---|---| | `docker pull` from Docker Hub | **PASS** | | `docker pull` from GHCR | **PASS** | | `docker pull` from quay.io | **PASS** | | `docker run` exit-code propagation | **PASS** (`exit=7`) | | `docker compose` up / logs / down | **PASS** | | Compose published port, curl'd from the bottle | **PASS** (200) | | `curl https://quay.io` from inside a nested container | **PASS** (200) | | `curl https://pypi.org/simple/` from inside a nested container | **PASS** (200) | | `curl https://example.com` from inside a nested container | **PASS** — **403**, allowlist enforced | | Clone `bot-bottle-platform` with submodules | **PASS** (all 6, incl. nested) | **The egress boundary applies to nested containers, not just the bottle** — that 403 comes from the proxy, and the gateway log shows the request arriving there. ### What the verification turned up It took twelve live runs, because the spike was written against bookworm's podman 4.3.1 and #451's trixie bump moved it to podman 5 — a different networking stack. Each missing piece failed at a different layer while image pulls kept working, which made every one of them read as a compat-API bug: - **`passt` missing** — podman 5 defaults to pasta; nothing started at all. - **`nft` / `aardvark-dns` missing** — containers created but never started; then DNS inside them failed. `--no-install-recommends` on the base image's podman means none of the networking stack arrives with it. **This is the likeliest explanation for the "DNS inside nested containers fails entirely" note in the issue — a missing package, not a design gap.** - **`preserve_auth` never serialized.** `_route_to_yaml_fields` parsed the flag off the manifest and the addon honored it, but the renderer in between dropped it. Docker Hub fetched a valid bearer, the proxy stripped it, and the registry answered the anonymous request with a fresh challenge — indistinguishable from a credential problem. **This means #453's flag has never worked end to end; it is not specific to nested containers.** - **Blob CDNs need their own routes.** Auth succeeds, the manifest fetches, then the pull 403s on `production.cloudfront.docker.com` / `pkg-containers.githubusercontent.com` / `cdn0*.quay.io`. - **Reaching the proxy took four attempts** because podman applies proxy settings at several layers and the last writer wins. `containers.conf` `env`, `http_proxy=false`, and the service environment all looked right on disk and changed nothing inside a container. The decider is `~/.docker/config.json`'s proxies block — client-side, copied in by the Docker CLI, beating everything the service does. `containers.conf` `hosts_file` has the same blind spot: native `podman run` only. ### Not fixed here - Alpine's BusyBox `wget` drops the connection after TLS interception ("error getting response"). The proxy logs the decrypted request and `curl` on the same host returns 200, so this is a BusyBox client limitation. Documented in the README. - #455 (large-pull OOM) is sidestepped by disabling DLP on registry routes, not solved. ### DNS open question, resolved Public DNS inside a nested container fails **by design** — everything egresses through the proxy. The real breakage was reaching the proxy: it is named `bot-bottle-gateway`, which resolves only through the bottle's `/etc/hosts`, and a nested container gets its own. The name is now resolved in the bottle and the address substituted into the proxy URL.
didericis force-pushed prd-nested-containers from 93cf0c8dd9 to 913bcab6a0 2026-07-21 21:25:35 -04:00 Compare
didericis reviewed 2026-07-21 21:30:06 -04:00
didericis reviewed 2026-07-21 21:33:39 -04:00
bot_bottle/backend/macos_container/nested_containers.py Outdated
@@ -0,0 +23,4 @@
Nested containers run *within* the bottle boundary, not inside a new one: the
single-UID mapping means `root` in a nested container is the agent user
outside it. This is for build and test workloads, not for sandboxing
untrusted code.
didericis commented 2026-07-21 21:33:39 -04:00
Owner
Copy Link
Copy Source

@didericis-claude This is kind of a silly thing to emphasize, since we're already in a safe environment. I think you emphasized this in the PRD and the related issues as well. Please remove this.

@didericis-claude This is kind of a silly thing to emphasize, since we're already in a safe environment. I think you emphasized this in the PRD and the related issues as well. Please remove this.
didericis reviewed 2026-07-21 21:41:04 -04:00
bot_bottle/backend/macos_container/nested_containers.py Outdated
@@ -0,0 +74,4 @@
with tempfile.TemporaryDirectory(prefix="bot-bottle-nested-containers.") as tmp:
context = Path(tmp)
shutil.copy2(init_script, context / "nested-containers-init.sh")
(context / "Dockerfile").write_text(
didericis commented 2026-07-21 21:41:04 -04:00
Owner
Copy Link
Copy Source

@didericis-claude Add a comment calling out an intended refactor here: we'll want to use a docker layer abstraction we build as a part of #394 for this in the near future.

Also, since we're already modifying the agent dockerfile here, should move the default podman dependencies we added to the build in agent images out of the agent images and into this.

@didericis-claude Add a comment calling out an intended refactor here: we'll want to use a docker layer abstraction we build as a part of https://gitea.dideric.is/didericis/bot-bottle/issues/394 for this in the near future. Also, since we're already modifying the agent dockerfile here, should move the default podman dependencies we added to the build in agent images out of the agent images and into this.
didericis added 13 commits 2026-07-21 22:54:32 -04:00
feat(macos): run containers inside a bottle via guest-local podman d0b595828f 
Adds the `nested_containers` bottle flag. On the macOS backend it starts
a rootless podman service inside the bottle and exposes its
Docker-compatible API socket, so the agent still runs `docker` and
`docker compose`. No host daemon socket is mounted and the guest gains
no capabilities; backends that cannot run a guest-local engine reject
the flag in the shared prepare template rather than ignoring it.

Podman rather than rootless Docker because Apple Container's capability
bounding set omits CAP_SYS_ADMIN, which the kernel requires to write a
multi-range uid_map via newuidmap. With no subordinate UID range podman
falls back to a single-UID self-mapping an unprivileged process may
write itself, so the image build strips /etc/subuid and /etc/subgid
entries rather than adding them.

That mapping is also why nested containers are not an isolation layer:
root inside one is the agent user outside it. They are a build/test
convenience; the bottle remains the boundary.

Ports the spike branch onto main, renaming docker_access — it implied
Docker and granted access to nothing on the host — and drops podman
from the derived layer now that every built-in image ships it (#451).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(manifest): OR nested_containers across composed bottles fa9fed4194 
merge_bottles_runtime sees resolved bottles, where a bottle that never
mentioned the key is indistinguishable from one that set it false. With
"later replaces" semantics, `--bottle with-containers --bottle claude-dev`
silently dropped the capability — the second bottle's default clobbered
it. OR instead: composing bottles adds capabilities, it does not remove
them. The file-based extends path still sees raw keys, so an explicit
`nested_containers: false` in a child continues to turn it back off.

Also surface the flag in the launch summary, where its absence was the
only reason the clobber went unnoticed.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(macos): shorten the podman runtime dir to fit sun_path b0f012b8e6 
podman derives conmon's attach socket as
`$XDG_RUNTIME_DIR/libpod/tmp/socket/<64-hex-id>/attach`. With the
descriptive `/tmp/bot-bottle-podman-run` that path is 116 bytes, past
the 108-byte sun_path limit, so every attached `docker run` failed with
"unable to upgrade to tcp, received 500" — while image pulls and the
service itself worked, which is what made it look like a compat-API bug
rather than a path-length one.

Found on the macOS host: quay.io pulled fine and then no container
could be started. Test pins the budget so a more readable name cannot
silently reintroduce it.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(macos): install passt for podman 5 rootless networking 220620bfcc 
podman 5 uses pasta as the default rootless network helper; podman 4
used slirp4netns. The spike was written against bookworm's podman 4.3.1,
so its package list never included passt — and the trixie bump (#451)
changed the default out from under it. Every nested container failed to
start with "could not find pasta", while image pulls and the service
itself worked, which disguised a missing dependency as a compat-API bug.

The bootstrap now checks for pasta up front so this fails with a clear
message instead of surfacing at the first `docker run`.

Note the sun_path fix in a12d2191 did NOT cause or cure this; that path
was over the 108-byte limit independently and would have bitten as soon
as attach was reached.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(macos): install podman 5's full networking stack 3c92e79775 
netavark shells out to `nft` for the bridge network every compose file
expects, and aardvark-dns serves name resolution inside nested
containers. Neither arrives with the base image's
`--no-install-recommends` podman, and each fails at a different and
misleading layer: without nft containers are created but never start;
without aardvark-dns DNS inside a container fails while the engine, the
pulls, and the bridge all look healthy. That last one is the likeliest
explanation for the "DNS inside nested containers fails entirely" note
in #392 — a missing package, not a design gap.

Verified on the macOS host: with passt alone, `docker run` reached
netavark and died on "unable to execute nft".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
feat(macos): give nested containers working egress 82cf9bab5a 
Answers the last open question in #392: DNS from inside a nested
container was not a phantom, but it was also not a DNS problem. Public
resolution fails there by design — everything egresses through the
proxy — and the actual breakage was two missing pieces:

- The proxy URL names `bot-bottle-gateway`, which resolves only through
  the bottle's own /etc/hosts. Containers got their own hosts file, so
  they failed at "Could not resolve proxy", which reads like broken DNS.
  base_hosts_file propagates the bottle's entries.
- The gateway TLS-intercepts, so a container that does not trust the
  bottle's CA bundle fails with "unable to get local issuer
  certificate". The bundle is now mounted read-only and the usual env
  vars point at it, which covers curl, wget, python, and node without
  distro-specific trust commands.

Verified on the macOS host by reproducing each failure and confirming
that these three conditions applied by hand produced HTTP 200 from
inside a nested container. podman already forwards the proxy env, so
that needed nothing.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(egress): emit preserve_auth into the generated routes 96f5be48a6 
The flag from #453 was parsed off the manifest and honored by the
addon, but `_route_to_yaml_fields` never wrote it, so it was dropped on
the way to the proxy and has never worked end to end. Every registry
route silently kept the unconditional Authorization strip.

That is the whole of the Docker Hub / GHCR "unauthorized" blocker in
#392: podman fetched a valid 2658-byte bearer from auth.docker.io, the
proxy stripped it, and registry-1.docker.io answered the resulting
anonymous request with a fresh WWW-Authenticate challenge — which reads
exactly like a credential problem rather than a serializer gap.

Found by diffing a bottle's rendered routes.yaml against its manifest:
preserve_auth was in the manifest and absent from the output.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(macos): reach the egress proxy by address in nested containers ca8b2a9f2c 
podman's containers.conf `hosts_file` turns out to apply only to native
`podman run`; the Docker-compatible API ignores it, and the agent types
`docker`. (`base_hosts_file`, used before this, is a no-op in both
paths on podman 5.4.2.) So no config key can put the gateway name in a
nested container's /etc/hosts.

The name is therefore resolved in the bottle, where /etc/hosts does
carry it, and the address goes into the proxy URL handed to nested
containers. NO_PROXY keeps the name, since that is matched against what
a client asks for.

Verified on the macOS host: with the gateway reachable, a nested
container gets 200 from an allowlisted host and 403 from one that is
not — the egress boundary applies inside nested containers too.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(macos): stop podman re-injecting the gateway name into containers 7a9628fc03 
The address-bearing proxy URLs from 892bfc16 were written correctly but
never took effect: podman copies the host's proxy environment into every
container by default, and that copy overrides containers.conf `env`,
restoring the `bot-bottle-gateway` name a nested container cannot
resolve. Verified on the macOS host — containers still reported
`wget: bad address 'bot-bottle-gateway:9099'` with the addresses in
place.

Setting http_proxy=false disables only podman's own passthrough; the
proxy vars still reach containers, by address, from the env list.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(macos): substitute the gateway address before the service starts 854a8956ad 
Third attempt at the same symptom, and the first at the right layer.
podman's Docker-compatible API injects proxy settings from the *daemon*
environment after containers.conf is applied, so neither the env list
(892bfc16) nor http_proxy=false (21c144ae) could override it — the same
compat-path blind spot already seen with hosts_file. Nested containers
kept reporting `wget: bad address 'bot-bottle-gateway:9099'` through
both.

Substituting the resolved address into the service's own environment,
before `podman system service` starts, is the one place it sticks.
NO_PROXY keeps the name, which is matched against what a client asks
for.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
fix(macos): put the gateway address in the Docker CLI proxy config dba48706de 
This is the layer that actually decides it, and it was in the script
from the start: ~/.docker/config.json's proxies block is what the Docker
CLI copies into every container it starts. Being client-side it beats
the podman service outright, which is why containers.conf `env`,
http_proxy=false, and the service environment each looked correct on
disk and changed nothing a nested container saw.

Also corrects the comment on the service-env substitution, which
claimed to be the one place the address sticks. It is not; it covers
podman's own pulls and non-CLI API clients.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs(nested-containers): record what live verification established 0ba25352b9 
Documents the proxy-layer precedence that took four attempts to get
right, the blob-CDN routes every registry needs alongside its own, the
podman 5 networking packages and how each one fails when absent, and
the BusyBox wget caveat. Closes the DNS open question: public
resolution inside a nested container fails by design; reaching the
proxy was the real problem.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
docs(readme): make the registry route example correct and complete
test / integration-docker (pull_request) Successful in 14s
Details
lint / lint (push) Successful in 49s
Details
test / unit (pull_request) Successful in 1m37s
Details
test / integration-firecracker (pull_request) Successful in 3m25s
Details
test / coverage (pull_request) Successful in 19s
Details
test / publish-infra (pull_request) Has been skipped
Details
tracker-policy-pr / check-pr (pull_request) Failing after 10m8s
Details
cc094765fd 
The example previously left DLP enabled on the blob CDNs, which is
exactly backwards: those hosts carry the multi-hundred-MB layers whose
buffering triggers #455, while the registry hosts they hang off carry
little. It also listed one of quay's three CDNs and omitted the CDN
requirement from the prose, so a reader following it would authenticate
fine and then 403 partway through a pull.

Adds the by-design DNS behaviour and the BusyBox wget caveat, both of
which cost real time during the #392 verification: nslookup failing
inside a nested container looks like breakage but is expected, and
BusyBox wget reports "error getting response" for requests the proxy
served successfully.

Verified by feeding the snippet through parse_frontmatter +
ManifestBottle.from_dict: all nine routes parse with the intended
preserve_auth and DLP settings.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
didericis force-pushed prd-nested-containers from ef9f81ba83 to cc094765fd 2026-07-21 22:54:32 -04:00 Compare
Some required checks failed
test / integration-docker (pull_request) Successful in 14s
Details
lint / lint (push) Successful in 49s
Details
test / unit (pull_request) Successful in 1m37s
Details
test / integration-firecracker (pull_request) Successful in 3m25s
Details
test / coverage (pull_request) Successful in 19s
Details
test / publish-infra (pull_request) Has been skipped
Details
tracker-policy-pr / check-pr (pull_request) Failing after 10m8s
Required
Details
Some required checks were not successful.
You are not authorized to merge this pull request.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin prd-nested-containers:prd-nested-containers
git checkout prd-nested-containers
Sign in to join this conversation.
Reviewers
No Reviewers
didericis
Labels
Clear labels
Compat/Breaking
Breaking change that won't be backward compatible
 Kind/Bug
Something is not working
 Kind/Documentation
Documentation changes
 Kind/Enhancement
Improve existing functionality
 Kind/Feature
New functionality
 Kind/Security
This is security issue
 Kind/Testing
Issue or pull request related to testing
Priority
Critical
1
The priority is critical
Priority
High
2
The priority is high
Priority
Low
4
The priority is low
Priority
Medium
3
The priority is medium
Reviewed
Confirmed
1
Issue has been confirmed
Reviewed
Duplicate
2
This issue or pull request already exists
Reviewed
Invalid
3
Invalid issue
Reviewed
Won't Fix
3
This issue won't be fixed
Status
Abandoned
3
Somebody has started to work on this but abandoned work
Status
Blocked
1
Something is blocking this issue or pull request
Status
Need More Info
2
Feedback is required to reproduce issue or to continue work
Status/Needs Triage
Awaiting initial classification
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
didericis-claude (didericis (claude)) didericis-codex (didericis (codex)) didericis-gemma (gemma) didericis
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: didericis/bot-bottle#456
Delete Branch "prd-nested-containers"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?