firecracker: rootless rootfs build loses uid/gid ownership (everything owned by node) #347

New Issue

2026-07-11T18:19:19-04:00

didericis-claude commented

2026-07-11 18:19:19 -04:00

Problem

The Firecracker guest rootfs is built rootless — docker export <container> | tar -x run as the invoking (non-root) user, then mke2fs -d (see bot_bottle/backend/firecracker/util.py build_base_rootfs_dir). A non-root tar -x can't restore uid 0, so every path in the rootfs ends up owned by the build uid, which maps to node (uid 1000) inside the guest — including /root, /etc, /usr, etc.

This was invisible until the first real end-to-end launch. It broke SSH auth: dropbear (like OpenSSH) refuses root's authorized_keys when /root isn't root-owned. Worked around in bb-init with chown -R 0:0 /root (commit "fix(firecracker): make the launch path work end-to-end").

Why it's worth fixing properly

The chown /root patch unblocks login, but the underlying ownership is still wrong for the whole tree:

System dirs owned by node. The agent runs as node (uid 1000), which now owns /etc, /usr, /bin, … so a compromised agent can modify system files / binaries in its own rootfs. The VM is ephemeral and network-isolated (nft boundary), so this isn't a host-escape, but it does weaken in-guest integrity (e.g. tampering with the injected CA, sudoers-style files, setuid bits).
setuid/setgid bits are dropped by rootless tar, so anything relying on them won't work.
Any provisioning that assumes correct ownership (files created for root vs node) needs per-path chown band-aids like the /root one.

Options

fakeroot / fakeroot-ng around the export+extract so tar restores the original uids/gids into a fake map, then mke2fs -d from that tree. Cheapest change; no new privileges.
User namespaces: extract inside a unshare -Ur (rootless userns) so uid 0 in the ns maps to the invoking uid but tar sees uid 0. Then mke2fs -d reads correct ownership. Also rootless.
Ownership manifest: capture docker export's tar member uids/gids and replay them via an mke2fs device table / a post-pass that chowns inside the image (needs a tool that writes ext4 ownership without mounting).
Keep the status quo + targeted chowns in bb-init for the few paths that matter (current approach), and document that the guest rootfs is node-owned by design.

Option 1 or 2 is the real fix and keeps the build rootless. Worth measuring the build-time cost (the rootfs is cached by image digest, so it's a one-time-per-image hit).

Notes

Keep the build rootless (no host root needed at launch is a core property).
Once ownership is correct, the bb-init chown -R 0:0 /root workaround can be removed.

Ref: surfaced while getting the first end-to-end firecracker launch working.

## Problem The Firecracker guest rootfs is built rootless — `docker export <container> | tar -x` run as the invoking (non-root) user, then `mke2fs -d` (see `bot_bottle/backend/firecracker/util.py` `build_base_rootfs_dir`). A non-root `tar -x` can't restore uid 0, so **every path in the rootfs ends up owned by the build uid**, which maps to `node` (uid 1000) inside the guest — including `/root`, `/etc`, `/usr`, etc. This was invisible until the first real end-to-end launch. It broke SSH auth: dropbear (like OpenSSH) refuses root's `authorized_keys` when `/root` isn't root-owned. Worked around in `bb-init` with `chown -R 0:0 /root` (commit "fix(firecracker): make the launch path work end-to-end"). ## Why it's worth fixing properly The `chown /root` patch unblocks login, but the underlying ownership is still wrong for the whole tree: - **System dirs owned by `node`.** The agent runs as `node` (uid 1000), which now owns `/etc`, `/usr`, `/bin`, … so a compromised agent can modify system files / binaries in its own rootfs. The VM is ephemeral and network-isolated (nft boundary), so this isn't a host-escape, but it does weaken in-guest integrity (e.g. tampering with the injected CA, sudoers-style files, setuid bits). - **setuid/setgid bits are dropped** by rootless tar, so anything relying on them won't work. - Any provisioning that assumes correct ownership (files created for `root` vs `node`) needs per-path `chown` band-aids like the `/root` one. ## Options 1. **fakeroot / `fakeroot-ng`** around the export+extract so tar restores the original uids/gids into a fake map, then `mke2fs -d` from that tree. Cheapest change; no new privileges. 2. **User namespaces**: extract inside a `unshare -Ur` (rootless userns) so uid 0 in the ns maps to the invoking uid but tar sees uid 0. Then `mke2fs -d` reads correct ownership. Also rootless. 3. **Ownership manifest**: capture `docker export`'s tar member uids/gids and replay them via an `mke2fs` device table / a post-pass that `chown`s inside the image (needs a tool that writes ext4 ownership without mounting). 4. Keep the status quo + targeted `chown`s in `bb-init` for the few paths that matter (current approach), and document that the guest rootfs is node-owned by design. Option 1 or 2 is the real fix and keeps the build rootless. Worth measuring the build-time cost (the rootfs is cached by image digest, so it's a one-time-per-image hit). ## Notes - Keep the build rootless (no host root needed at launch is a core property). - Once ownership is correct, the `bb-init` `chown -R 0:0 /root` workaround can be removed. Ref: surfaced while getting the first end-to-end firecracker launch working.

didericis-claude referenced this issue

2026-07-12 16:39:52 -04:00

Firecracker-native, Docker-free backend (PRD 0069) #348

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: didericis/bot-bottle#347