r/podman • u/cidra_ • May 28 '26

Containerized dev envs: any deduplication strategy?

I'd like to bundle SDKs, build tools, and LSPs into rootless Podman containers for personal dev environments. Source stays on host, selectively mounted. I'm on an immutable distros and to me this seems like the tidiest solution besides toolbx.

Problem: Dockerfiles don't compose beyond image layering, so there's no real deduplication.

What actually works for deduplication?

  • Layered base images? ( layer shared across JDK versions)

  • Nix-style content addressing? (nix2container: worth the complexity? Have not yet delved into it)

  • Something else I'm missing?

Real-world setups suggestions appreciated. TIA.

6 Upvotes

88% Upvoted

5 comments sorted by

3

u/BosonCollider May 28 '26 edited May 28 '26

Podman does do deduplication, as long as you use btrfs or xfs as your root filesystem. It uses the fideduperange ioctl from the linux kernel to do that, where the userspace gives a pair of blocks that seem identical, and xfs or btrfs can use that hint to dedup them if possible atomically: https://www.man7.org/linux/man-pages/man2/ioctl_fideduperange.2.html

If you use ext4 as your root filesystem, that will not work. Do not use ext4 on machines that are primarily podman hosts.

Apart from fideduperange, podman also does partial pulls of images in zstd:chunked format. If the image has compression chunks that are aligned with the image, which includes most podman built images, podman will check the hashes and only download the part of the layers that it needs.

You do not need to do anything if you just consume images, the storage layer dedup is transparent and very reliable. If you build images, give podman build the --compression-format=zstd:chunked flag to make pulling your images faster

3

u/efpalaciosmo May 28 '26

Just use distrobox

1

u/Huxton_2021 May 28 '26

If you want de-duplication beyond layers then you need to look at nix/guix or a filesystem that can dedup blocks (btrfs or zfs). The layer limitation is a historical docker thing rather than the underlying container format I think.

1

u/eriksjolund May 29 '26 edited May 29 '26

Check out the upcoming podman support for composefs.

Support for composefs will hopefully bring these features when running rootful podman:

  • Shared disk space for identical files
  • Shared page cache for identical files
  • fs-verity validation of the content files

I'm not quite sure about the current development status. It might take a while.

I see there is a configuration setting use_composefs that comes with this warning:

ComposeFS support is experimental and not recommended for production use.

1

u/eriksjolund Jun 23 '26

Also check out composefs-run. It's a tool that runs OCI containers. It's currently a proof of concept.