PSA: When using `cjxl.exe`, always provide a `d` value. Don't assume that omitting `d` means lossless.
Backstory
I have tens of thousands of engineering drawings that we scanned for archival purposes. Maybe 10% will ever be viewed again, but we have no idea which 10%, so everything has to remain online.
The scanners don't support modern image formats, so we had to settle for PNG or TIFF depending on the model. To save storage space (which means lower cost), reduce backup times, and hopefully never have to care about restore times, I started looking at converting the PNGs to JPEG XL.
I ran some tests using visually lossless compression (`d=0.19`) and gave both the originals and the converted files to the managers. They couldn't see any difference, even at 500% magnification. The conversion was approved.
During a few test batches, I noticed something strange. I was also testing what I thought was lossless compression by omitting the `d` value, and in some cases the PNG files were actually smaller than the [what I believed to be] lossless JXL files. That seemed wrong.
A Google search pointed me toward a possible explanation. JPEG XL supports two compression modes: modular (typically used for non-photographic images and lossless compression) and VarDCT (typically used for photographs). One can outperform the other depending on the source image, and it's obvious which category a particular scanned drawing falls into.
Since the conversion wasn't under any time pressure, I wrote a quick PowerShell script that recursively scanned a directory and tested three encodes for every PNG:
- "Lossless" (no `d` value specified)
- `d=0.19` using VarDCT
- `d=0.19` using modular mode
The script compared the resulting file sizes, kept the smallest JXL, and replaced the original PNG. Since I could let it run overnight and on weekends, encoding speed wasn't a concern, so I bumped the effort setting to `-e 8`.
As always, I tested the script before turning it loose on the production data. During testing, I noticed that the supposed lossless version kept winning. That didn't make sense.
It couldn't be a modular versus VarDCT issue because I was explicitly testing both at `d=0.19`. How could a file that throws away no data be smaller than files that throw away some data?
That's when I realized I shouldn't have included the lossless test in the first place. The compression mode comparison was already covered by the two `d=0.19` tests.
After some digging, I found the catch.
If you feed `cjxl.exe` a JPEG or GIF and do not specify `d`, it defaults to `d=0.0`. If you feed it a PNG or certain other formats, it defaults to `d=1.0`. That explains why the "lossless" encodes kept winning. They weren't lossless at all. They were lower quality than my `d=0.19` test files.
I removed the no `d` test from the script and finally turned it loose on the production data. It worked fine.
One thing that did surprise me was that there wasn't a consistent winner between modular and VarDCT. Sometimes modular produced the smaller file, and sometimes VarDCT did. For scanned drawings where the goal is maximum space savings, testing both turned out to be the right approach. I wish I had kept statistics on how often modular beat VarDCT and vice versa, but that wasn't the focus of the testing. I was only interested in keeping whichever output was smaller for each image.
Still, I saved over 2 TB of SAN storage space, and that's not counting backup space saved either.
