Perceptual Image Compression

[BezPowell]

For some time now, there's been an active feature request to add perceptual image compression to Zola (a static site generator) in the same vein as tools such as jpeg-recompress and pio. This week, I started work on a small library to hopefully provide that for a number of lossy encoders in the image crate; the library is essentially a very thin wrapper over the image and image-compare crates, that allows running a binary search over a number of different encoder 'quality levels' to find which one most closely matches a target SSIM value.

While it would theoretically be possible to just make an external call to one of the aforementioned projects, I thought it would be more useful in the long run to write a library that tightly integrates with image-rs, and is generic over encoders to allow easily adding others in future. The primary aim is for integration with Zola, but I would like to make the API generic enough to be useful to other projects.

My motivation for posting this here is two-fold: In the hope that it might be useful to others, and to ask for feedback on the API if you do use it; how easy / hard is to integrate into other projects? Is there any functionality missing that you would need? Could I make a better choice of default values?

While I feel I'm getting reasonably competent writing rust, I'm far from an expert, and image encoder internals are still well beyond me. Any feedback from those more knowledgeable on the subject would be much appreciated.

[fintelia]

Thanks for sharing this!

I do think that we could be better about allowing code to be generic over encoders, though it is a tricky area because different formats have different sorts of knobs. Like, we'd probably want to keep the same API across both lossless and lossy encoders (and formats like WebP that support both!)

[197g]

One interesting point here is that the definition of the knob is very actionable: We can say level x is defined as trying spend at most the processing power to achieve a compression of factor x. Aka, if not achievable we might do something exhaustively but try not to spend work on more than that. Now this does not need to be achieved precisely per file, a stochastical argument over a representative corpus for instance would satisfy me, but it would make it easy to have a common type to be translated by format / encoding library.

[fintelia]

The knob means different things for lossless and lossy formats. For a lossless format there's usually an "effort" control to indicate the tradeoff between time spent and compression level. While lossy formats usually have a "quality" control that doesn't impact time very much, but controls the tradeoff between compression ratio and accuracy to the original image. And then there's weirder stuff like DDS which is lossy with a fixed compression ratio, and has an effort slider to control compression time vs. accuracy.

[BezPowell]

Speaking as an end-user of the crate, I think the lack of a generic API for encoders is only annoying until you realize the differences between them and that each format has its own set of strengths and weaknesses. When I first started playing around with webp, I found it strange that its quality settings didn't match to jpegs, until I realized that 'quality' can mean different things for different use cases; am I saying that I want to achieve X level of compression, regardless of the effect on encode time and perceived quality, or am I saying that I have a certain threshold of perceived quality that I don't want to drop below, and that the codec should find the smallest file-size that remains above this?

I feel that encoders should almost be offering 3 targets: visual 'quality', file-size, and encode time, and that the end user selects one that they want to target and lets the encoder figure out the other 2 dependent variables. Something like that, with closely correlated variables between encoders, could be a useful feature but, on the other hand, some kind of lookup table of how one codec's quality level compares to another's, calculated from a large corpus of images, might by just as useful.

It's an area that I really should put some more thought into. Thank you for the kind words, and for giving me some interesting food for thought!