Simple PNG differs between 11.0.0 and 11.1.0 (macOS only)

[dolfim-ibm]

What did you do?

I create a simple image with ImageDraw, and then save to PNG.

What did you expect to happen?

Independent from which OS and which Pillow version, the file should be the same.

What actually happened?

With version 11.1.0, and only on macOS, the output between Pillow 11.0.0 and 11.1.0 is different.

In Linux, the same example is correct.

What are your OS, Python and Pillow versions?

• OS: macOS 15.3.1
• Python: 3.12
• Pillow: 11.1.0 and 11.0.0

--------------------------------------------------------------------
Pillow 11.1.0
Python 3.12.7 (main, Oct  1 2024, 02:05:46) [Clang 15.0.0 (clang-1500.3.9.4)]
--------------------------------------------------------------------
Python executable is /Users/dol/scratch/pillow-scratch/venv-11.1.0/bin/python3
Environment Python files loaded from /Users/dol/scratch/pillow-scratch/venv-11.1.0
System Python files loaded from /opt/homebrew/opt/python@3.12/Frameworks/Python.framework/Versions/3.12
--------------------------------------------------------------------
Python Pillow modules loaded from /Users/dol/scratch/pillow-scratch/venv-11.1.0/lib/python3.12/site-packages/PIL
Binary Pillow modules loaded from /Users/dol/scratch/pillow-scratch/venv-11.1.0/lib/python3.12/site-packages/PIL
--------------------------------------------------------------------
--- PIL CORE support ok, compiled for 11.1.0
*** TKINTER support not installed
--- FREETYPE2 support ok, loaded 2.13.2
--- LITTLECMS2 support ok, loaded 2.16
--- WEBP support ok, loaded 1.5.0
--- JPEG support ok, compiled for libjpeg-turbo 3.1.0
--- OPENJPEG (JPEG2000) support ok, loaded 2.5.3
--- ZLIB (PNG/ZIP) support ok, loaded 1.3.1.zlib-ng, compiled for zlib-ng 2.2.2
--- LIBTIFF support ok, loaded 4.6.0
--- RAQM (Bidirectional Text) support ok, loaded 0.10.1, fribidi 1.0.16, harfbuzz 10.1.0
*** LIBIMAGEQUANT (Quantization method) support not installed
--- XCB (X protocol) support ok
--------------------------------------------------------------------

import importlib.metadata
from PIL import Image, ImageDraw

version = importlib.metadata.version("Pillow")
print(f"{version=}")

size = (16, 16)
im = Image.new("RGB", size, "black")
draw = ImageDraw.Draw(im)
draw.ellipse((0, 0, size[0] - 1, size[1] - 1), fill="red")

im.save(f"im-{version}.png", format="PNG")

This generates files like im-11.0.0.png, im-11.1.0.png, im-11.2.0.dev0.png (from the current main).

On mac

$ diff im-11.0.0.png im-11.1.0.png
Binary files im-11.0.0.png and im-11.1.0.png differ

$ diff im-11.0.0.png im-11.2.0.dev0.png
no diff

On linux

$ diff im-11.0.0.png im-11.1.0.png
no diff

[dolfim-ibm]

We believe this is fixed with the latest main (but we don't know exactly which PR).

If you can confirm it, would it please be possible to cut a Pillow fix release with this?

Waiting for the 11.2.0 in April is definitely too late and we would then have to restrict the supported Pillow version in our package, which we would like to avoid.

[radarhere]

Hi. I don't think that main does have different behaviour.

Instead, you're experiencing the effects of #8500, where we switched from zlib to zlib-ng in our wheels. See also https://pillow.readthedocs.io/en/stable/releasenotes/11.1.0.html#zlib-ng-in-wheels. As noted in the PR, this changes the output, but improves performance.

So 11.0.0 wheels use zlib, and 11.1.0 wheels use zlib-ng (you can see this in the report you provided, ZLIB (PNG/ZIP) support ok, loaded 1.3.1.zlib-ng). You would be experiencing zlib behaviour with main because you are building main from source, and presumably you have zlib installed locally.

To demonstrate, I've put together radarhere@9e8a4e5, and you can see at https://github.com/radarhere/Pillow/actions/runs/13316174521/job/37190653827 that installing all three versions from source match in their output, because they're using the same zlib version, but the 11.1.0 wheel differs.

version='11.0.0'
11.0.0 from source matches 11.2.0.dev0 from source
version='11.1.0'
11.1.0 from source matches 11.2.0.dev0 from source
version='11.0.0'
11.0.0 wheel matches 11.2.0.dev0 from source
version='11.1.0'
And now, check 11.1.0 wheel against 11.2.0.dev0 from source
Binary files im-11.1.0.png and im-11.2.0.dev0.png differ

[dolfim-ibm]

Interesting, thanks a lot for the follow up.

We currently have the issue that running our tests locally produces a different file than on the Github CI.
Do you have any suggestion for having some stability between local dev (on mac) and cloud CI (on ubuntu)?

[wiredfool]

I don’t think that file bytes comparison is a valid test for PNG files. There are several different ways that a PNG can be encoded losslessly into different bytes — there are different filter methods that can be chosen arbitrarily by the encoder to improve compression on a scan line by scan line basis. On top of that, the underlying compression can be done that are bit different, but still decompress to the same stream of bits.

So, all in all, I think that bit differences in png files is an invalid failure for a test.

[radarhere]

Speaking theoretically, I don't think you can completely control this.
Speaking practically, I find that yes, python3 -m pip install Pillow gives different results on macOS and Ubuntu. Building Pillow from source gives the same result on both environments though, and it also matches between manylinux and macOS in our wheel-building jobs where we build zlib-ng ourselves.

However, I also don't think it's a useful test - even if you could configure your test environments to produce identical output, we sometimes receive reports from users who have run our test suite on their own (e.g. #6858), and if your users do that with their own Pillow installations, that could be a problem for you.

I'm not sure what the purpose of the test is, but I would suggest you compare the pixels in the images, rather than the bytes.

[radarhere]

@dolfim-ibm was that helpful? Is there anything further we can do for you?

[dolfim-ibm]

At the moment we changed some pipeline to just check if the image size is the same. It would be great if you have some suggestion for:

• some flag which could be use for making the generated PNG the same among platforms (among PIL versions is not necessary)
• or a simple helper function in PIL which can be used to verify the images, which takes the differences into account.

The analysis and understanding was anyway very helpful, many thanks.

[radarhere]

I would suggest you compare the pixels in the images, rather than the bytes.

If you wanted to do this, you could use https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.tobytes

from PIL import Image
with Image.open("im1.png") as im1:
  with Image.open("im2.png") as im2:
    assert im1.tobytes() == im2.tobytes()

Or you could more generally just check for image equality

from PIL import Image
with Image.open("im1.png") as im1:
  with Image.open("im2.png") as im2:
    assert im1 == im2

Pillow/src/PIL/Image.py

Lines 667 to 677 in 8a3c4f5

	def __eq__(self, other: object) -> bool:
	if self.__class__ is not other.__class__:
	return False
	assert isinstance(other, Image)
	return (
	self.mode == other.mode
	and self.size == other.size
	and self.info == other.info
	and self.getpalette() == other.getpalette()
	and self.tobytes() == other.tobytes()
	)

[radarhere]

some flag which could be use for making the generated PNG the same among platforms (among PIL versions is not necessary)

As been discussed, I don't think this is possible.

or a simple helper function in PIL which can be used to verify the images, which takes the differences into account.

Hopefully my previous comment is an answer to this.

Closing, unless there is anything further.