12.0.0 release was re-tagged / re-created?

[Antiz96]

Hello,

When downloading upstream sources for packaging purposes, we are "locking" them via a checksum mechanism.
We do this to ensure that sources have not been tampered with since the first build of the package, as well as for reproducible builds purposes.

We have identified that the pillow source's checksum for the 12.0.0 release have changed since the first time we built the related package, which might indicate that said release has either been tampered with or got re-tagged / re-created, which seems to be confirmed by the fact that we have first built the package for the 12.0.0 release at October 15, 2025 at 2:35 PM GMT+2, while the latest 12.0.0 tag was (re-?)created at October 15, 2025 at 7:06 PM GMT+2.

Can you confirm if the 12.0.0 release was re-tagged / re-created?
If so, can we safely update our pillow package to the "new" source for the 12.0.0 release?
Also, given said re-tag / re-creation was intentional, would it be possible to create a new point release instead of re-creating the same release in the future?

Thanks in advance 🙂

[aclark4life]

Not that I know of …

[radarhere]

I can see there was a commit to increase the version at 1:40pm GMT+2 - f2790b4

It looks like there were some problems in our CI pipelines at the time, and it led to #9258 being merged and #9178 (comment)

Old 12.0.0 tag deleted.

New release started, and new 12.0.0 tag pushed and building:

Our final version increase was committed at 7:06am GMT+2.

Hopefully that answers your question about what happened.

As far as future handling of releases, I'll ping @hugovk so that he's aware of your feedback.

[aclark4life]

Oh, I do remember that thanks @radarhere .

would it be possible to create a new point release instead of re-creating the same release in the future?

I think we'd be open to improving our release workflow but I'm not sure about what to do in this specific case. For example, if we released a "bad" 12.0.0 to PyPI the fix would be to follow up with a point release. Presumably the CI failed prior to upload so we didn't need a point release, at least based on our current understanding of what determines when a point release is required.

[Antiz96]

[...]

Hopefully that answers your question about what happened.

It does, thanks for the information!

As far as future handling of releases, I'll ping @hugovk so that he's aware of your feedback.

Thanks, appreciated 🙏

[Antiz96]

Oh, I do remember that thanks @radarhere .

would it be possible to create a new point release instead of re-creating the same release in the future?

I think we'd be open to improving our release workflow but I'm not sure about what to do in this specific case. For example, if we released a "bad" 12.0.0 to PyPI the fix would be to follow up with a point release. Presumably the CI failed prior to upload so we didn't need a point release, at least based on our current understanding of what determines when a point release is required.

I totally understand that their might be some intricacies on your side, and I'm also aware that the following may sound like downstream specific expectations.
But regarding the transparency & security of our packaging / re-distribution process, we are kinda expecting an immutability for upstream tags / releases which we are using as our package sources. Once a tag is created, we ideally expect it to never move, otherwise it prevents packages rebuilds, breaks reproducible builds and may indicate a supply chain attack that needs to be investigated.

As a general rule of thumb, I would personally advocate to always go for a new point release, even if the CI presumably failed prior to upload. That is the most reliable way to indicate to downstream redistributors / Linux distributions that the previous release is bad and shouldn't be used (as also confirmed by other distro maintainers, see e.g. this comment from a Gentoo developer).

But again, I'm not necessarily aware of all the intricacies of your release process so I get that it might be easier said than done from your side. Thanks for considering it though 🙏

Regardless, I was primarily looking for an explanation here and I got it.
All good from my side, I guess this issue could be closed 🙂

[hugovk]

Hi, thanks for opening the issue to ask 👍

Yeah, we've done this before:

• 8.2.0 Release Pillow 8.2.0 on April 1, 2021 #5287 (comment)
• 6.1.0 Release Pillow 6.1.0 on July 1, 2019 #3835 (comment)

And in one case we deleted a tag without a re-tag, after hitting the PyPI quota and needing to yank the partial PyPI release and rethink the release contents:

• 11.2.0 Release Pillow 11.2.0 on April 1, 2025 #8722 (comment)

The rationale has been that a release is not done until it's been actually released, not just that we've tagged.

That was the case here (as detailed in the release issue). The original 12.0.0 tag was hanging the wheel build, so I deleted the tag, fixed the CI, and restarted with a new tag. There was only a one-line GitHub Actions change, the rest of the code was the same.

It sounds like your automation is triggering from the tag on GitHub.

We need to push the tag near the start of the release process to trigger the release automation, including the multi-hour wheel build. This sometimes (and thankfully, rarely) fails because it's so slow we don't run it as part of regular PRs. We do try and mitigate by running it on cron twice a week, and often manually trigger it just before a release.

see e.g. this comment from a Gentoo developer).

I see they also follow the PyPI release rather than GitHub tag. You could also use that:

• https://docs.pypi.org/api/feeds/
• https://pypi.org/rss/project/pillow/releases.xml

That shows 12.0.0 was released at 18:21 GMT, which matches with the re-tag.

Or better yet, follow the GitHub "Release":

• https://github.com/python-pillow/Pillow/releases
• https://github.com/python-pillow/Pillow/releases.atom

That shows the 12.0.0 release was published at 18:27 GMT. This is only published at the end of a successful release, and would have also helped in the case of 11.2.0's partial release to PyPI.

[Antiz96]

Hi @hugovk,

Thanks for the detailed response!

I would still personally advocate that a general good practice would be for VCS object (such as git tags) to be immutable once created / pushed.
Aside from downstream expectations on that front, it also avoids eventual suspicious concerns. I guess one could imagine my internal feelings when our automation detected a re-tag for the 12.0.0 tag but the creator of the project stated not being aware of that (specifically given the popularity of pillow and therefore the potential impact of a supply chain attack here). Since some fairly recent events, we are even more cautious with those things then we already used to be. 🚨 😁

That being said, I understand that this may not be always easy to do given your release workflow.
Once again, I was primarily looking for a confirmation that this re-tag was intentional / known / expected.
Thanks again for your crystal clear explanations, that helps to understand the "why" here! 🙏

Given your input, I agree that monitoring GitHub releases specifically (rather than git tags) makes sense here.
I made the according change in our automation, this should hopefully cover those cases for the future on our side!

Thanks once again!
From my side, this issue is complete. 🤗

[hugovk]

Great, thanks again!

[aclark4life]

I guess one could imagine my internal feelings when our automation detected a re-tag for the 12.0.0 tag but the creator of the project stated not being aware of that (specifically given the popularity of pillow and therefore the potential impact of a supply chain attack here).

Hah! My apologies. That was the best I could come up with in part due to my PTSD about supply chain attacks and I wanted to at least ACK the issue … in the future if you suspect a security issue please do follow the Security process which is to say email: security@tidelift.com.

I figured since it was a public disclosure and I was vaguely aware that we hadn't released to PyPI that we could discuss in public and as it turns out, we can! Still, you never know these days, as we're all aware. Anyway, thanks again for raising the issue. As I said, I think we're always willing to discuss and improve when/where we can.

[Antiz96]

Hah! My apologies. That was the best I could come up with in part due to my PTSD about supply chain attacks and I wanted to at least ACK the issue ...

Haha, no problem! I appreciated the quick ACK actually 🤗

in the future if you suspect a security issue please do follow the Security process which is to say email: security@tidelift.com.

I figured since it was a public disclosure and I was vaguely aware that we hadn't released to PyPI that we could discuss in public and as it turns out, we can! Still, you never know these days, as we're all aware.

Fair point! I did not actually suspected anything malicious when reporting this issue at first but, as you said, you never know these days. Out of precaution, I will follow the Security process next time (if that ever happens).

Anyway, thanks again for raising the issue. As I said, I think we're always willing to discuss and improve when/where we can.

Thanks to all of you for the prompt and detailed responses!
That is greatly appreciated 🙏