Skip to content

feat: Support large zip files#3118

Merged
messense merged 1 commit into
PyO3:mainfrom
EndPositive:main
Apr 9, 2026
Merged

feat: Support large zip files#3118
messense merged 1 commit into
PyO3:mainfrom
EndPositive:main

Conversation

@EndPositive

Copy link
Copy Markdown
Contributor

Exposes zip's large file support through a new cli arg --compression-enable-large-file-support. This supports use cases where the zip's content is larger than 4GB.

💥 maturin failed
  Caused by: Failed to write to zip archive for "wheel/abc.abi3.so"
  Caused by: Large file option has not been set

Looks like the readmes where not up to date, so the diff is a bit larger than just the compression option.

@messense

messense commented Apr 8, 2026

Copy link
Copy Markdown
Member

Thanks for the PR, is it possible to use https://docs.rs/zip/latest/zip/write/struct.ZipWriter.html#method.set_auto_large_file?

@EndPositive

Copy link
Copy Markdown
Contributor Author

We could do that 👍 only risk is that the consumer of the wheel does not support zip64. Both pip and uv support it, but I don't know about all package registries for example.

@messense

messense commented Apr 8, 2026

Copy link
Copy Markdown
Member

https://github.com/zip-rs/zip2/blob/8ed3d36a4947c6378831bbd944623cd4b5102a0f/src/types.rs#L722-L735

I think it only enables zip64 when the compression size exceeds normal zip size so it's safe to enable.

@EndPositive

Copy link
Copy Markdown
Contributor Author

Sorry I think the point I was trying to make is that if the consumer of the wheel (e.g. uv, pip, whatever registry) does not support unzipping zip64, then your wheel is unusable. Perhaps it's better to have a flag when building the wheel to decide whether you want to take that risk or not.

Anyways, I'm fine with going forward with auto-enable zip64. All consumers I checked do support it.

@messense

messense commented Apr 9, 2026

Copy link
Copy Markdown
Member

Great, let's use set_auto_large_file, thanks!

@EndPositive

Copy link
Copy Markdown
Contributor Author

Actually it doesn't seem like the ZipWriter::set_auto_large_file doesn't work? It only changes whether a 64b data descriptor is written: see zip-rs/zip2#468.

@messense

messense commented Apr 9, 2026

Copy link
Copy Markdown
Member

You are right, guess this is good middle ground.

@messense messense merged commit 1e5d362 into PyO3:main Apr 9, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants