estargz: parallelize MinChunkSize builds - #2340
Open
simonepri wants to merge 1 commit into
Open
Conversation
simonepri
marked this pull request as draft
June 12, 2026 13:58
simonepri
force-pushed
the
estargz-parallelize-min-chunk-size
branch
from
June 12, 2026 14:18
874ebd2 to
8a5b171
Compare
simonepri
marked this pull request as ready for review
June 12, 2026 14:45
Contributor
Author
simonepri
force-pushed
the
estargz-parallelize-min-chunk-size
branch
from
June 15, 2026 08:05
8a5b171 to
98ba4fa
Compare
Contributor
Author
|
@AkihiroSuda @ktock friendly ping |
simonepri
force-pushed
the
estargz-parallelize-min-chunk-size
branch
from
August 7, 2026 10:43
98ba4fa to
1577e1b
Compare
Builds with MinChunkSize > 0 run on a single core, no matter how large the layer is. They are serial because the writer maintains one invariant -- every gzip stream except the last holds at least MinChunkSize compressed bytes -- and does so by placing each stream boundary based on the compressed size of everything written before it. The build can be parallelized while preserving the invariant as long as: - each worker's slice of the tar holds at least MinChunkSize * 1032 uncompressed bytes (1032 is DEFLATE's maximum compression ratio), so every slice fills at least one full stream; - the trailing stream of a slice, which usually ends below the minimum, is folded into the stream before it. The writer now withholds a full stream's terminator until the next stream also reaches the minimum, and folds the tail back by replaying its buffered raw bytes. Only the short tail is ever recompressed. The invariant also comes out stronger: the trailing stream of the whole blob, previously allowed to end short, is folded as well. A stream now ends below MinChunkSize if and only if the data itself is smaller (or a prefetch landmark forces a boundary). Note that with this change MinChunkSize layer digests differ from previous releases: large layers build in parallel (the layout depends on GOMAXPROCS, like every other eStargz build) and trailing short streams are folded away. eStargz makes no cross-version byte stability promise. Signed-off-by: Simone Primarosa <simone.primarosa@gmail.com>
simonepri
force-pushed
the
estargz-parallelize-min-chunk-size
branch
from
August 8, 2026 11:39
1577e1b to
4fe854e
Compare
Contributor
Author
|
@ktock rebased this on main again after the --parallelism flag merge |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds with
MinChunkSize > 0run on a single core, no matter how large the layer is.They are serial because the writer maintains one invariant -- every gzip stream except the last holds at least
MinChunkSizecompressed bytes -- and does so by placing each stream boundary based on the compressed size of everything written before it.The build can be parallelized while preserving the invariant as long as:
each worker's slice of the tar holds at least
MinChunkSize * 1032uncompressed bytes (1032is DEFLATE's maximum compression ratio), so every slice fills at least one full stream;the trailing stream of a slice, which usually ends below the minimum, is folded into the stream before it. The writer now withholds a full stream's terminator until the next stream also reaches the minimum, and folds the tail back by replaying its buffered raw bytes. Only the short tail is ever recompressed.
After this change the invariant also comes out stronger: the trailing stream of the whole blob, previously allowed to end short, is folded as well. A stream now ends below
MinChunkSizeif and only if the data itself is smaller (or a prefetch landmark forces a boundary).Note that with this change
MinChunkSizelayer digests differ from previous releases: large layers build in parallel (the layout depends on GOMAXPROCS, like every other eStargz build) and trailing short streams are folded away. eStargz makes no cross-version byte stability promise.