Skip to content

estargz: parallelize MinChunkSize builds - #2340

Open
simonepri wants to merge 1 commit into
containerd:mainfrom
simonepri:estargz-parallelize-min-chunk-size
Open

estargz: parallelize MinChunkSize builds#2340
simonepri wants to merge 1 commit into
containerd:mainfrom
simonepri:estargz-parallelize-min-chunk-size

Conversation

@simonepri

Copy link
Copy Markdown
Contributor

Builds with MinChunkSize > 0 run on a single core, no matter how large the layer is.

They are serial because the writer maintains one invariant -- every gzip stream except the last holds at least MinChunkSize compressed bytes -- and does so by placing each stream boundary based on the compressed size of everything written before it.

The build can be parallelized while preserving the invariant as long as:

  • each worker's slice of the tar holds at least MinChunkSize * 1032 uncompressed bytes (1032 is DEFLATE's maximum compression ratio), so every slice fills at least one full stream;

  • the trailing stream of a slice, which usually ends below the minimum, is folded into the stream before it. The writer now withholds a full stream's terminator until the next stream also reaches the minimum, and folds the tail back by replaying its buffered raw bytes. Only the short tail is ever recompressed.

After this change the invariant also comes out stronger: the trailing stream of the whole blob, previously allowed to end short, is folded as well. A stream now ends below MinChunkSize if and only if the data itself is smaller (or a prefetch landmark forces a boundary).

Note that with this change MinChunkSize layer digests differ from previous releases: large layers build in parallel (the layout depends on GOMAXPROCS, like every other eStargz build) and trailing short streams are folded away. eStargz makes no cross-version byte stability promise.

@simonepri
simonepri marked this pull request as draft June 12, 2026 13:58
@simonepri
simonepri force-pushed the estargz-parallelize-min-chunk-size branch from 874ebd2 to 8a5b171 Compare June 12, 2026 14:18
@simonepri
simonepri marked this pull request as ready for review June 12, 2026 14:45
@simonepri

simonepri commented Jun 12, 2026

Copy link
Copy Markdown
Contributor Author

@ktock since you implemented the serial version of this, could you take a look.
The idea is that once #2341 lands then user would be able to go back to the previous implementation (modulo the last chunk being still merged back) by setting parallelism to 1.

@simonepri
simonepri force-pushed the estargz-parallelize-min-chunk-size branch from 8a5b171 to 98ba4fa Compare June 15, 2026 08:05
@simonepri

Copy link
Copy Markdown
Contributor Author

@AkihiroSuda @ktock friendly ping

@simonepri
simonepri force-pushed the estargz-parallelize-min-chunk-size branch from 98ba4fa to 1577e1b Compare August 7, 2026 10:43
Builds with MinChunkSize > 0 run on a single core, no matter how large
the layer is.

They are serial because the writer maintains one invariant -- every
gzip stream except the last holds at least MinChunkSize compressed
bytes -- and does so by placing each stream boundary based on the
compressed size of everything written before it.

The build can be parallelized while preserving the invariant as long
as:

- each worker's slice of the tar holds at least MinChunkSize * 1032
  uncompressed bytes (1032 is DEFLATE's maximum compression ratio), so
  every slice fills at least one full stream;

- the trailing stream of a slice, which usually ends below the minimum,
  is folded into the stream before it. The writer now withholds a full
  stream's terminator until the next stream also reaches the minimum,
  and folds the tail back by replaying its buffered raw bytes. Only the
  short tail is ever recompressed.

The invariant also comes out stronger: the trailing stream of the whole
blob, previously allowed to end short, is folded as well. A stream now
ends below MinChunkSize if and only if the data itself is smaller (or a
prefetch landmark forces a boundary).

Note that with this change MinChunkSize layer digests differ from
previous releases: large layers build in parallel (the layout depends
on GOMAXPROCS, like every other eStargz build) and trailing short
streams are folded away. eStargz makes no cross-version byte stability
promise.

Signed-off-by: Simone Primarosa <simone.primarosa@gmail.com>
@simonepri
simonepri force-pushed the estargz-parallelize-min-chunk-size branch from 1577e1b to 4fe854e Compare August 8, 2026 11:39
@simonepri

Copy link
Copy Markdown
Contributor Author

@ktock rebased this on main again after the --parallelism flag merge

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant