Skip to content

digestset: cleanup/refactor, and touch-up GoDoc - #120

Open
thaJeztah wants to merge 11 commits into
opencontainers:masterfrom
thaJeztah:cleanup_digestset
Open

digestset: cleanup/refactor, and touch-up GoDoc#120
thaJeztah wants to merge 11 commits into
opencontainers:masterfrom
thaJeztah:cleanup_digestset

Conversation

@thaJeztah

Copy link
Copy Markdown
Member

Refactor / cleanup the code; slight improvements in Benchmarks, and very minor regression in some, but probably mostly noise;

pkg: github.com/opencontainers/go-digest/digestset
cpu: Apple M3 Pro
                │  before.txt  │             after2.txt              │
                │    sec/op    │   sec/op     vs base                │
Add10-2           4.806µ ±  1%   4.858µ ± 1%   +1.07% (p=0.000 n=10)
Add100-2          59.79µ ±  1%   60.36µ ± 1%   +0.95% (p=0.005 n=10)
Add1000-2         680.6µ ±  1%   683.0µ ± 1%        ~ (p=0.089 n=10)
Remove10-2        6.123µ ±  1%   6.162µ ± 0%   +0.65% (p=0.000 n=10)
Remove100-2       63.32µ ±  0%   63.84µ ± 0%   +0.81% (p=0.001 n=10)
Remove1000-2      670.3µ ± 21%   672.4µ ± 1%        ~ (p=0.218 n=10)
Lookup10-2        29.88n ±  0%   27.66n ± 0%   -7.40% (p=0.000 n=10)
Lookup100-2       41.53n ±  1%   36.71n ± 0%  -11.60% (p=0.000 n=10)
Lookup1000-2      75.70n ±  1%   62.02n ± 3%  -18.08% (p=0.000 n=10)
ShortCode10-2     220.7n ±  1%   215.4n ± 1%   -2.36% (p=0.000 n=10)
ShortCode100-2    1.657µ ±  1%   1.595µ ± 1%   -3.77% (p=0.000 n=10)
ShortCode1000-2   19.44µ ±  1%   18.81µ ± 1%   -3.22% (p=0.000 n=10)
geomean           4.242µ         4.084µ        -3.71%

                │   before.txt   │              after2.txt               │
                │      B/op      │     B/op      vs base                 │
Add10-2             608.0 ± 0%       608.0 ± 0%       ~ (p=1.000 n=10) ¹
Add100-2          5.609Ki ± 0%     5.609Ki ± 0%       ~ (p=1.000 n=10) ¹
Add1000-2         54.93Ki ± 0%     54.93Ki ± 0%       ~ (p=1.000 n=10) ¹
Remove10-2          128.0 ± 0%       128.0 ± 0%       ~ (p=1.000 n=10) ¹
Remove100-2         944.0 ± 0%       944.0 ± 0%       ~ (p=1.000 n=10) ¹
Remove1000-2      8.047Ki ± 0%     8.047Ki ± 0%       ~ (p=1.000 n=10) ¹
Lookup10-2          0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Lookup100-2         0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
Lookup1000-2        0.000 ± 0%       0.000 ± 0%       ~ (p=1.000 n=10) ¹
ShortCode10-2       664.0 ± 0%       664.0 ± 0%       ~ (p=1.000 n=10) ¹
ShortCode100-2    4.836Ki ± 0%     4.836Ki ± 0%       ~ (p=1.000 n=10) ¹
ShortCode1000-2   80.12Ki ± 0%     80.12Ki ± 0%       ~ (p=1.000 n=10) ¹
geomean                        ²                 +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

                │  before.txt   │              after2.txt              │
                │   allocs/op   │  allocs/op   vs base                 │
Add10-2            12.00 ± 0%      12.00 ± 0%       ~ (p=1.000 n=10) ¹
Add100-2           102.0 ± 0%      102.0 ± 0%       ~ (p=1.000 n=10) ¹
Add1000-2         1.002k ± 0%     1.002k ± 0%       ~ (p=1.000 n=10) ¹
Remove10-2         2.000 ± 0%      2.000 ± 0%       ~ (p=1.000 n=10) ¹
Remove100-2        2.000 ± 0%      2.000 ± 0%       ~ (p=1.000 n=10) ¹
Remove1000-2       2.000 ± 0%      2.000 ± 0%       ~ (p=1.000 n=10) ¹
Lookup10-2         0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Lookup100-2        0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
Lookup1000-2       0.000 ± 0%      0.000 ± 0%       ~ (p=1.000 n=10) ¹
ShortCode10-2      4.000 ± 0%      4.000 ± 0%       ~ (p=1.000 n=10) ¹
ShortCode100-2     4.000 ± 0%      4.000 ± 0%       ~ (p=1.000 n=10) ¹
ShortCode1000-2    6.000 ± 0%      6.000 ± 0%       ~ (p=1.000 n=10) ¹
geomean                       ²                +0.00%                ²
¹ all samples are equal
² summaries must be >0 to compute geomean

digestset: simplify short digest matching

Use strings.HasPrefix for full-length and shortened values, consolidate the
optional algorithm check, and use strings.HasPrefix directly where no
algorithm is specified.

This preserves the existing behavior while making the matching logic easier
to follow.

digestset: remove redundant digestEntries type

The digestEntries type was added as part of the original implementation in
distribution@8aacddd; it implements sort.Interface, but this was never
used (there's no sort.Sort on the slice, and sort.Search does not use
it).

Replace it with a plain slice, and remove initialization from NewSet,
because the zero-value is usable for this package.

digestset: document, and use zero values for Set

Document that the zero-value of Set is usable, borrowing some wording
from stdlib's bytes.NewBuffer (which is similar), and replace uses of
NewSet in our tests.

digestset: rename hex var for clarity

The hex value is either a full hex (encoded) value, or a prefix;
rename the var to more clearly indicate it may be partial.

digestset: inline searchFuncs

This avoids a function-scoped variable for the searchFunc, whereas
we're only interested in the result (idx).

digestset: simplify entries loop

Use a basic range loop, starting at the idx from the binary-search,
and store the digest itself as match, instead of using digestEntry
as intermediate.

digestset: inline checkShortMatch

Now that it's only used in a single place, we can open-code it to
make it more transparent what logic is used.

digestset: remove redundant lookup search

Use a single binary search to find the start of the matching encoded-value
prefix range. The subsequent scan must already inspect the complete range to
detect ambiguity and already filters by algorithm, making the second
algorithm-aware search redundant.

digestset: Remove: remove intermediate digestEntry struct

digestset: simplify short-code generation

Range over entries directly and refactor the loop to make the
prefix extension and completion logic easier to follow.

digestset: touch-up some GoDoc

@thaJeztah

Copy link
Copy Markdown
Member Author

Split this in small, incremental changes to make the logic easier to review.

There's some cleanups we can do once we update the minimum Go version (slices.Clone, builtin max version etc).

Use strings.HasPrefix for full-length and shortened values, consolidate the
optional algorithm check, and use strings.HasPrefix directly where no
algorithm is specified.

This preserves the existing behavior while making the matching logic easier
to follow.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The digestEntries type was added as part of the original implementation in
[distribution@8aacddd]; it implements `sort.Interface`, but this was never
used (there's no `sort.Sort` on the slice, and `sort.Search` does not use
it).

Replace it with a plain slice, and remove initialization from `NewSet`,
because the zero-value is usable for this package.

[distribution@8aacddd]: distribution/distribution@7258fda

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Document that the zero-value of Set is usable, borrowing some wording
from stdlib's bytes.NewBuffer (which is similar), and replace uses of
NewSet in our tests.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The hex value is either a full hex (encoded) value, or a prefix;
rename the var to more clearly indicate it may be partial.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This avoids a function-scoped variable for the searchFunc, whereas
we're only interested in the result (idx).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use a basic range loop, starting at the idx from the binary-search,
and store the digest itself as match, instead of using digestEntry
as intermediate.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Now that it's only used in a single place, we can open-code it to
make it more transparent what logic is used.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use a single binary search to find the start of the matching encoded-value
prefix range. The subsequent scan must already inspect the complete range to
detect ambiguity and already filters by algorithm, making the second
algorithm-aware search redundant.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Range over entries directly and refactor the loop to make the
prefix extension and completion logic easier to follow.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>

@sudo-bmitch sudo-bmitch left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking suggestions. I think everything in here looks good even without my suggested changes, but I'm less familiar with this package. Separate from this PR, I'd like to throw a fuzzer at it, to make sure user input can't trigger a panic.

Comment thread digestset/set.go
Comment on lines +80 to +87
if dgst, err := digest.Parse(d); errors.Is(err, digest.ErrDigestInvalidFormat) {
// An input without a valid algorithm separator is treated as an
// unqualified encoded-value prefix.
hexPrefix = d
} else {
hex = dgst.Encoded()
// digest.Parse still returns the parsed algorithm and encoded value for
// qualified short digests, together with digest.ErrDigestInvalidLength.
hexPrefix = dgst.Encoded()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels fragile to me. Particularly if digest.Parse throws an unexpected error in the future that doesn't populate dgst with a value containing a colon, we could trigger a panic. Instead of parsing the string, could we make the condition if ind := strings.Index(d, ":"); ind >= 0?

Suggested change
if dgst, err := digest.Parse(d); errors.Is(err, digest.ErrDigestInvalidFormat) {
// An input without a valid algorithm separator is treated as an
// unqualified encoded-value prefix.
hexPrefix = d
} else {
hex = dgst.Encoded()
// digest.Parse still returns the parsed algorithm and encoded value for
// qualified short digests, together with digest.ErrDigestInvalidLength.
hexPrefix = dgst.Encoded()
if i := strings.Index(d, ":"); i >= 0 {
alg = digest.Algorithm(d[:i])
hasPrefix = d[i+1:]
} else {
hasPrefix = d
}

(Note GitHub doesn't let me select the next two lines in the review)

Comment thread digestset/set.go
if dst.entries[i].alg == alg && dst.entries[i].val == hex {
// An exact whole-value match is unambiguous.
return dst.entries[i].digest, nil
if alg != "" && entry.val == hexPrefix {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be more clear to do an exact match on the full digest string? This could then be moved up to the first condition in the loop to cover a common use case.

Suggested change
if alg != "" && entry.val == hexPrefix {
if entry.digest.String() == d {

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants