Use rapidhashNano on folly::hasher<string/range> #2448
+1,342
−102
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
55d0af9 to
b91daf4
Compare
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
b91daf4 to
44c47aa
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: Pull Request resolved: facebook#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
44c47aa to
eb3aca2
Compare
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: Pull Request resolved: facebook#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
2 times, most recently
from
7ea999f to
7e93613
Compare
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/hhvm#9617 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
7e93613 to
373feb1
Compare
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/hhvm#9617 Pull Request resolved: facebook#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
373feb1 to
dd47609
Compare
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 5, 2025
Summary: Pull Request resolved: facebook#9617 X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
dd47609 to
bdc3669
Compare
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/hhvm#9617 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/hhvm#9617 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
bdc3669 to
171c287
Compare
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/hhvm#9617 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
f0b18b4 to
219ca40
Compare
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 5, 2025
Summary: Pull Request resolved: facebook#9617 X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 5, 2025
Summary: Pull Request resolved: facebook#9617 X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
force-pushed
the
export-D76052916
branch
from
219ca40 to
0447249
Compare
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 5, 2025
Summary: X-link: facebook/hhvm#9617 Pull Request resolved: facebook#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
0447249 to
225b8e7
Compare
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 6, 2025
Summary: X-link: facebook/hhvm#9617 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 6, 2025
Summary: X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
force-pushed
the
export-D76052916
branch
from
225b8e7 to
ad2da75
Compare
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 6, 2025
Summary: X-link: facebook/hhvm#9617 Pull Request resolved: facebook#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 6, 2025
Summary: Pull Request resolved: facebook#9617 X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 7, 2025
Summary: X-link: facebook/hhvm#9617 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
ad2da75 to
48ce311
Compare
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 7, 2025
Summary: X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 7, 2025
Summary: X-link: facebook/hhvm#9617 Pull Request resolved: facebook#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
48ce311 to
2195dee
Compare
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 7, 2025
Summary: Pull Request resolved: facebook#9617 X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Summary: We've seen high cpu usage on these two hashing algorithms: std::_Hash_bytes and multifeed/common/Hash.h::hashBytesImpl For the record, std::_Hash_bytes compiles to ~60 instructions on aarch64 and ~100 instructions on AMD64: https://godbolt.org/z/xeoqf1aaE hashBytesImpl compiles to slightly over 100 instructions on aarch64 and slightly over 160 instructions on AMD64: https://godbolt.org/z/bTroqGE7o The diff adds three new hash functions: rapidhash, rapidhashMicro and rapidhashNano RapidhashNano is designed for situations where keeping a small code size is a top priority. Clang-19 compiles it to less than 100 instructions without stack usage, both on x86-64 and aarch64. The fastest for sizes up to 48 bytes, but may be considerably slower for larger inputs. RapidhashMicro is designed for situations where cache misses make a noticeable performance detriment. Clang-19 compiles it to ~140 instructions without stack usage, both on x86-64 and aarch64. Faster for sizes up to 512 bytes, just 15%-20% slower for inputs above 1kb. rapidhash provides formidable speed across all input sizes Clang-19 compiles it to ~185 instructions, both on x86-64 and aarch64. Benchmark results on BGM: P1826606121, and Grace: P1826591223 On AMD64, RapidhashNano should be strictly better than both std::_Hash_bytes and hashBytesImpl On aarch64, std::_Hash_bytes compiles to fewer instructions. RapidhashNano should still be faster in most situations, given its much higher throughput. It should also be strictly better than hashBytesImpl In many situations, RapidhashMicro should be a better choice, due to its higher throughput. This diff allows us to analyze workloads on a case by case basis. rapidhash seems to be the fastest high-quality hash function for aarch64 systems. It may still find usage on large-input cases. Folly's benchmark results have been updated to include runs from Bergamo and Neoverse-V2 Differential Revision: D66326393
Nicoshev
added a commit
to Nicoshev/folly
that referenced
this pull request
Jun 9, 2025
Summary: X-link: facebook/hhvm#9617 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Nicoshev
force-pushed
the
export-D76052916
branch
from
2195dee to
7aeaeeb
Compare
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 9, 2025
Summary: X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Summary: X-link: facebook/hhvm#9617 Pull Request resolved: facebook#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
|
This pull request was exported from Phabricator. Differential Revision: D76052916 |
Nicoshev
force-pushed
the
export-D76052916
branch
from
7aeaeeb to
aeeb809
Compare
Nicoshev
added a commit
to Nicoshev/hhvm
that referenced
this pull request
Jun 9, 2025
Summary: Pull Request resolved: facebook#9617 X-link: facebook/folly#2448 Replacing SpookyHashV2 with rapidhashNano folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3 Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries Differential Revision: D76052916
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Replacing SpookyHashV2 with rapidhashNano
folly::hasher::operator() accounts for almost 3M$ in $cpu_t1_equiv_per_year_q2_2025 https://fburl.com/strobelight/izute4k3
Given that integral hashing is the identity function, most of the registered cycles should come from strings/byteRanges
See D66326393 and D75697257 for a detailed discussion around benchmarks and canaries
Differential Revision: D76052916