Quantization improvements#87
Conversation
This reverts commit fdd5144.
…ization and distance calculations Updated comments in quantizer.cpp and space.cpp to include licensing information and attribution to Elasticsearch's Java implementation. Added references to the RaBitQ paper for algorithm context.
Quantization improvements
There was a problem hiding this comment.
Pull Request Overview
This PR refactors quantization functionality and updates several APIs to support improved quantization settings and binary space representations. Key changes include:
- New quantization-related functions and API modifications in util_private and knn modules.
- Updates to file mapping (reader) methods to support write modes.
- Adjustments to index building and distance calculation interfaces for improved quantization support.
Reviewed Changes
Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| util/util_private.h | Added CalcNorm and templated vector distance functions; updated NormalizeVec signature. |
| util/util_private.cpp | Updated NormalizeVec implementation to use division factor and return norm. |
| util/util.h | Introduced a new resize function implementation using memset. |
| util/reader.{h,cpp} | Modified file open methods to support write access by adding a bWrite parameter. |
| secondary/builder.cpp | Updated call to Open with new bWrite parameter. |
| knn/space.h | Modified SetQuantizationSettings signature and updated data size calculations. |
| knn/quantizer.h | Revised Encode signature to include error handling and added GetPoolFetcher. |
| knn/knn.{h,cpp} | Updated quantization settings I/O and modified distance/index building APIs accordingly. |
|
|
||
| protected: | ||
| float CalcAlpha ( const QuantizationSettings_t & tSettings ) const override { return ( tSettings.m_fMax-tSettings.m_fMin ) / 15.0; } | ||
| size_t get_data_size() override { return ( (m_uDim+3)>>1 ) + sizeof(float); } |
There was a problem hiding this comment.
[nitpick] The modified get_data_size() now adds sizeof(float) to the computed size; please ensure this addition is thoroughly documented and intentional since it mixes element count with a size in bytes.
| size_t get_data_size() override { return ( (m_uDim+3)>>1 ) + sizeof(float); } | |
| size_t get_data_size() override { | |
| // The addition of sizeof(float) accounts for extra space required for metadata or alignment. | |
| // This ensures compatibility with the internal data representation. | |
| return ( (m_uDim+3)>>1 ) + sizeof(float); | |
| } |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Linux debug test results 8 files 8 suites 11m 20s ⏱️ Results for commit e70a5a5. ♻️ This comment has been updated with latest results. |
Linux release test results 8 files 8 suites 5m 57s ⏱️ Results for commit e70a5a5. ♻️ This comment has been updated with latest results. |
Windows test results 5 files 5 suites 15m 39s ⏱️ Results for commit e70a5a5. ♻️ This comment has been updated with latest results. |
…added and small vectors
previous builds, however, still depend on master
|
|
also use special cache for x86-64-v3
configures and builds columnar for x86-64-v3 arch (avx2), and adds built artefacts to package/debugsymbols.
on clang, _mm_popcnt_u32 expanded to __builtin_popcount, that makes PopCnt32 useless (both branches are the same). Also, 'FORCE_INLINE' doesn't work by unknown reason. Build fails on 'unity' flavour, when about all files 'glued' together, and we have finally two similar definitions in unified file.
we don't need second copy from external project
streamvbyte and hnsw was't build on our current cmake
This update introduces a step in the GitHub Actions workflows to determine the branch name for caching purposes. The branch name is set based on whether the event is a pull request or a regular push, allowing for more granular cache keys in the build and test workflows. This change is applied across build_template.yml, test_template.yml, and win_test_template.yml.
This update introduces print statements in the GitHub Actions workflows to display the value of the MANTICORE_LOCATOR input. It provides feedback on whether a custom locator is set or if the default sources will be used, enhancing visibility during the build and test processes. Changes are applied across build_template.yml, test_template.yml, and win_test_template.yml.
This update adds a step to determine the branch name for caching in the Windows workflow. The cache key is dynamically set based on whether the event is a pull request or a regular push.
Quantization improvements