Add HNSW vector storage

[NumberChiffre]

Description

TLDR; fast querying with slow insertion, using hnswlib faster than the HNSW from faiss implementation.

Updates

• Added HNSWVectorDBStorage.
• Added basic unit tests.
• Added basic benchmarking against NanoVectorDBStorage.
• Added an example under examples with GraphRAG (Easily triggering rate limit but unrelated to this PR).

Benchmark results

> python benchmarks/hnsw_vs_nano_vector_storage.py 
Running NanoVectorDB benchmark...
INFO:nano-graphrag:Creating working directory ./nano_graphrag_cache_benchmark_hnsw_vs_nano_vector_storage
INFO:nano-graphrag:Load KV full_docs with 0 data
INFO:nano-graphrag:Load KV text_chunks with 0 data
INFO:nano-graphrag:Load KV llm_response_cache with 0 data
INFO:nano-graphrag:Load KV community_reports with 0 data
INFO:nano-vectordb:Init {'embedding_dim': 1024, 'metric': 'cosine', 'storage_file': './nano_graphrag_cache_benchmark_hnsw_vs_nano_vector_storage/vdb_entities.json'} 0 data
INFO:nano-vectordb:Init {'embedding_dim': 1024, 'metric': 'cosine', 'storage_file': './nano_graphrag_cache_benchmark_hnsw_vs_nano_vector_storage/vdb_benchmark_nano.json'} 0 data
Benchmarking nano...
nano Benchmark:   0%|                                                                                                                                                                                                      | 0/100000 [00:00<?, ?it/s]INFO:nano-graphrag:Inserting 100000 vectors to benchmark_nano
nano Benchmark: 100101it [00:04, 22470.39it/s]                                                                                                                                                                                                        
nano - Insert: 1.44s, Save: 1.81s, Avg Query: 0.0120s

Running HNSWVectorStorage benchmark...
INFO:nano-graphrag:Load KV full_docs with 0 data
INFO:nano-graphrag:Load KV text_chunks with 0 data
INFO:nano-graphrag:Load KV llm_response_cache with 0 data
INFO:nano-graphrag:Load KV community_reports with 0 data
INFO:nano-vectordb:Init {'embedding_dim': 1024, 'metric': 'cosine', 'storage_file': './nano_graphrag_cache_benchmark_hnsw_vs_nano_vector_storage/vdb_entities.json'} 0 data
INFO:nano-graphrag:Created new index for benchmark_hnsw
Benchmarking hnsw...
hnsw Benchmark:   0%|                                                                                                                                                                                                      | 0/100000 [00:00<?, ?it/s]INFO:nano-graphrag:Inserting 100000 vectors to benchmark_hnsw
hnsw Benchmark: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████hnsw Benchmark: 100101it [01:24, 1187.70it/s]                                                                                                                                                                                                         
hnsw - Insert: 83.67s, Save: 0.44s, Avg Query: 0.0017s

Benchmark Results:
NanoVectorDB - Insert: 1.44s, Save: 1.81s, Avg Query: 0.0120s
HNSWVectorStorage - Insert: 83.67s, Save: 0.44s, Avg Query: 0.0017s

[gusye1234]

Great Work and thank you!
I'm curious, does hnswlib support Windows?

[NumberChiffre]

Great Work and thank you! I'm curious, does hnswlib support Windows?

No access to Windows on my part, but it should work:
https://github.com/nmslib/hnswlib?tab=readme-ov-file#for-developers

Can confirm this works on M3 Mac and worked on Ubuntu 20.04.

**Why don't we use docker containerization with docker, docker-compose, and even poetry?

[gusye1234]

Great Work and thank you! I'm curious, does hnswlib support Windows?

No access to Windows on my part, but it should work: https://github.com/nmslib/hnswlib?tab=readme-ov-file#for-developers

Can confirm this works on M3 Mac and worked on Ubuntu 20.04.

**Why don't we use docker containerization with docker, docker-compose, and even poetry?

Cool! Few responses:

• I don't think add a docker image for current nano-graphrag is urgent. This project should be nano and light-weight, not something must run in a container.
• Is poetry a better option for dependency management? I'm really lack of understanding of this. Maybe we should move to poetry.

[NumberChiffre]

Great Work and thank you! I'm curious, does hnswlib support Windows?

No access to Windows on my part, but it should work: https://github.com/nmslib/hnswlib?tab=readme-ov-file#for-developers
Can confirm this works on M3 Mac and worked on Ubuntu 20.04.
**Why don't we use docker containerization with docker, docker-compose, and even poetry?

Cool! Few responses:

• I don't think add a docker image for current nano-graphrag is urgent. This project should be nano and light-weight, not something must run in a container.
• Is poetry a better option for dependency management? I'm really lack of understanding of this. Maybe we should move to poetry.

Sounds good!

poetry works well with docker and in general is more effective than pip (handling complex dependency trees more effectively than pip).

*** I'm done with commits for this PR, let me know what you think 👍

[gusye1234]

Cool, I have only one review about this PR and maybe you can look at it.
Then we're all set.

[gusye1234]

LGTM

[gusye1234]

Why unpack vector_db_storage_cls_kwargs here?
You get this dict from global_config in HNSWStorage.
Unpack vector_db_storage_cls_kwargs here maybe break other vector storage's initialization.

[gusye1234]

It's fixed by 20219d0

[zhaofangtao]

The library "hnswlib" can be installed and used in the Windows 11 environment.