Looking for collaborators

[ekzhu]

I am looking for open-source collaborators for the datasketch project.

I created datasketch 10 years ago to open source the code I wrote for my PhD work in dataset search. Many great folks helped build this with me. You can find their contributions.

Now, it is downloaded 4.5M times per month on PyPI -- driving about 1% of all numpy downloads. At this point, datasketch is a foundational block of the Python ecosystem.

Today, I am spending most of my time building AI agent systems so I need help modernizing this project. There are several things I would love to do but need help with:

• Modernize the build using pyproject setup and uv
• Release version selector for Sphinx-generated doc
• Async API for indexes -- moving them out of experimental.
• Refactor the storage backend abstraction to better support batch loading directly from disk file utilizing the underlying storage's API
• Better benchmark utility and comparison of different arithmetic tricks to speedup MinHash signature computation and improved compression.
• Better benchmark utility to measure performance of various storage backends, and propose optimization.
• GPU-based algorithms.

The goal is to make datasketch more efficient and scalable, while keeping the API stable.

Don't direct message me, just comment on this thread, and create an issue to address one of the above topics and we can discuss there. I will create a collaborator team and add folks to it as appropriate.

Please, no AI slops. I love using AI coding agents but I always review AI-generated code carefully and communicate with human directly. Don't overburden reviewers with massive code changes just because you can now generate code.

[KilimAnnejaro]

I am interested in part #1, "Modernize the build using pyproject setup and uv". I will create an issue for this.

[ekzhu]

Thank you. Looking forward to it.

[bhimrazy]

Hi @ekzhu , I’d love to help out with some CI improvements and get familiar with the project. Once I’m up to speed, I can contribute to other areas as well. Looking forward to collaborating!

[bhimrazy]

Sure, thanks.

[ekzhu]

@bhimrazy thanks a lot for the recent patches and code reviews. I took a look into the permission configuration of personal projects on GitHub and I think it is a quite challenging to use it for open source collaboration with a lot of people. So I think it is better to keep the permission setup the way it is right now, and we can migrate to organization when it becomes obvious that there will be multiple long-term maintainers.

[bhimrazy]

That makes sense — thanks for looking into it, @ekzhu!

[ekzhu]

@bhimrazy do you have an x.com and/or LinkedIn handle? I would like to post about your contribution

[bhimrazy]

Yes, @ekzhu!
X: https://x.com/bhimrazyadav
LinkedIn: https://www.linkedin.com/in/bhimrazy

Thank you so much for this — really appreciate it! 🙏

[sarang-pratham]

Hi @ekzhu
Created a PR for the [Modernize the build using pyproject setup and uv] #254

This is my first contribution (ever on github) to the best of my knowledge.
looking forward for feedback.

[dipeshbabu]

Hey @ekzhu, GPU-based algorithms sounds exciting - I would love to contribute to it.

[ekzhu]

The idea is cool. I have a few comments:

1. Some discussion/exploration needs to be done on the minhash values obtained from different backends. I assume value hashing is still done on CPU but the critical code that computes minhash values will be on GPU -- need to understand whether there will be discrepancies.
2. Let's start with smaller PRs. The first one can address MinHash's update_batch only.
3. Look at some reference implementations regarding Weighted MinHash: https://github.com/src-d/minhashcuda?utm_source=chatgpt.com

[dipeshbabu]

The idea is cool. I have a few comments:

1. Some discussion/exploration needs to be done on the minhash values obtained from different backends. I assume value hashing is still done on CPU but the critical code that computes minhash values will be on GPU -- need to understand whether there will be discrepancies.
2. Let's start with smaller PRs. The first one can address MinHash's update_batch only.
3. Look at some reference implementations regarding Weighted MinHash: https://github.com/src-d/minhashcuda?utm_source=chatgpt.com

Sounds good! I will open PRs then.

[dipeshbabu]

@ekzhu could you review the pr?

[ekzhu]

Apologies for the delay. Just made comments

[dipeshbabu]

@ekzhu I just wanted to confirm is anyone working on this: Refactor the storage backend abstraction to better support batch loading directly from disk file utilizing the underlying storage's API? If not, I can continue on this.

[benita-chinemerem]

Hi Ekzhu, I’d love to start by working on benchmarking the performance of different storage backends and exploring possible optimizations. I’ll create an issue to discuss my approach and initial findings. This is going to be my first ever open source contribution too

[ekzhu]

Sounds good. Please also check out other on-going PRs and discussions that may be related to your interest.

[yudhiesh]

@ekzhu I have started work on a new benchmark framework in this PR would love to get some feedback on the initial version which handles fetching and caching data to Numpy Memory Mapped arrays. I have detailed the next steps to carry out and the framework I am following.

[ekzhu]

Thank you! I will take a look.

[rohanpoudel2]

@ekzhu , I would like to work on Better benchmark utility to measure performance of various storage backends, and propose optimization. It sounds interesting.