Add deduplication logic

[PaliC]

This pull request introduces deduplication functionality to the export.py script and updates the documentation to include testing instructions. The key changes include integrating a deduplication module, handling deduplicated datasets, and providing a detailed test setup for verifying the deduplication logic.

The deduplication functionality is specifically does 3 things

1. It bins the submissions by success, run mode, and score (if available) or duration of the run. We then deduplicate within these bins as minhash takes a while (deduping this dataset took an hour)
2. We first dedup using a hash
3. We then dedup using minhash lsh (this is what takes a while). You can find an explanation of this process here https://medium.com/@omkarsoak/from-min-hashing-to-locality-sensitive-hashing-the-complete-process-b88b298d71a1

Deduplication Integration:

• Added the dedup_df function from the dedup module to deduplicate submissions in the dataset. The deduplicated data is saved as a separate Parquet file, and the process is logged with details about the number of records before and after deduplication. (export.py, [1] [2]
• Extended the script to create and save a deduplicated version of successful submissions, including logging the count of records pre- and post-deduplication. (export.py, export.pyR235-R243)

Documentation Updates:

• Added a "Tests" section in the README.md file, describing how to test the deduplication scripts using a fake dataset with various features such as exact duplicates, fuzzy duplicates, and realistic structured data. (README.md, README.mdR37-R51)

[msaroufim]

Thanks! Mind sharing some more stats on the real dataset?

1. How much data was around before your change
2. How much data is around after
3. Some manual vibe checks as well of kernels that were filtered out would be nice to sanity check

Code is quite long to review but at least above should give us some more confidence before merge

[ngc92]

I think scores are still lower = better; run.duration is the end-to-end wallclock time for the entire run, including, e.g., testing code, whereas score is the geomean of all benchmarks

[PaliC]

oops, I see. I'll rerun things and reupload

[PaliC]

@msaroufim for the first two questions we have these

✓ Loaded submissions.parquet: 40,095 entries
✓ Loaded deduplicated_submissions.parquet: 15,238 entries

============================================================
COUNT COMPARISON

Original submissions: 40,095
Deduplicated submissions: 15,238
Removed entries: 24,857
Percentage removed: 62.00%

So it seems like a lot of the kernels are actually similar. I'll rerun things and grab some files for a sanity check

[PaliC]

@msaroufim https://www.diffchecker.com/KamzTAeT/ (I think this one is more similar) and https://www.diffchecker.com/MtK5pbWL/ show a entry which was removed due to depulicaton (on right) compared to two other entries that remained in the dataset. We can be a bit less aggressive with deduping, as they look sortof different.