Document ID match

[Justinius]

I'm working on uploading files, and I ran into a case where two PPT files had the same text content but had other differences when opened in PowerPoint. However, since the text was the same it was marked as a duplicate.

The immediate problem was that the track id for the "duplicate" didn't resolve to anything, so there was no way to know what happened through the api.

Talking about this with a colleague, we thought maybe the doc id could be based on the md5 of the whole file and not just the text content. You would have duplicate chunks, and everything, but it is an option. The other idea was possibly adding a status for something like "duplicate" and be like a failure - don't insert/chunk/etc, but the track id would resolve, and a status would be available for review.

Anyone else run into issues like this? Willing to hear more feedback on this and how to possibly handle this.

[Justinius]

Status: Open.
#2468 - this is the bug report that was already opened by someone else, but I think overlaps with my need. I tested the update from the mentioned PR 2469 and updated that Issue.

For exact matches (name and content) - it seems to return duplicate
For content only duplicate - it moves the file over to enqueued, returns success with a tracking id, but the tracking ID returns nothing in the document info or status summary