Using text hash as id to prevent document duplication

[lalitpagaria]

Proposed changes:

This PR is in response to slack discussion.

New haystack users most of the time encounter duplicate answer in their result. This happen due to duplicate ingestion of same text passage multiple time because each time haystack generate new id via uuid. In order to prevent this we will be using text hash as id to prevent document duplication. Also providing a way customize it where user pass value such that cleaned text, file hash, paragraph number or page no to customize id generation via hashing. Fast hashing algorithm MurmurHash is being used for 128 bit hash generation.

Status (please check what you already did):

• First draft (up for discussions & feedback)
• Final code
• Added tests
• Updated documentation

[lalitpagaria]

Not sure why these changes are in this PR. I have not added these

[lalitpagaria]

@oryx1729 @tholor
Can you please review.
Test is failing to some unrelated error (failing to download model Helsinki-NLP/opus-mt-en-de).
I think rerun of test will solve this.

[tholor]

Thanks, @lalitpagaria . I will review it later today.

BTW this is #1000 🎉

[tholor]

I like the implementation. Looks easy and solid.

One thing that is not clear to me yet: What's the behaviour if I add now a second "duplicate" document with the same hash? Will I replace the existing doc with the new one or will I ignore the new doc? Is this behavior consistent across all document stores? Probably we can cover it by adding a few additional test cases and ideally adding a warning message in write_document() to inform users about duplicates.

(Sorry didn't have time today to test the behavior myself)

[lalitpagaria]

I have added a test for that. But there is constancy issue -

• Memory store - Allow duplicate
• ES based store - Throw BulkIndexError
• SQL based store - Throw IntegrityError exception because of UNIQUE constraint

Even though I added the test but I think better to make memory store also constant ie throw exception in case of writing document with duplicate id.

Check here to know what happen when we write documents with duplicate ids https://github.com/deepset-ai/haystack/runs/2442736307

[tholor]

Thanks for checking the behavior and adding the test. We should definitely make it consistent across doc stores.

I think just throwing an exception is not an ideal user experience here. Imagine I add 100 docs in a batch via write_documents() and one of them is a duplicate. I will receive the error but how should I go on? How can I index the other 99 docs?

IMO it would be nicer to issue a warning including the problematic duplicate document and make sure that the rest of the documents get indexed correctly. What do you think @lalitpagaria @oryx1729 ?

[lalitpagaria]

I agree with you @tholor
In this case we need to provide three options in write_documents function -

1. Ignore duplicate (with warning) - Default option
2. Overwrite if exist
3. Fail if duplicate

In all cases write_documents should return inserted document ids (same in case of rest API). So user knows what was written and what was skipped.

We can tackle this in separate PR as resultant changes will be bigger, also handling these in ES, memory and SQL based store would be different.

[lalitpagaria]

@tholor @oryx1729 Please let me know if anything needed in this PR.

[tholor]

I am fine with tackling the above behavior in a separate PR. However, let's at least make sure that the Documentstore's have consistent behavior in the meantime. So I'd suggest:

• throw exception also in memorystore (e.g. by checking ids before inserting docs)
• throw the same exception across all stores so that it can easily be catched

[lalitpagaria]

I agree this make sense -

throw exception also in memorystore (e.g. by checking ids before inserting docs)

Implementing this is not easy as elastic search always return bulkexception even in case of network error during transaction. Similarly SQL store can throw another constraint error during write documents function.

throw the same exception across all stores so that it can easily be catched

[tholor]

Ready to merge.
We will address the aforementioned limitations in a separate PR.