Improve handling of duplicate documents during write_documents()

[tholor]

Is your feature request related to a problem? Please describe.
In #1000 we introduced a hashing mechanism that creates the file id depending on document content (default: the text).

The current behaviour:

If you call write_documents() with a batch of documents and one of them is already existing in the document_store (determined by the ID), the request is failing:

Memory store - ValueError
ES based store - Throw BulkIndexError
SQL based store - Throw IntegrityError exception because of UNIQUE constraint

Describe the solution you'd like
Add an argument handle_duplicates to write_documents() that allows users to change the behavior in case of duplicates:

• Ignore duplicate (with warning) - Default option
• Overwrite if exist
• Fail if duplicate

In all cases write_documents should return inserted document ids (same in case of rest API). So the user knows what was written and what was skipped.

Ideally, we also make the thrown exceptions more consistent across document stores.

[venuraja79]

Just noticed this, will include this for Weaviate store once the method signature is decided.

[akkefa]

@tholor There is update_existing_documents arg in every document_store constructor, Which looks like added to handle the duplicate functionality. It's conflicting with the current implementation handle_duplicates I m working on.

E.g
If self.update_existing_documents = True and doc_store.write_documents(handle_duplicates="Fail") then Which option to used?

[tholor]

Yep, good point. I'd suggest we rename both arguments to duplicate_documents with options skip, overwrite + fail and use a similar pattern as for some of the other arts (e.g top_k). You can set the behavior when initializing the documentstore and overwrite it optionally in the method (e.f. for top_k in query())

[akkefa]

@tholor There are two types of duplicates. The first one when hash ID exists in the Database and 2nd are those when we are creating a list of Document object from Dicts.

E.g

{'text': '\n==References==', 'score': None, 'probability': None, 'question': None, 'meta': {'name': '133_Game_of_Thrones__Season_5__soundtrack_.txt'}, 'embedding': None, 'id': '7ed12f389f7f085bb30c7d00abd26f81'} {'text': '\n==References==', 'score': None, 'probability': None, 'question': None, 'meta': {'name': '25_Game_of_Thrones__Season_2__soundtrack_.txt'}, 'embedding': None, 'id': '7ed12f389f7f085bb30c7d00abd26f81'}

Should we drop those duplicates when we creating document_objects based on unique Hash Id.

[vblagoje]

Hey guys, I just noticed that the notebook tutorial with game of thrones documents started to fail on write_documents. Is there a workaround for the time being?

[lalitpagaria]

add following lines before document_store.write_documents(dicts) till proper fix -

import uuid
for doc in dicts:
    doc["id"] = str(uuid.uuid4())