Error when using InMemoryDocumentStore()

[raghav-menon]

I had installed haystack and was trying to use inMemoryDocumentStore(). After initializing the document store, when I try to index it I receive the following error

---

ValueError Traceback (most recent call last)
in ()
----> 1 document_store.write_documents(dicts)

/usr/local/lib/python3.7/dist-packages/haystack/document_store/memory.py in write_documents(self, documents, index)
84 if document.id in self.indexes[index]:
85 # TODO Make error type consistent across document stores and add user options to deal with duplicate documents (ignore, overwrite, fail)
---> 86 raise ValueError(f"Duplicate Documents: write_documents() failed - Document with id '{document.id} already exists in index '{index}'")
87 self.indexes[index][document.id] = document
88

ValueError: Duplicate Documents: write_documents() failed - Document with id '4cdbad5973235417ea3e71769dc9c9ae already exists in index 'document'

---

I had tried this previously (about a month back) it worked perfectly well. Not sure why this is arising. Would be grateful if you could point me in the right direction.

Thanks

Raghav

[tholor]

Hey,

We recently introduced in #1000 a basic hash mechanism that prevents writing duplicate documents to the documentstore. This was a common source of error and confusion for users in the past. Per default we generate the ID by hashing the documents text - you can customize this to other fields (see #1000).

In #1088 we will improve the user options to specify what happens in the case of duplicates (skip, overwrite, fail...).

So your options are:

• make sure you don't pass duplicate documents
• pass your own custom IDs for your docs (see Improve handling of duplicate documents during write_documents() #1069 (comment))
• wait for Add options for handling duplicate documents (skip, fail, overwrite) #1088

[raghav-menon]

Thank you. Tried the workaround and it worked.

Regards.

[Timoeller]

Nice, can we close this issue then?

[raghav-menon]

Sure we can close the issue. Thanks and Regards