DocBin.to_bytes fails with a "ValueError: bytes object is too large"

[kuppulur]

How to reproduce the behaviour

I am trying to train a sense2vec model from scratch on a corpus which has around 5 million sentences (lines/docs). As the first step is to parse it and create a .spacy file, I ran the parse script on this corpus and the code crashes at doc_bin_bytes = doc_bin.to_bytes() with a value error saying the bytes object is too large. Can somebody help me with this issue? Thanks

Your Environment

• Operating System: Ubuntu 16.04.6 LTS
• Python Version Used: 3.5.2
• spaCy Version Used: 2.2.4
• Environment Information:

[svlandeg]

Do I understand it correctly that each line is parsed as a separate Doc? So you have 5M Docs in one DocBin? Can you paste the entire error trace?

[kuppulur]

@svlandeg yes that is correct. Please find the error trace below.

[kuppulur]

@svlandeg any pointers on this issue? Thanks

[svlandeg]

Apologies for the late follow-up.

This seems related to Issue explosion/sense2vec#102, which was addressed in PR explosion/sense2vec#103.
So basically what you want to do in your code, is to use & store several doc_bin objects, instead of just one. Hope that works for you!

[github-actions]

This issue has been automatically closed because it was answered and there was no follow-up discussion.

[github-actions]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.