Conversation
|
✔️ Deploy Preview for multinet-client ready! 🔨 Explore the source changes: ad9ecd0 🔍 Inspect the deploy log: https://app.netlify.com/sites/multinet-client/deploys/622a59edea910700081566b1 😎 Browse the preview: https://deploy-preview-218--multinet-client.netlify.app/ |
|
On very large uploads I've also ran into times when celery would run out of memory and crash with the upload stuck in the 'started' state. This is likely not the best place to fix that, but I wanted to make a note of it Should I open an issue in the API repo? |
Hmm, have you tested this after the chunking change? multinet-app/multinet-api#90 In either case, that kind of surprises me, since it'd have to be quite a large file to run out of memory. Do you remember how large the file was? |
I have. The problem is that even though the inserts are chunked into Arango, they're not chunked into the celery worker, it has to open the whole file first. That's what leads to the the crash.
On the order of hundreds of megabytes, so quite large. |
Closes #216
Previously, the upload in progress spinner only checked if the status was
FINISHED.FAILEDis possible as well, and would result in an infinite spinner. This updates that check.