Skip to content

Only check for pending and started uploads#218

Merged
jjnesbitt merged 2 commits intomainfrom
fix-upload
Mar 10, 2022
Merged

Only check for pending and started uploads#218
jjnesbitt merged 2 commits intomainfrom
fix-upload

Conversation

@jjnesbitt
Copy link
Copy Markdown
Member

@jjnesbitt jjnesbitt commented Feb 11, 2022

Closes #216

Previously, the upload in progress spinner only checked if the status was FINISHED. FAILED is possible as well, and would result in an infinite spinner. This updates that check.

@netlify
Copy link
Copy Markdown

netlify bot commented Feb 11, 2022

✔️ Deploy Preview for multinet-client ready!

🔨 Explore the source changes: ad9ecd0

🔍 Inspect the deploy log: https://app.netlify.com/sites/multinet-client/deploys/622a59edea910700081566b1

😎 Browse the preview: https://deploy-preview-218--multinet-client.netlify.app/

@JackWilb
Copy link
Copy Markdown
Member

JackWilb commented Feb 14, 2022

On very large uploads I've also ran into times when celery would run out of memory and crash with the upload stuck in the 'started' state. This is likely not the best place to fix that, but I wanted to make a note of it

Should I open an issue in the API repo?

@jjnesbitt
Copy link
Copy Markdown
Member Author

On very large uploads I've also ran into times when celery would run out of memory and crash with the upload stuck in the 'started' state. This is likely not the best place to fix that, but I wanted to make a note of it

Hmm, have you tested this after the chunking change? multinet-app/multinet-api#90

In either case, that kind of surprises me, since it'd have to be quite a large file to run out of memory. Do you remember how large the file was?

@JackWilb
Copy link
Copy Markdown
Member

Hmm, have you tested this after the chunking change? multinet-app/multinet-api#90

I have. The problem is that even though the inserts are chunked into Arango, they're not chunked into the celery worker, it has to open the whole file first. That's what leads to the the crash.

In either case, that kind of surprises me, since it'd have to be quite a large file to run out of memory. Do you remember how large the file was?

On the order of hundreds of megabytes, so quite large.

@jjnesbitt jjnesbitt marked this pull request as ready for review March 10, 2022 17:55
@jjnesbitt jjnesbitt requested a review from JackWilb March 10, 2022 17:55
@jjnesbitt jjnesbitt merged commit e0b46d7 into main Mar 10, 2022
@jjnesbitt jjnesbitt deleted the fix-upload branch March 10, 2022 22:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Upload silently fails for example dataset

2 participants