Ingest: Multiple (repeated) ingests of tabular data files

[landreev]

Back in 2018 we have enabled ingest on tab-delimited text files. (This was done as part of the CSV ingest improvement; by reusing the same parsing code, but with the TAB for the delimiter character).

Apparently, we now have a condition where a successfully ingested file gets picked up for ingest AGAIN - because the content type is tab-delimited, and tab-delimited files are now ingestable... this of course should never happen, because the file already has a datatable object associated with it. But apparently it does occasionally, and an ingested file gets ingested again, corrupting the tab file and the saved original in the process.

I have a list. There are relatively few of these cases, but this is still very annoying and I believe we should treat it as urgent.

[djbrooke]

@landreev I moved this into the ready column and we'll take it on in a sprint soon. Just to clarify, this would be a code change and some cleanup?

[landreev]

@landreev I moved this into the ready column and we'll take it on in a sprint soon. Just to clarify, this would be a code change and some cleanup?

Correct.

[landreev]

This doesn't seem to have affected any older files - meaning, any tabular files ingested prior to the introduction of ingest of text/tab-separated-values. So I'm going to guess that it's something that only happens when the initial ingest session goes wrong and starts an extra iteration - ?
Meaning, my initial "relatively few of these cases" statement appears to still be true.

[landreev]

The remaining tasks are mostly cleanup.
In the emergency PR that was merged yesterday (#6517) extra code was added that would prevent an ingest process from being performed again, should a datafile with an existing datatable be placed in the ingest queue again.
The original plan was to maintain 2 distinct mime types, "text/tab-separated-values" for ingested files, and "text/tsv" for uningested text files that just happen to be tab-delimited; with only the latter treated as potentially ingestable. Unfortunately it wasn't being done consistently, but the PR above addresses that as well.
It also fixed an unrelated (but annoying) issue, where some uningested tab-delimeted files were mistakenly showing with "explore" buttons. (I thought there was an already opened issue for that, but couldn't find one).

[landreev]

All that said, I still have no idea how it was possible, for an ingested file to be added to the processing queue again. Our system of status flags - separate flags for "ingest scheduled" and "ingest in progress" - was supposed to prevent that. But there must be some ways in which things can go wrong that could make it possible... a page operating on a stale copy of the dataset, allowing to bypass an existing ingest lock? a system crash leaving the flags and the queue in an inconsistent state? something specific to how ingest queue is activated when files are uploaded via the API? ... Whatever it is, it is apparently possible. But if it does happen again, the added checks, and the fact that the ingested mime format is no longer considered ingestable, should prevent any repeated processing from being done.

[djbrooke]

• We should add in some more code to allow tsvs and text/tab delimited to be ingested when using the browser (carefully!). This can be defined using the API but when a browser is used we let it handle this
• We need to contact the authors of the datasets where issues exist