Data File Ingest

[alejandratenorio]

Currently when a datafile is ingested it is not possible to publish the dataset.
Sometimes our users need to be able to publish a dataset while a datafile is ingesting but this action are blocked. It's practical to allow publish a Dataset while a Datafile is ingested?, while the file ingesting process finish the file contain a legend like "Download not available". We analyze the source code and we know how to do it.

[djbrooke]

@alejandratenorio thanks for opening this. Can you provide some more details about the use case here?

@landreev, @scolapasta, what are your thoughts on this?

Two quick things in mind for me are:

• a completed ingest changes the citation (adding a UNF), so someone could publish, cite, and then the citation would have some information added
• if the ingest fails and the depositor wants to make a correction to get ingest to work, they would need to do it in a new version instead of in the draft version.

I don't think these are major issues. Are there other implications that I'm missing here?

[alejandratenorio]

Hello @djbrooke.

Depending on the datafile, ingest takes a long time. While ingested some users need to publish the Dataset and share the URL, they must wait and they get nervous.

[djbrooke]

Thanks @alejandratenorio for the details. I guess another advantage (in addition to making people feel more comfortable :)) is that it would allow someone to only log in once for upload and publishing, without the need to wait and/or come back later to click publish once the ingest is complete. I can see the reasons for adding this but I'll check in with @scolapasta, @landreev and others to learn more.

[scolapasta]

As @djbrooke points out the biggest issue is the citation. Once published, a dataset version's citation should not change. But ingest will always change it - by either adding, modifying or removing the UNF. We don't want users to copy a citation for a dataset version that is not "complete".

From the technical side, the other main reason we don't allow edits to happen while an ingest is occurring is so that when the ingest completes, it doesn't run into any issues making changes in the db. Without our dataverse lock, an optimistic lock exception could occur.

My initial sense is that allowing publish while ingest is not complete will open up a pandora's box.

An alternative idea is that we provide a notification when the ingest is complete. That way the user doesn't just have to wait around, but they are notified directly when they can go back and publish.

[pdurbin]

My initial sense is that allowing publish while ingest is not complete will open up a pandora's box.

I agree 100% with this.

[djbrooke]

Thanks @scolapasta and @pdurbin

@alejandratenorio - I'm going to close this out as we don't want to implement this in the core application at this time. If you already have code that implements this, you're welcome to share it here for others who may be interested in maintaining a fork with this modification.

[alejandratenorio]

As @djbrooke points out the biggest issue is the citation. Once published, a dataset version's citation should not change. But ingest will always change it - by either adding, modifying or removing the UNF. We don't want users to copy a citation for a dataset version that is not "complete".

From the technical side, the other main reason we don't allow edits to happen while an ingest is occurring is so that when the ingest completes, it doesn't run into any issues making changes in the db. Without our dataverse lock, an optimistic lock exception could occur.

My initial sense is that allowing publish while ingest is not complete will open up a pandora's box.

An alternative idea is that we provide a notification when the ingest is complete. That way the user doesn't just have to wait around, but they are notified directly when they can go back and publish.

@scolapasta, I understand, it's complex. As a user, a notification would be very helpful. Could we send an email?

[djbrooke]

I'll create a new issue for tracking the discussion about this: #6214

[scolapasta]

While this has moved elsewhere, let me just answer the last question here:

Yes, the notification would exist both as a dataverse notification and an e-mail.