Optimize Zipping Process on the backend

[djbrooke]

We currently restrict zip file downloads to a max size (based on a setting) due to performance of zipping large files via glassfish. We should investigate:

• how to store files in such a way that on-demand zipping is not needed
• to modularize the functionality so that this does not tax the application server,
• some other option

This will allow for better system stability and to avoid the issue where the user downloads a zip file and doesn't find out that it's not a complete archive until after the fact (because it exceeds the limit).

For S3, it will be interesting to architect this with an eye towards cost, that is, if we store zipped versions of the dataset so that we can provide zips without processing time, we incur storage cost, but if we optimize the zipping so it takes place on demand we incur some computation cost on lambda/fargate whatever. Not sure what's preferred.

[poikilotherm]

It seems like a good idea to have this kind of batch processing offloaded with one of the great options around these days (JBatch, FaaS, separated webservice, ...).

While reading this I wondered if this kind of work isn't even more widespread in Dataverse. I know this is kinda off-scope for this issue, but as you already generalized from #6093: wouldn't it make sense to discuss about a more general offloading modularization first?

I'm thinking about Ingress processing, ZIP file creation and fulltext indexing here as the most ressource intense tasks in Dataverse I am aware of.

[djbrooke]

@poikilotherm, I'm not opposed to a more general architecture discussion and will defer to @scolapasta but I'm very interested in implementing this for zipping first. Note that ingest modularization is included in TRSA branch.

[mheppler]

In the issue __ @shlake pointed out that "a tabular file in Harvard Dataverse v4.5.1." does not include the "All File Formats + Information" option under the download dropdown menu. In another comment @scolapasta points out the "need to retest and confirm that it's no longer causing trouble, if/when we re-enable these bundle downloads".

I suggest that this use case and the request to return this feature to the UI also be considered in this download ZIP optimization effort.

[djbrooke]

@mheppler thanks for linking up that issue. I'd like to keep anything that may have a front end impact, or other feature requests, as a separate issue and not consider those use cases as part of this issue. This is solely to optimize how things work in the backend as it would tremendously reduce the incomplete zipping issues and increase stability. I'll update the title to clarify.

[mheppler]

After discussing with @djbrooke and @scolapasta the file level ZIP creation might not fit into the final solution for this issue, but the use case can still be considered in the discussion.

From what I can tell, to return this functionality, we only need to remove the ui:remove tags around the "All File Formats + Information" link in file-download-button-fragment.xhtml.

            <ui:remove>
                <li>
                    <p:commandLink styleClass="highlightBold" rendered="#{!(downloadPopupRequired)}"
                                   process="@this"
                                   actionListener="#{fileDownloadService.writeGuestbookAndStartFileDownload(guestbookResponse, fileMetadata, 'bundle')}">
                        #{bundle['file.downloadBtn.format.all']}
                    </p:commandLink>
                    <p:commandLink styleClass="highlightBold" rendered="#{downloadPopupRequired}"
                                   process="@this"
                                   action="#{guestbookResponseService.modifyDatafileAndFormat(guestbookResponse, fileMetadata, 'bundle' )}"
                                   update="@widgetVar(downloadPopup)"
                                   oncomplete="PF('downloadPopup').show();handleResizeDialog('downloadPopup');">
                        #{bundle['file.downloadBtn.format.all']}
                    </p:commandLink>
                </li>
                <li role="presentation" class="divider"></li>
            </ui:remove>

[scolapasta]

Discussed with team during tech hours - our approach here will be to separate it from the core Dataverse app as a service that can be called. Some details will need to be hashed out in development, but generally:
• a request for multiple files that need to be zipped comes in the the main app
• permissions are checked, list of files (or likely file locations) is sent to separate service
• service creates zip and either streams back to end user or puts in a temp location and informs user to download (we may want to use this 2nd option if we think we need to create multiple zips for one download)