diff --git a/doc/sphinx-guides/source/developers/big-data-support.rst b/doc/sphinx-guides/source/developers/big-data-support.rst index 50bb2224610..0782fd239a1 100644 --- a/doc/sphinx-guides/source/developers/big-data-support.rst +++ b/doc/sphinx-guides/source/developers/big-data-support.rst @@ -120,6 +120,34 @@ To configure the options mentioned above, an administrator must set two JVM opti ``./asadmin create-jvm-options "-Ddataverse.files..public=true"`` ``./asadmin create-jvm-options "-Ddataverse.files..ingestsizelimit="`` +.. _globus-support: + +Globus File Transfer +-------------------- + +Note: Globus file transfer is still experimental but feedback is welcome! See :ref:`support`. + +Users can transfer files via `Globus `_ into and out of datasets when their Dataverse installation is configured to use a Globus accessible S3 store and a community-developed `dataverse-globus `_ "transfer" app has been properly installed and configured. + +Due to differences in the access control models of a Dataverse installation and Globus, enabling the Globus capability on a store will disable the ability to restrict and embargo files in that store. + +As Globus aficionados know, Globus endpoints can be in a variety of places, from data centers to personal computers. This means that from within the Dataverse software, a Globus transfer can feel like an upload or a download (with Globus Personal Connect running on your laptop, for example) or it can feel like a true transfer from one server to another (from a cluster in a data center into a Dataverse dataset or vice versa). + +Globus transfer uses a very efficient transfer mechanism and has additional features that make it suitable for large files and large numbers of files: + +* robust file transfer capable of restarting after network or endpoint failures +* third-party transfer, which enables a user accessing a Dataverse installation in their desktop browser to initiate transfer of their files from a remote endpoint (i.e. on a local high-performance computing cluster), directly to an S3 store managed by the Dataverse installation + +Globus transfer requires use of the Globus S3 connector which requires a paid Globus subscription at the host institution. Users will need a Globus account which could be obtained via their institution or directly from Globus (at no cost). + +The setup required to enable Globus is described in the `Community Dataverse-Globus Setup and Configuration document `_ and the references therein. + +As described in that document, Globus transfers can be initiated by choosing the Globus option in the dataset upload panel. (Globus, which does asynchronous transfers, is not available during dataset creation.) Analogously, "Globus Transfer" is one of the download options in the "Access Dataset" menu and optionally the file landing page download menu (if/when supported in the dataverse-globus app). + +An overview of the control and data transfer interactions between components was presented at the 2022 Dataverse Community Meeting and can be viewed in the `Integrations and Tools Session Video `_ around the 1 hr 28 min mark. + +See also :ref:`Globus settings <:GlobusBasicToken>`. + Data Capture Module (DCM) ------------------------- diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index ab0bad70206..17d88c8ea31 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -676,7 +676,7 @@ In addition to having the type "remote" and requiring a label, Trusted Remote St These and other available options are described in the table below. Trusted remote stores can range from being a static trusted website to a sophisticated service managing access requests and logging activity -and/or managing access to a secure enclave. For specific remote stores, consult their documentation when configuring the remote store in your Dataverse installation. +and/or managing access to a secure enclave. See :doc:`/developers/big-data-support` for additional information on how to use a trusted remote store. For specific remote stores, consult their documentation when configuring the remote store in your Dataverse installation. Note that in the current implementation, activites where Dataverse needs access to data bytes, e.g. to create thumbnails or validate hash values at publication will fail if a remote store does not allow Dataverse access. Implementers of such trusted remote stores should consider using Dataverse's settings to disable ingest, validation of files at publication, etc. as needed. @@ -2982,3 +2982,35 @@ The URL of an LDN Inbox to which the LDN Announce workflow step will send messag ++++++++++++++++++++++++++ The list of parent dataset field names for which the LDN Announce workflow step should send messages. See :doc:`/developers/workflows` for details. + +.. _:GlobusBasicToken: + +:GlobusBasicToken ++++++++++++++++++ + +GlobusBasicToken encodes credentials for Globus integration. See :ref:`globus-support` for details. + +:GlobusEndpoint ++++++++++++++++ + +GlobusEndpoint is Globus endpoint id used with Globus integration. See :ref:`globus-support` for details. + +:GlobusStores ++++++++++++++ + +A comma-separated list of the S3 stores that are configured to support Globus integration. See :ref:`globus-support` for details. + +:GlobusAppURL ++++++++++++++ + +The URL where the `dataverse-globus `_ "transfer" app has been deployed to support Globus integration. See :ref:`globus-support` for details. + +:GlobusPollingInterval +++++++++++++++++++++++ + +The interval in seconds between Dataverse calls to Globus to check on upload progress. Defaults to 50 seconds. See :ref:`globus-support` for details. + +:GlobusSingleFileTransfer ++++++++++++++++++++++++++ + +A true/false option to add a Globus transfer option to the file download menu which is not yet fully supported in the dataverse-globus app. See :ref:`globus-support` for details. diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java b/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java index d0ba86ab68e..7b857545c20 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetLock.java @@ -77,7 +77,10 @@ public enum Reason { /** DCM (rsync) upload in progress */ DcmUpload, - + + /** Globus upload in progress */ + GlobusUpload, + /** Tasks handled by FinalizeDatasetPublicationCommand: Registering PIDs for DS and DFs and/or file validation */ finalizePublication, diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java index d80aa4ab141..0a8db69bf5b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java @@ -113,6 +113,7 @@ import edu.harvard.iq.dataverse.engine.command.impl.SubmitDatasetForReviewCommand; import edu.harvard.iq.dataverse.externaltools.ExternalTool; import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; import edu.harvard.iq.dataverse.export.SchemaDotOrgExporter; import edu.harvard.iq.dataverse.externaltools.ExternalToolHandler; import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean; @@ -251,6 +252,8 @@ public enum DisplayMode { LicenseServiceBean licenseServiceBean; @Inject DataFileCategoryServiceBean dataFileCategoryService; + @Inject + GlobusServiceBean globusService; private Dataset dataset = new Dataset(); @@ -334,7 +337,7 @@ public void setSelectedHostDataverse(Dataverse selectedHostDataverse) { private Boolean hasRsyncScript = false; private Boolean hasTabular = false; - + /** * If the dataset version has at least one tabular file. The "hasTabular" @@ -1191,7 +1194,7 @@ public String getComputeUrl(FileMetadata metadata) { } catch (IOException e) { logger.info("DatasetPage: Failed to get storageIO"); } - if (settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)) { + if (isHasPublicStore()) { return settingsWrapper.getValueForKey(SettingsServiceBean.Key.ComputeBaseUrl) + "?" + this.getPersistentId() + "=" + swiftObject.getSwiftFileName(); } @@ -1828,15 +1831,21 @@ public void updateOwnerDataverse() { // initiate from scratch: (isolate the creation of a new dataset in its own method?) init(true); - // rebuild the bred crumbs display: + // rebuild the bread crumbs display: dataverseHeaderFragment.initBreadcrumbs(dataset); } } public boolean rsyncUploadSupported() { - return settingsWrapper.isRsyncUpload() && DatasetUtil.isAppropriateStorageDriver(dataset); + return settingsWrapper.isRsyncUpload() && DatasetUtil.isRsyncAppropriateStorageDriver(dataset); + } + + public boolean globusUploadSupported() { + return settingsWrapper.isGlobusUpload() && settingsWrapper.isGlobusEnabledStorageDriver(dataset.getEffectiveStorageDriverId()); } + + private String init(boolean initFull) { @@ -2006,10 +2015,10 @@ private String init(boolean initFull) { } } catch (RuntimeException ex) { logger.warning("Problem getting rsync script(RuntimeException): " + ex.getLocalizedMessage()); - FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", ex.getLocalizedMessage())); + FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", ex.getLocalizedMessage())); } catch (CommandException cex) { logger.warning("Problem getting rsync script (Command Exception): " + cex.getLocalizedMessage()); - FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", cex.getLocalizedMessage())); + FacesContext.getCurrentInstance().addMessage(null, new FacesMessage(FacesMessage.SEVERITY_ERROR, "Problem getting rsync script:", cex.getLocalizedMessage())); } } @@ -2065,7 +2074,7 @@ private String init(boolean initFull) { updateDatasetFieldInputLevels(); } - if (settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)){ + if (isHasPublicStore()){ JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("dataset.message.label.fileAccess"), BundleUtil.getStringFromBundle("dataset.message.publicInstall")); } @@ -2178,6 +2187,10 @@ private void displayLockInfo(Dataset dataset) { BundleUtil.getStringFromBundle("file.rsyncUpload.inProgressMessage.details")); lockedDueToDcmUpload = true; } + if (dataset.isLockedFor(DatasetLock.Reason.GlobusUpload)) { + JH.addMessage(FacesMessage.SEVERITY_WARN, BundleUtil.getStringFromBundle("file.globusUpload.inProgressMessage.summary"), + BundleUtil.getStringFromBundle("file.globusUpload.inProgressMessage.details")); + } //This is a hack to remove dataset locks for File PID registration if //the dataset is released //in testing we had cases where datasets with 1000 files were remaining locked after being published successfully @@ -2899,7 +2912,7 @@ public String editFileMetadata(){ public String deleteDatasetVersion() { DeleteDatasetVersionCommand cmd; - + Map deleteStorageLocations = datafileService.getPhysicalFilesToDelete(dataset.getLatestVersion()); boolean deleteCommandSuccess = false; try { @@ -2911,7 +2924,7 @@ public String deleteDatasetVersion() { JH.addMessage(FacesMessage.SEVERITY_FATAL, BundleUtil.getStringFromBundle("dataset.message.deleteFailure")); logger.severe(ex.getMessage()); } - + if (deleteCommandSuccess && !deleteStorageLocations.isEmpty()) { datafileService.finalizeFileDeletes(deleteStorageLocations); } @@ -5026,7 +5039,7 @@ public boolean isFileAccessRequestMultiButtonRequired(){ } for (FileMetadata fmd : workingVersion.getFileMetadatas()){ //Change here so that if all restricted files have pending requests there's no Request Button - if ((!this.fileDownloadHelper.canDownloadFile(fmd) && (fmd.getDataFile().getFileAccessRequesters() == null + if ((!this.fileDownloadHelper.canDownloadFile(fmd) && (fmd.getDataFile().getFileAccessRequesters() == null || ( fmd.getDataFile().getFileAccessRequesters() != null && !fmd.getDataFile().getFileAccessRequesters().contains((AuthenticatedUser)session.getUser()))))){ return true; @@ -5754,7 +5767,7 @@ public boolean isFileDeleted (DataFile dataFile) { return dataFile.getDeleted(); } - + public String getEffectiveMetadataLanguage() { return getEffectiveMetadataLanguage(false); } @@ -5765,16 +5778,16 @@ public String getEffectiveMetadataLanguage(boolean ofParent) { } return mdLang; } - + public String getLocaleDisplayName(String code) { String displayName = settingsWrapper.getBaseMetadataLanguageMap(false).get(code); if(displayName==null && !code.equals(DvObjectContainer.UNDEFINED_METADATA_LANGUAGE_CODE)) { //Default (for cases such as :when a Dataset has a metadatalanguage code but :MetadataLanguages is no longer defined). - displayName = new Locale(code).getDisplayName(); + displayName = new Locale(code).getDisplayName(); } - return displayName; + return displayName; } - + public Set> getMetadataLanguages() { return settingsWrapper.getBaseMetadataLanguageMap(false).entrySet(); } @@ -5786,7 +5799,7 @@ public List getVocabScripts() { public String getFieldLanguage(String languages) { return fieldService.getFieldLanguage(languages,session.getLocaleCode()); } - + public void setExternalStatus(String status) { try { dataset = commandEngine.submit(new SetCurationStatusCommand(dvRequestService.getDataverseRequest(), dataset, status)); @@ -6017,7 +6030,7 @@ public void validateTerms(FacesContext context, UIComponent component, Object va } } } - + public boolean downloadingRestrictedFiles() { if (fileMetadataForAction != null) { return fileMetadataForAction.isRestricted(); @@ -6029,4 +6042,24 @@ public boolean downloadingRestrictedFiles() { } return false; } + + + //Determines whether this Dataset uses a public store and therefore doesn't support embargoed or restricted files + public boolean isHasPublicStore() { + return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, StorageIO.isPublicStore(dataset.getEffectiveStorageDriverId())); + } + + public void startGlobusTransfer() { + ApiToken apiToken = null; + User user = session.getUser(); + if (user instanceof AuthenticatedUser) { + apiToken = authService.findApiTokenByUser((AuthenticatedUser) user); + } else if (user instanceof PrivateUrlUser) { + PrivateUrlUser privateUrlUser = (PrivateUrlUser) user; + PrivateUrl privUrl = privateUrlService.getPrivateUrlFromDatasetId(privateUrlUser.getDatasetId()); + apiToken = new ApiToken(); + apiToken.setTokenString(privUrl.getToken()); + } + PrimeFaces.current().executeScript(globusService.getGlobusDownloadScript(dataset, apiToken)); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java index ce8420b91e5..e1ca5c19a90 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetServiceBean.java @@ -16,23 +16,17 @@ import edu.harvard.iq.dataverse.engine.command.impl.FinalizeDatasetPublicationCommand; import edu.harvard.iq.dataverse.engine.command.impl.GetDatasetStorageSizeCommand; import edu.harvard.iq.dataverse.export.ExportService; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; import edu.harvard.iq.dataverse.harvest.server.OAIRecordServiceBean; import edu.harvard.iq.dataverse.search.IndexServiceBean; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.SystemConfig; import edu.harvard.iq.dataverse.workflows.WorkflowComment; -import java.io.File; -import java.io.IOException; -import java.io.InputStream; + +import java.io.*; import java.text.SimpleDateFormat; -import java.util.ArrayList; -import java.util.Date; -import java.util.HashMap; -import java.util.HashSet; -import java.util.List; -import java.util.Map; -import java.util.Set; +import java.util.*; import java.util.logging.FileHandler; import java.util.logging.Level; import java.util.logging.Logger; @@ -96,6 +90,12 @@ public class DatasetServiceBean implements java.io.Serializable { @EJB SystemConfig systemConfig; + @EJB + GlobusServiceBean globusServiceBean; + + @EJB + UserNotificationServiceBean userNotificationService; + private static final SimpleDateFormat logFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH-mm-ss"); @PersistenceContext(unitName = "VDCNet-ejbPU") @@ -1130,4 +1130,5 @@ public void deleteHarvestedDataset(Dataset dataset, DataverseRequest request, Lo hdLogger.warning("Failed to destroy the dataset"); } } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java b/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java index f53e2377a69..6cf294ffd6d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java @@ -650,8 +650,8 @@ public String init() { setUpRsync(); } - if (settingsService.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)){ - JH.addMessage(FacesMessage.SEVERITY_WARN, getBundleString("dataset.message.publicInstall")); + if (isHasPublicStore()){ + JH.addMessage(FacesMessage.SEVERITY_WARN, getBundleString("dataset.message.label.fileAccess"), getBundleString("dataset.message.publicInstall")); } return null; @@ -3051,13 +3051,21 @@ public boolean rsyncUploadSupported() { // ToDo - rsync was written before multiple store support and currently is hardcoded to use the DataAccess.S3 store. // When those restrictions are lifted/rsync can be configured per store, the test in the // Dataset Util method should be updated - if (settingsWrapper.isRsyncUpload() && !DatasetUtil.isAppropriateStorageDriver(dataset)) { + if (settingsWrapper.isRsyncUpload() && !DatasetUtil.isRsyncAppropriateStorageDriver(dataset)) { //dataset.file.upload.setUp.rsync.failed.detail FacesMessage message = new FacesMessage(FacesMessage.SEVERITY_ERROR, BundleUtil.getStringFromBundle("dataset.file.upload.setUp.rsync.failed"), BundleUtil.getStringFromBundle("dataset.file.upload.setUp.rsync.failed.detail")); FacesContext.getCurrentInstance().addMessage(null, message); } - return settingsWrapper.isRsyncUpload() && DatasetUtil.isAppropriateStorageDriver(dataset); + return settingsWrapper.isRsyncUpload() && DatasetUtil.isRsyncAppropriateStorageDriver(dataset); + } + + // Globus must be one of the upload methods listed in the :UploadMethods setting + // and the dataset's store must be in the list allowed by the GlobusStores + // setting + public boolean globusUploadSupported() { + return settingsWrapper.isGlobusUpload() + && settingsWrapper.isGlobusEnabledStorageDriver(dataset.getEffectiveStorageDriverId()); } private void populateFileMetadatas() { @@ -3093,4 +3101,9 @@ public boolean isFileAccessRequest() { public void setFileAccessRequest(boolean fileAccessRequest) { this.fileAccessRequest = fileAccessRequest; } + + //Determines whether this Dataset uses a public store and therefore doesn't support embargoed or restricted files + public boolean isHasPublicStore() { + return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, StorageIO.isPublicStore(dataset.getEffectiveStorageDriverId())); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/FilePage.java b/src/main/java/edu/harvard/iq/dataverse/FilePage.java index 3fa6d4fdfff..7f2c6dfca5c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FilePage.java +++ b/src/main/java/edu/harvard/iq/dataverse/FilePage.java @@ -13,6 +13,7 @@ import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser; import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.dataaccess.StorageIO; import edu.harvard.iq.dataverse.engine.command.Command; import edu.harvard.iq.dataverse.engine.command.exception.CommandException; @@ -843,7 +844,7 @@ public String getComputeUrl() throws IOException { if (swiftObject != null) { swiftObject.open(); //generate a temp url for a file - if (settingsService.isTrueForKey(SettingsServiceBean.Key.PublicInstall, false)) { + if (isHasPublicStore()) { return settingsService.getValueForKey(SettingsServiceBean.Key.ComputeBaseUrl) + "?" + this.getFile().getOwner().getGlobalIdString() + "=" + swiftObject.getSwiftFileName(); } return settingsService.getValueForKey(SettingsServiceBean.Key.ComputeBaseUrl) + "?" + this.getFile().getOwner().getGlobalIdString() + "=" + swiftObject.getSwiftFileName() + "&temp_url_sig=" + swiftObject.getTempUrlSignature() + "&temp_url_expires=" + swiftObject.getTempUrlExpiry(); @@ -935,8 +936,8 @@ public String getPublicDownloadUrl() { try { SwiftAccessIO swiftIO = (SwiftAccessIO) storageIO; swiftIO.open(); - //if its a public install, lets just give users the permanent URL! - if (systemConfig.isPublicInstall()){ + //if its a public store, lets just give users the permanent URL! + if (isHasPublicStore()){ fileDownloadUrl = swiftIO.getRemoteUrl(); } else { //TODO: if a user has access to this file, they should be given the swift url @@ -1165,5 +1166,10 @@ public String getEmbargoPhrase() { public String getIngestMessage() { return BundleUtil.getStringFromBundle("file.ingestFailed.message", Arrays.asList(settingsWrapper.getGuidesBaseUrl(), settingsWrapper.getGuidesVersion())); } + + //Determines whether this File uses a public store and therefore doesn't support embargoed or restricted files + public boolean isHasPublicStore() { + return settingsWrapper.isTrueForKey(SettingsServiceBean.Key.PublicInstall, StorageIO.isPublicStore(DataAccess.getStorageDriverFromIdentifier(file.getStorageIdentifier()))); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java index fa5216140c2..2bfd342d899 100644 --- a/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/MailServiceBean.java @@ -570,6 +570,49 @@ public String getMessageTextBasedOnNotification(UserNotification userNotificatio logger.fine("fileImportMsg: " + fileImportMsg); return messageText += fileImportMsg; + case GLOBUSUPLOADCOMPLETED: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String uploadCompletedMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.upload.completed", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return uploadCompletedMessage; + + case GLOBUSDOWNLOADCOMPLETED: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String downloadCompletedMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.download.completed", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return downloadCompletedMessage; + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String uploadCompletedWithErrorsMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.upload.completedWithErrors", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return uploadCompletedWithErrorsMessage; + + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + dataset = (Dataset) targetObject; + messageText = BundleUtil.getStringFromBundle("notification.email.greeting.html"); + String downloadCompletedWithErrorsMessage = messageText + BundleUtil.getStringFromBundle("notification.mail.globus.download.completedWithErrors", Arrays.asList( + systemConfig.getDataverseSiteUrl(), + dataset.getGlobalIdString(), + dataset.getDisplayName(), + comment + )) ; + return downloadCompletedWithErrorsMessage; + case CHECKSUMIMPORT: version = (DatasetVersion) targetObject; String checksumImportMsg = BundleUtil.getStringFromBundle("notification.import.checksum", Arrays.asList( @@ -671,6 +714,11 @@ public Object getObjectOfNotification (UserNotification userNotification){ return datasetService.find(userNotification.getObjectId()); case FILESYSTEMIMPORT: return versionService.find(userNotification.getObjectId()); + case GLOBUSUPLOADCOMPLETED: + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + case GLOBUSDOWNLOADCOMPLETED: + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + return datasetService.find(userNotification.getObjectId()); case CHECKSUMIMPORT: return versionService.find(userNotification.getObjectId()); case APIGENERATED: diff --git a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java index aaf38af1b36..8f7f53de1a2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/PermissionServiceBean.java @@ -733,6 +733,9 @@ else if (dataset.isLockedFor(DatasetLock.Reason.Workflow)) { else if (dataset.isLockedFor(DatasetLock.Reason.DcmUpload)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.editNotAllowed"), command); } + else if (dataset.isLockedFor(DatasetLock.Reason.GlobusUpload)) { + throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.editNotAllowed"), command); + } else if (dataset.isLockedFor(DatasetLock.Reason.EditInProgress)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.editNotAllowed"), command); } @@ -768,6 +771,9 @@ else if (dataset.isLockedFor(DatasetLock.Reason.Workflow)) { else if (dataset.isLockedFor(DatasetLock.Reason.DcmUpload)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.publishNotAllowed"), command); } + else if (dataset.isLockedFor(DatasetLock.Reason.GlobusUpload)) { + throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.publishNotAllowed"), command); + } else if (dataset.isLockedFor(DatasetLock.Reason.EditInProgress)) { throw new IllegalCommandException(BundleUtil.getStringFromBundle("dataset.message.locked.publishNotAllowed"), command); } diff --git a/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java b/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java index 8370030e44b..aa40423000d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java +++ b/src/main/java/edu/harvard/iq/dataverse/SettingsWrapper.java @@ -93,7 +93,15 @@ public class SettingsWrapper implements java.io.Serializable { private Boolean rsyncUpload = null; - private Boolean rsyncDownload = null; + private Boolean rsyncDownload = null; + + private Boolean globusUpload = null; + private Boolean globusDownload = null; + private Boolean globusFileDownload = null; + + private String globusAppUrl = null; + + private List globusStoreList = null; private Boolean httpUpload = null; @@ -293,6 +301,42 @@ public boolean isRsyncDownload() { } return rsyncDownload; } + + public boolean isGlobusUpload() { + if (globusUpload == null) { + globusUpload = systemConfig.isGlobusUpload(); + } + return globusUpload; + } + + public boolean isGlobusDownload() { + if (globusDownload == null) { + globusDownload = systemConfig.isGlobusDownload(); + } + return globusDownload; + } + + public boolean isGlobusFileDownload() { + if (globusFileDownload == null) { + globusFileDownload = systemConfig.isGlobusFileDownload(); + } + return globusFileDownload; + } + + public boolean isGlobusEnabledStorageDriver(String driverId) { + if (globusStoreList == null) { + globusStoreList = systemConfig.getGlobusStoresList(); + } + return globusStoreList.contains(driverId); + } + + public String getGlobusAppUrl() { + if (globusAppUrl == null) { + globusAppUrl = settingsService.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost"); + } + return globusAppUrl; + + } public boolean isRsyncOnly() { if (rsyncOnly == null) { diff --git a/src/main/java/edu/harvard/iq/dataverse/UserNotification.java b/src/main/java/edu/harvard/iq/dataverse/UserNotification.java index 2a3c63c1c31..b68a1b9d13e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/UserNotification.java +++ b/src/main/java/edu/harvard/iq/dataverse/UserNotification.java @@ -37,7 +37,9 @@ public enum Type { ASSIGNROLE, REVOKEROLE, CREATEDV, CREATEDS, CREATEACC, SUBMITTEDDS, RETURNEDDS, PUBLISHEDDS, REQUESTFILEACCESS, GRANTFILEACCESS, REJECTFILEACCESS, FILESYSTEMIMPORT, CHECKSUMIMPORT, CHECKSUMFAIL, CONFIRMEMAIL, APIGENERATED, INGESTCOMPLETED, INGESTCOMPLETEDWITHERRORS, - PUBLISHFAILED_PIDREG, WORKFLOW_SUCCESS, WORKFLOW_FAILURE, STATUSUPDATED, DATASETCREATED, DATASETMENTIONED; + PUBLISHFAILED_PIDREG, WORKFLOW_SUCCESS, WORKFLOW_FAILURE, STATUSUPDATED, DATASETCREATED, DATASETMENTIONED, + GLOBUSUPLOADCOMPLETED, GLOBUSUPLOADCOMPLETEDWITHERRORS, + GLOBUSDOWNLOADCOMPLETED, GLOBUSDOWNLOADCOMPLETEDWITHERRORS; public String getDescription() { return BundleUtil.getStringFromBundle("notification.typeDescription." + this.name()); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Access.java b/src/main/java/edu/harvard/iq/dataverse/api/Access.java index cfb30cc0753..abeedf23b59 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Access.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Access.java @@ -332,6 +332,11 @@ public Response datafile(@PathParam("fileId") String fileId, @QueryParam("gbrecs dInfo.addServiceAvailable(new OptionalAccessService("preprocessed", "application/json", "format=prep", "Preprocessed data in JSON")); dInfo.addServiceAvailable(new OptionalAccessService("subset", "text/tab-separated-values", "variables=<LIST>", "Column-wise Subsetting")); } + + if(systemConfig.isGlobusFileDownload() && systemConfig.getGlobusStoresList().contains(DataAccess.getStorageDriverFromIdentifier(df.getStorageIdentifier()))) { + dInfo.addServiceAvailable(new OptionalAccessService("GlobusTransfer", df.getContentType(), "format=GlobusTransfer", "Download via Globus")); + } + DownloadInstance downloadInstance = new DownloadInstance(dInfo); downloadInstance.setRequestUriInfo(uriInfo); downloadInstance.setRequestHttpHeaders(headers); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index 4ab3d1105b8..ddc80aa00b6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -7,6 +7,7 @@ import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.authorization.Permission; import edu.harvard.iq.dataverse.authorization.RoleAssignee; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.batch.jobs.importer.ImportMode; @@ -59,6 +60,7 @@ import edu.harvard.iq.dataverse.ingest.IngestServiceBean; import edu.harvard.iq.dataverse.privateurl.PrivateUrl; +import edu.harvard.iq.dataverse.S3PackageImporter; import edu.harvard.iq.dataverse.api.dto.RoleAssignmentDTO; import edu.harvard.iq.dataverse.batch.util.LoggingUtil; import edu.harvard.iq.dataverse.dataaccess.DataAccess; @@ -98,6 +100,8 @@ import edu.harvard.iq.dataverse.workflow.WorkflowServiceBean; import edu.harvard.iq.dataverse.workflow.WorkflowContext.TriggerType; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; + import java.io.IOException; import java.io.InputStream; import java.io.StringReader; @@ -107,9 +111,10 @@ import java.text.SimpleDateFormat; import java.time.LocalDate; import java.time.LocalDateTime; +import java.util.*; +import java.util.concurrent.*; import java.time.ZoneId; import java.time.format.DateTimeFormatter; -import java.util.*; import java.util.Map.Entry; import java.util.logging.Level; import java.util.logging.Logger; @@ -117,7 +122,6 @@ import javax.ejb.EJB; import javax.ejb.EJBException; -import javax.faces.context.FacesContext; import javax.inject.Inject; import javax.json.*; import javax.json.stream.JsonParsingException; @@ -135,10 +139,7 @@ import javax.ws.rs.PathParam; import javax.ws.rs.Produces; import javax.ws.rs.QueryParam; -import javax.ws.rs.core.Context; -import javax.ws.rs.core.HttpHeaders; -import javax.ws.rs.core.MediaType; -import javax.ws.rs.core.Response; +import javax.ws.rs.core.*; import javax.ws.rs.core.Response.Status; import static javax.ws.rs.core.Response.Status.BAD_REQUEST; import javax.ws.rs.core.UriInfo; @@ -163,6 +164,9 @@ public class Datasets extends AbstractApiBean { @EJB DataverseServiceBean dataverseService; + @EJB + GlobusServiceBean globusService; + @EJB UserNotificationServiceBean userNotificationService; @@ -430,7 +434,7 @@ public Response setCitationDate( @PathParam("id") String id, String dsfTypeName) execCommand(new SetDatasetCitationDateCommand(req, findDatasetOrDie(id), dsfType)); return ok("Citation Date for dataset " + id + " set to: " + (dsfType != null ? dsfType.getDisplayName() : "default")); }); - } + } @DELETE @Path("{id}/citationdate") @@ -439,7 +443,7 @@ public Response useDefaultCitationDate( @PathParam("id") String id) { execCommand(new SetDatasetCitationDateCommand(req, findDatasetOrDie(id), null)); return ok("Citation Date for dataset " + id + " set to default"); }); - } + } @GET @Path("{id}/versions") @@ -455,9 +459,9 @@ public Response listVersions( @PathParam("id") String id ) { @Path("{id}/versions/{versionId}") public Response getVersion( @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { return response( req -> { - DatasetVersion dsv = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); + DatasetVersion dsv = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); return (dsv == null || dsv.getId() == null) ? notFound("Dataset version not found") - : ok(json(dsv)); + : ok(json(dsv)); }); } @@ -474,9 +478,9 @@ public Response getVersionFiles( @PathParam("id") String datasetId, @PathParam(" public Response getFileAccessFolderView(@PathParam("id") String datasetId, @QueryParam("version") String versionId, @QueryParam("folder") String folderName, @QueryParam("original") Boolean originals, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) { folderName = folderName == null ? "" : folderName; - versionId = versionId == null ? ":latest-published" : versionId; + versionId = versionId == null ? ":latest-published" : versionId; - DatasetVersion version; + DatasetVersion version; try { DataverseRequest req = createDataverseRequest(findUserOrDie()); version = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); @@ -588,7 +592,7 @@ public Response updateDatasetPIDMetadataAll() { } catch (WrappedResponse ex) { Logger.getLogger(Datasets.class.getName()).log(Level.SEVERE, null, ex); } - }); + }); return ok(BundleUtil.getStringFromBundle("datasets.api.updatePIDMetadata.success.for.update.all")); }); } @@ -776,7 +780,7 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav boolean found = false; for (DatasetField dsf : dsv.getDatasetFields()) { if (dsf.getDatasetFieldType().equals(updateField.getDatasetFieldType())) { - if (dsf.getDatasetFieldType().isAllowMultiples()) { + if (dsf.getDatasetFieldType().isAllowMultiples()) { if (updateField.getDatasetFieldType().isControlledVocabulary()) { if (dsf.getDatasetFieldType().isAllowMultiples()) { for (ControlledVocabularyValue cvv : updateField.getControlledVocabularyValues()) { @@ -841,7 +845,7 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav datasetFieldCompoundValueItemsToRemove.forEach((remove) -> { dsf.getDatasetFieldCompoundValues().remove(remove); }); - if (!found) { + if (!found) { logger.log(Level.SEVERE, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + deleteVal + " not found."); return error(Response.Status.BAD_REQUEST, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + deleteVal + " not found."); } @@ -861,12 +865,11 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav logger.log(Level.SEVERE, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + displayValue + " not found." ); return error(Response.Status.BAD_REQUEST, "Delete metadata failed: " + updateField.getDatasetFieldType().getDisplayName() + ": " + displayValue + " not found." ); } - } + } - boolean updateDraft = ds.getLatestVersion().isDraft(); - DatasetVersion managedVersion = updateDraft + DatasetVersion managedVersion = updateDraft ? execCommand(new UpdateDatasetVersionCommand(ds, req)).getEditVersion() : execCommand(new CreateDatasetVersionCommand(req, ds, dsv)); return ok(json(managedVersion)); @@ -885,13 +888,13 @@ private Response processDatasetFieldDataDelete(String jsonBody, String id, Datav private String getCompoundDisplayValue (DatasetFieldCompoundValue dscv){ String returnString = ""; - for (DatasetField dsf : dscv.getChildDatasetFields()) { - for (String value : dsf.getValues()) { - if (!(value == null)) { - returnString += (returnString.isEmpty() ? "" : "; ") + value.trim(); - } + for (DatasetField dsf : dscv.getChildDatasetFields()) { + for (String value : dsf.getValues()) { + if (!(value == null)) { + returnString += (returnString.isEmpty() ? "" : "; ") + value.trim(); } } + } return returnString; } @@ -920,13 +923,13 @@ private Response processDatasetUpdate(String jsonBody, String id, DataverseReque DatasetVersion dsv = ds.getEditVersion(); dsv.getTermsOfUseAndAccess().setDatasetVersion(dsv); List fields = new LinkedList<>(); - DatasetField singleField = null; + DatasetField singleField = null; JsonArray fieldsJson = json.getJsonArray("fields"); - if( fieldsJson == null ){ - singleField = jsonParser().parseField(json, Boolean.FALSE); + if (fieldsJson == null) { + singleField = jsonParser().parseField(json, Boolean.FALSE); fields.add(singleField); - } else{ + } else { fields = jsonParser().parseMultipleFields(json); } @@ -1087,15 +1090,15 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S case "major": isMinor = false; break; - case "updatecurrent": - if(user.isSuperuser()) { - updateCurrent=true; - } else { - return error(Response.Status.FORBIDDEN, "Only superusers can update the current version"); - } - break; + case "updatecurrent": + if (user.isSuperuser()) { + updateCurrent = true; + } else { + return error(Response.Status.FORBIDDEN, "Only superusers can update the current version"); + } + break; default: - return error(Response.Status.BAD_REQUEST, "Illegal 'type' parameter value '" + type + "'. It needs to be either 'major', 'minor', or 'updatecurrent'."); + return error(Response.Status.BAD_REQUEST, "Illegal 'type' parameter value '" + type + "'. It needs to be either 'major', 'minor', or 'updatecurrent'."); } Dataset ds = findDatasetOrDie(id); @@ -1115,7 +1118,7 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S * set and if so, if it after the modification time. If the modification time is * set and the index time is null or is before the mod time, the 409/conflict * error is returned. - * + * */ if ((ds.getModificationTime()!=null && (ds.getIndexTime() == null || (ds.getIndexTime().compareTo(ds.getModificationTime()) <= 0))) || (ds.getPermissionModificationTime()!=null && (ds.getPermissionIndexTime() == null || (ds.getPermissionIndexTime().compareTo(ds.getPermissionModificationTime()) <= 0)))) { @@ -1179,10 +1182,10 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S .build(); } } else { - PublishDatasetResult res = execCommand(new PublishDatasetCommand(ds, + PublishDatasetResult res = execCommand(new PublishDatasetCommand(ds, createDataverseRequest(user), - isMinor)); - return res.isWorkflow() ? accepted(json(res.getDataset())) : ok(json(res.getDataset())); + isMinor)); + return res.isWorkflow() ? accepted(json(res.getDataset())) : ok(json(res.getDataset())); } } catch (WrappedResponse ex) { return ex.getResponse(); @@ -1283,7 +1286,7 @@ public Response publishMigratedDataset(String jsonldBody, @PathParam("id") Strin @Path("{id}/move/{targetDataverseAlias}") public Response moveDataset(@PathParam("id") String id, @PathParam("targetDataverseAlias") String targetDataverseAlias, @QueryParam("forceMove") Boolean force) { try { - User u = findUserOrDie(); + User u = findUserOrDie(); Dataset ds = findDatasetOrDie(id); Dataverse target = dataverseService.findByAlias(targetDataverseAlias); if (target == null) { @@ -1565,21 +1568,21 @@ public Response removeFileEmbargo(@PathParam("id") String id, String jsonBody){ @PUT - @Path("{linkedDatasetId}/link/{linkingDataverseAlias}") - public Response linkDataset(@PathParam("linkedDatasetId") String linkedDatasetId, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { - try{ - User u = findUserOrDie(); + @Path("{linkedDatasetId}/link/{linkingDataverseAlias}") + public Response linkDataset(@PathParam("linkedDatasetId") String linkedDatasetId, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { + try { + User u = findUserOrDie(); Dataset linked = findDatasetOrDie(linkedDatasetId); Dataverse linking = findDataverseOrDie(linkingDataverseAlias); if (linked == null){ return error(Response.Status.BAD_REQUEST, "Linked Dataset not found."); - } - if (linking == null){ + } + if (linking == null) { return error(Response.Status.BAD_REQUEST, "Linking Dataverse not found."); - } + } execCommand(new LinkDatasetCommand( createDataverseRequest(u), linking, linked - )); + )); return ok("Dataset " + linked.getId() + " linked successfully to " + linking.getAlias()); } catch (WrappedResponse ex) { return ex.getResponse(); @@ -1634,8 +1637,8 @@ public Response getLinks(@PathParam("id") String idSupplied ) { /** * Add a given assignment to a given user or group - * @param ra role assignment DTO - * @param id dataset id + * @param ra role assignment DTO + * @param id dataset id * @param apiKey */ @POST @@ -1647,7 +1650,7 @@ public Response createAssignment(RoleAssignmentDTO ra, @PathParam("identifier") RoleAssignee assignee = findAssignee(ra.getAssignee()); if (assignee == null) { return error(Response.Status.BAD_REQUEST, BundleUtil.getStringFromBundle("datasets.api.grant.role.assignee.not.found.error")); - } + } DataverseRole theRole; Dataverse dv = dataset.getOwner(); @@ -1699,10 +1702,10 @@ public Response deleteAssignment(@PathParam("id") long assignmentId, @PathParam( @GET @Path("{identifier}/assignments") public Response getAssignments(@PathParam("identifier") String id) { - return response( req -> - ok( execCommand( - new ListRoleAssignments(req, findDatasetOrDie(id))) - .stream().map(ra->json(ra)).collect(toJsonArray())) ); + return response(req -> + ok(execCommand( + new ListRoleAssignments(req, findDatasetOrDie(id))) + .stream().map(ra -> json(ra)).collect(toJsonArray()))); } @GET @@ -1710,8 +1713,8 @@ public Response getAssignments(@PathParam("identifier") String id) { public Response getPrivateUrlData(@PathParam("id") String idSupplied) { return response( req -> { PrivateUrl privateUrl = execCommand(new GetPrivateUrlCommand(req, findDatasetOrDie(idSupplied))); - return (privateUrl != null) ? ok(json(privateUrl)) - : error(Response.Status.NOT_FOUND, "Private URL not found."); + return (privateUrl != null) ? ok(json(privateUrl)) + : error(Response.Status.NOT_FOUND, "Private URL not found."); }); } @@ -1721,7 +1724,7 @@ public Response createPrivateUrl(@PathParam("id") String idSupplied,@DefaultValu if(anonymizedAccess && settingsSvc.getValueForKey(SettingsServiceBean.Key.AnonymizedFieldTypeNames)==null) { throw new NotAcceptableException("Anonymized Access not enabled"); } - return response( req -> + return response(req -> ok(json(execCommand( new CreatePrivateUrlCommand(req, findDatasetOrDie(idSupplied), anonymizedAccess))))); } @@ -1855,13 +1858,13 @@ public Response getRsync(@PathParam("identifier") String id) { } /** - * This api endpoint triggers the creation of a "package" file in a dataset - * after that package has been moved onto the same filesystem via the Data Capture Module. + * This api endpoint triggers the creation of a "package" file in a dataset + * after that package has been moved onto the same filesystem via the Data Capture Module. * The package is really just a way that Dataverse interprets a folder created by DCM, seeing it as just one file. * The "package" can be downloaded over RSAL. - * + * * This endpoint currently supports both posix file storage and AWS s3 storage in Dataverse, and depending on which one is active acts accordingly. - * + * * The initial design of the DCM/Dataverse interaction was not to use packages, but to allow import of all individual files natively into Dataverse. * But due to the possibly immense number of files (millions) the package approach was taken. * This is relevant because the posix ("file") code contains many remnants of that development work. @@ -1885,7 +1888,7 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String try { Dataset dataset = findDatasetOrDie(id); if ("validation passed".equals(statusMessageFromDcm)) { - logger.log(Level.INFO, "Checksum Validation passed for DCM."); + logger.log(Level.INFO, "Checksum Validation passed for DCM."); String storageDriver = dataset.getDataverseContext().getEffectiveStorageDriverId(); String uploadFolder = jsonFromDcm.getString("uploadFolder"); @@ -1947,10 +1950,10 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String JsonObjectBuilder job = Json.createObjectBuilder(); return ok(job); - } catch (IOException e) { + } catch (IOException e) { String message = e.getMessage(); return error(Response.Status.INTERNAL_SERVER_ERROR, "Uploaded files have passed checksum validation but something went wrong while attempting to move the files into Dataverse. Message was '" + message + "'."); - } + } } else { return error(Response.Status.INTERNAL_SERVER_ERROR, "Invalid storage driver in Dataverse, not compatible with dcm"); } @@ -2003,7 +2006,7 @@ public Response returnToAuthor(@PathParam("id") String idSupplied, String jsonBo JsonObject json = Json.createReader(rdr).readObject(); try { Dataset dataset = findDatasetOrDie(idSupplied); - String reasonForReturn = null; + String reasonForReturn = null; reasonForReturn = json.getString("reasonForReturn"); // TODO: Once we add a box for the curator to type into, pass the reason for return to the ReturnDatasetToAuthorCommand and delete this check and call to setReturnReason on the API side. if (reasonForReturn == null || reasonForReturn.isEmpty()) { @@ -2056,7 +2059,7 @@ public Response setCurationStatus(@PathParam("id") String idSupplied, @QueryPara return Response.fromResponse(wr.getResponse()).status(Response.Status.BAD_REQUEST).build(); } } - + @DELETE @Path("{id}/curationStatus") public Response deleteCurationStatus(@PathParam("id") String idSupplied) { @@ -2076,228 +2079,228 @@ public Response deleteCurationStatus(@PathParam("id") String idSupplied) { return Response.fromResponse(wr.getResponse()).status(Response.Status.BAD_REQUEST).build(); } } - -@GET -@Path("{id}/uploadsid") -@Deprecated -public Response getUploadUrl(@PathParam("id") String idSupplied) { - try { - Dataset dataset = findDatasetOrDie(idSupplied); - - boolean canUpdateDataset = false; - try { - canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset).canIssue(UpdateDatasetVersionCommand.class); - } catch (WrappedResponse ex) { - logger.info("Exception thrown while trying to figure out permissions while getting upload URL for dataset id " + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - if (!canUpdateDataset) { - return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); - } - S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); - if(s3io == null) { - return error(Response.Status.NOT_FOUND,"Direct upload not supported for files in this dataset: " + dataset.getId()); - } - String url = null; - String storageIdentifier = null; - try { - url = s3io.generateTemporaryS3UploadUrl(); - storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); - } catch (IOException io) { - logger.warning(io.getMessage()); - throw new WrappedResponse(io, error( Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); - } - - JsonObjectBuilder response = Json.createObjectBuilder() - .add("url", url) - .add("storageIdentifier", storageIdentifier ); - return ok(response); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} -@GET -@Path("{id}/uploadurls") -public Response getMPUploadUrls(@PathParam("id") String idSupplied, @QueryParam("size") long fileSize) { - try { - Dataset dataset = findDatasetOrDie(idSupplied); - - boolean canUpdateDataset = false; - try { - canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset) - .canIssue(UpdateDatasetVersionCommand.class); - } catch (WrappedResponse ex) { - logger.info( - "Exception thrown while trying to figure out permissions while getting upload URLs for dataset id " - + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - if (!canUpdateDataset) { - return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); - } - S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); - if (s3io == null) { - return error(Response.Status.NOT_FOUND, - "Direct upload not supported for files in this dataset: " + dataset.getId()); - } - JsonObjectBuilder response = null; - String storageIdentifier = null; - try { - storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); - response = s3io.generateTemporaryS3UploadUrls(dataset.getGlobalId().asString(), storageIdentifier, fileSize); - - } catch (IOException io) { - logger.warning(io.getMessage()); - throw new WrappedResponse(io, - error(Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); - } - - response.add("storageIdentifier", storageIdentifier); - return ok(response); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} + @GET + @Path("{id}/uploadsid") + @Deprecated + public Response getUploadUrl(@PathParam("id") String idSupplied) { + try { + Dataset dataset = findDatasetOrDie(idSupplied); -@DELETE -@Path("mpupload") -public Response abortMPUpload(@QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { - try { - Dataset dataset = datasetSvc.findByGlobalId(idSupplied); - //Allow the API to be used within a session (e.g. for direct upload in the UI) - User user =session.getUser(); - if (!user.isAuthenticated()) { - try { - user = findAuthenticatedUserOrDie(); - } catch (WrappedResponse ex) { - logger.info( - "Exception thrown while trying to figure out permissions while getting aborting upload for dataset id " - + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - } - boolean allowed = false; - if (dataset != null) { - allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) - .canIssue(UpdateDatasetVersionCommand.class); - } else { - /* - * The only legitimate case where a global id won't correspond to a dataset is - * for uploads during creation. Given that this call will still fail unless all - * three parameters correspond to an active multipart upload, it should be safe - * to allow the attempt for an authenticated user. If there are concerns about - * permissions, one could check with the current design that the user is allowed - * to create datasets in some dataverse that is configured to use the storage - * provider specified in the storageidentifier, but testing for the ability to - * create a dataset in a specific dataverse would requiring changing the design - * somehow (e.g. adding the ownerId to this call). - */ - allowed = true; - } - if (!allowed) { - return error(Response.Status.FORBIDDEN, - "You are not permitted to abort file uploads with the supplied parameters."); - } - try { - S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); - } catch (IOException io) { - logger.warning("Multipart upload abort failed for uploadId: " + uploadId + " storageidentifier=" - + storageidentifier + " dataset Id: " + dataset.getId()); - logger.warning(io.getMessage()); - throw new WrappedResponse(io, - error(Response.Status.INTERNAL_SERVER_ERROR, "Could not abort multipart upload")); - } - return Response.noContent().build(); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} + boolean canUpdateDataset = false; + try { + canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset).canIssue(UpdateDatasetVersionCommand.class); + } catch (WrappedResponse ex) { + logger.info("Exception thrown while trying to figure out permissions while getting upload URL for dataset id " + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + if (!canUpdateDataset) { + return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); + } + S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); + if (s3io == null) { + return error(Response.Status.NOT_FOUND, "Direct upload not supported for files in this dataset: " + dataset.getId()); + } + String url = null; + String storageIdentifier = null; + try { + url = s3io.generateTemporaryS3UploadUrl(); + storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); + } catch (IOException io) { + logger.warning(io.getMessage()); + throw new WrappedResponse(io, error(Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); + } -@PUT -@Path("mpupload") -public Response completeMPUpload(String partETagBody, @QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { - try { - Dataset dataset = datasetSvc.findByGlobalId(idSupplied); - //Allow the API to be used within a session (e.g. for direct upload in the UI) - User user =session.getUser(); - if (!user.isAuthenticated()) { - try { - user=findAuthenticatedUserOrDie(); - } catch (WrappedResponse ex) { - logger.info( - "Exception thrown while trying to figure out permissions to complete mpupload for dataset id " - + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } - } - boolean allowed = false; - if (dataset != null) { - allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) - .canIssue(UpdateDatasetVersionCommand.class); - } else { - /* - * The only legitimate case where a global id won't correspond to a dataset is - * for uploads during creation. Given that this call will still fail unless all - * three parameters correspond to an active multipart upload, it should be safe - * to allow the attempt for an authenticated user. If there are concerns about - * permissions, one could check with the current design that the user is allowed - * to create datasets in some dataverse that is configured to use the storage - * provider specified in the storageidentifier, but testing for the ability to - * create a dataset in a specific dataverse would requiring changing the design - * somehow (e.g. adding the ownerId to this call). - */ - allowed = true; - } - if (!allowed) { - return error(Response.Status.FORBIDDEN, - "You are not permitted to complete file uploads with the supplied parameters."); - } - List eTagList = new ArrayList(); - logger.info("Etags: " + partETagBody); - try { - JsonReader jsonReader = Json.createReader(new StringReader(partETagBody)); - JsonObject object = jsonReader.readObject(); - jsonReader.close(); - for(String partNo : object.keySet()) { - eTagList.add(new PartETag(Integer.parseInt(partNo), object.getString(partNo))); - } - for(PartETag et: eTagList) { - logger.fine("Part: " + et.getPartNumber() + " : " + et.getETag()); - } - } catch (JsonException je) { - logger.info("Unable to parse eTags from: " + partETagBody); - throw new WrappedResponse(je, error( Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); - } - try { - S3AccessIO.completeMultipartUpload(idSupplied, storageidentifier, uploadId, eTagList); - } catch (IOException io) { - logger.warning("Multipart upload completion failed for uploadId: " + uploadId +" storageidentifier=" + storageidentifier + " globalId: " + idSupplied); - logger.warning(io.getMessage()); - try { - S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); - } catch (IOException e) { - logger.severe("Also unable to abort the upload (and release the space on S3 for uploadId: " + uploadId +" storageidentifier=" + storageidentifier + " globalId: " + idSupplied); - logger.severe(io.getMessage()); - } - - throw new WrappedResponse(io, error( Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); - } - return ok("Multipart Upload completed"); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } -} + JsonObjectBuilder response = Json.createObjectBuilder() + .add("url", url) + .add("storageIdentifier", storageIdentifier); + return ok(response); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + @GET + @Path("{id}/uploadurls") + public Response getMPUploadUrls(@PathParam("id") String idSupplied, @QueryParam("size") long fileSize) { + try { + Dataset dataset = findDatasetOrDie(idSupplied); + + boolean canUpdateDataset = false; + try { + canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset) + .canIssue(UpdateDatasetVersionCommand.class); + } catch (WrappedResponse ex) { + logger.info( + "Exception thrown while trying to figure out permissions while getting upload URLs for dataset id " + + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + if (!canUpdateDataset) { + return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); + } + S3AccessIO s3io = FileUtil.getS3AccessForDirectUpload(dataset); + if (s3io == null) { + return error(Response.Status.NOT_FOUND, + "Direct upload not supported for files in this dataset: " + dataset.getId()); + } + JsonObjectBuilder response = null; + String storageIdentifier = null; + try { + storageIdentifier = FileUtil.getStorageIdentifierFromLocation(s3io.getStorageLocation()); + response = s3io.generateTemporaryS3UploadUrls(dataset.getGlobalId().asString(), storageIdentifier, fileSize); + + } catch (IOException io) { + logger.warning(io.getMessage()); + throw new WrappedResponse(io, + error(Response.Status.INTERNAL_SERVER_ERROR, "Could not create process direct upload request")); + } + + response.add("storageIdentifier", storageIdentifier); + return ok(response); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + @DELETE + @Path("mpupload") + public Response abortMPUpload(@QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { + try { + Dataset dataset = datasetSvc.findByGlobalId(idSupplied); + //Allow the API to be used within a session (e.g. for direct upload in the UI) + User user = session.getUser(); + if (!user.isAuthenticated()) { + try { + user = findAuthenticatedUserOrDie(); + } catch (WrappedResponse ex) { + logger.info( + "Exception thrown while trying to figure out permissions while getting aborting upload for dataset id " + + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + } + boolean allowed = false; + if (dataset != null) { + allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) + .canIssue(UpdateDatasetVersionCommand.class); + } else { + /* + * The only legitimate case where a global id won't correspond to a dataset is + * for uploads during creation. Given that this call will still fail unless all + * three parameters correspond to an active multipart upload, it should be safe + * to allow the attempt for an authenticated user. If there are concerns about + * permissions, one could check with the current design that the user is allowed + * to create datasets in some dataverse that is configured to use the storage + * provider specified in the storageidentifier, but testing for the ability to + * create a dataset in a specific dataverse would requiring changing the design + * somehow (e.g. adding the ownerId to this call). + */ + allowed = true; + } + if (!allowed) { + return error(Response.Status.FORBIDDEN, + "You are not permitted to abort file uploads with the supplied parameters."); + } + try { + S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); + } catch (IOException io) { + logger.warning("Multipart upload abort failed for uploadId: " + uploadId + " storageidentifier=" + + storageidentifier + " dataset Id: " + dataset.getId()); + logger.warning(io.getMessage()); + throw new WrappedResponse(io, + error(Response.Status.INTERNAL_SERVER_ERROR, "Could not abort multipart upload")); + } + return Response.noContent().build(); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + + @PUT + @Path("mpupload") + public Response completeMPUpload(String partETagBody, @QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { + try { + Dataset dataset = datasetSvc.findByGlobalId(idSupplied); + //Allow the API to be used within a session (e.g. for direct upload in the UI) + User user = session.getUser(); + if (!user.isAuthenticated()) { + try { + user = findAuthenticatedUserOrDie(); + } catch (WrappedResponse ex) { + logger.info( + "Exception thrown while trying to figure out permissions to complete mpupload for dataset id " + + dataset.getId() + ": " + ex.getLocalizedMessage()); + throw ex; + } + } + boolean allowed = false; + if (dataset != null) { + allowed = permissionSvc.requestOn(createDataverseRequest(user), dataset) + .canIssue(UpdateDatasetVersionCommand.class); + } else { + /* + * The only legitimate case where a global id won't correspond to a dataset is + * for uploads during creation. Given that this call will still fail unless all + * three parameters correspond to an active multipart upload, it should be safe + * to allow the attempt for an authenticated user. If there are concerns about + * permissions, one could check with the current design that the user is allowed + * to create datasets in some dataverse that is configured to use the storage + * provider specified in the storageidentifier, but testing for the ability to + * create a dataset in a specific dataverse would requiring changing the design + * somehow (e.g. adding the ownerId to this call). + */ + allowed = true; + } + if (!allowed) { + return error(Response.Status.FORBIDDEN, + "You are not permitted to complete file uploads with the supplied parameters."); + } + List eTagList = new ArrayList(); + logger.info("Etags: " + partETagBody); + try { + JsonReader jsonReader = Json.createReader(new StringReader(partETagBody)); + JsonObject object = jsonReader.readObject(); + jsonReader.close(); + for (String partNo : object.keySet()) { + eTagList.add(new PartETag(Integer.parseInt(partNo), object.getString(partNo))); + } + for (PartETag et : eTagList) { + logger.info("Part: " + et.getPartNumber() + " : " + et.getETag()); + } + } catch (JsonException je) { + logger.info("Unable to parse eTags from: " + partETagBody); + throw new WrappedResponse(je, error(Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); + } + try { + S3AccessIO.completeMultipartUpload(idSupplied, storageidentifier, uploadId, eTagList); + } catch (IOException io) { + logger.warning("Multipart upload completion failed for uploadId: " + uploadId + " storageidentifier=" + storageidentifier + " globalId: " + idSupplied); + logger.warning(io.getMessage()); + try { + S3AccessIO.abortMultipartUpload(idSupplied, storageidentifier, uploadId); + } catch (IOException e) { + logger.severe("Also unable to abort the upload (and release the space on S3 for uploadId: " + uploadId + " storageidentifier=" + storageidentifier + " globalId: " + idSupplied); + logger.severe(io.getMessage()); + } + + throw new WrappedResponse(io, error(Response.Status.INTERNAL_SERVER_ERROR, "Could not complete multipart upload")); + } + return ok("Multipart Upload completed"); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } /** * Add a File to an existing Dataset - * + * * @param idSupplied * @param jsonData * @param fileInputStream * @param contentDispositionHeader * @param formDataBodyPart - * @return + * @return */ @POST @Path("{id}/add") @@ -2322,7 +2325,7 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } catch (WrappedResponse ex) { return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") - ); + ); } @@ -2335,7 +2338,7 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, try { dataset = findDatasetOrDie(idSupplied); } catch (WrappedResponse wr) { - return wr.getResponse(); + return wr.getResponse(); } //------------------------------------ @@ -2354,12 +2357,12 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, // (2a) Load up optional params via JSON //--------------------------------------- OptionalFileParams optionalFileParams = null; - msgt("(api) jsonData: " + jsonData); + msgt("(api) jsonData: " + jsonData); try { optionalFileParams = new OptionalFileParams(jsonData); } catch (DataFileTagException ex) { - return error( Response.Status.BAD_REQUEST, ex.getMessage()); + return error(Response.Status.BAD_REQUEST, ex.getMessage()); } catch (ClassCastException | com.google.gson.JsonParseException ex) { return error(Response.Status.BAD_REQUEST, BundleUtil.getStringFromBundle("file.addreplace.error.parsing")); @@ -2411,7 +2414,7 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } } - + //------------------- // (3) Create the AddReplaceFileHelper object //------------------- @@ -2419,11 +2422,11 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, DataverseRequest dvRequest2 = createDataverseRequest(authUser); AddReplaceFileHelper addFileHelper = new AddReplaceFileHelper(dvRequest2, - ingestService, - datasetService, - fileService, - permissionSvc, - commandEngine, + ingestService, + datasetService, + fileService, + permissionSvc, + commandEngine, systemConfig, licenseSvc); @@ -2432,16 +2435,16 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, // (4) Run "runAddFileByDatasetId" //------------------- addFileHelper.runAddFileByDataset(dataset, - newFilename, - newFileContentType, - newStorageIdentifier, - fileInputStream, - optionalFileParams); + newFilename, + newFileContentType, + newStorageIdentifier, + fileInputStream, + optionalFileParams); - if (addFileHelper.hasError()){ + if (addFileHelper.hasError()) { return error(addFileHelper.getHttpErrorCode(), addFileHelper.getErrorMessagesAsString("\n")); - }else{ + } else { String successMsg = BundleUtil.getStringFromBundle("file.addreplace.success.add"); try { //msgt("as String: " + addFileHelper.getSuccessResult()); @@ -2467,71 +2470,77 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } } - + } // end: addFileToDataset - - private void msg(String m){ + private void msg(String m) { //System.out.println(m); logger.fine(m); } - private void dashes(){ + + private void dashes() { msg("----------------"); } - private void msgt(String m){ - dashes(); msg(m); dashes(); + + private void msgt(String m) { + dashes(); + msg(m); + dashes(); } - - - public static T handleVersion( String versionId, DsVersionHandler hdl ) - throws WrappedResponse { + + + public static T handleVersion(String versionId, DsVersionHandler hdl) + throws WrappedResponse { switch (versionId) { - case ":latest": return hdl.handleLatest(); - case ":draft": return hdl.handleDraft(); - case ":latest-published": return hdl.handleLatestPublished(); + case ":latest": + return hdl.handleLatest(); + case ":draft": + return hdl.handleDraft(); + case ":latest-published": + return hdl.handleLatestPublished(); default: try { String[] versions = versionId.split("\\."); switch (versions.length) { case 1: - return hdl.handleSpecific(Long.parseLong(versions[0]), (long)0.0); + return hdl.handleSpecific(Long.parseLong(versions[0]), (long) 0.0); case 2: - return hdl.handleSpecific( Long.parseLong(versions[0]), Long.parseLong(versions[1]) ); + return hdl.handleSpecific(Long.parseLong(versions[0]), Long.parseLong(versions[1])); default: - throw new WrappedResponse(error( Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'")); + throw new WrappedResponse(error(Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'")); } - } catch ( NumberFormatException nfe ) { - throw new WrappedResponse( error( Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'") ); + } catch (NumberFormatException nfe) { + throw new WrappedResponse(error(Response.Status.BAD_REQUEST, "Illegal version identifier '" + versionId + "'")); } } } - - private DatasetVersion getDatasetVersionOrDie( final DataverseRequest req, String versionNumber, final Dataset ds, UriInfo uriInfo, HttpHeaders headers) throws WrappedResponse { - DatasetVersion dsv = execCommand( handleVersion(versionNumber, new DsVersionHandler>(){ - @Override - public Command handleLatest() { - return new GetLatestAccessibleDatasetVersionCommand(req, ds); - } + private DatasetVersion getDatasetVersionOrDie(final DataverseRequest req, String versionNumber, final Dataset ds, UriInfo uriInfo, HttpHeaders headers) throws WrappedResponse { + DatasetVersion dsv = execCommand(handleVersion(versionNumber, new DsVersionHandler>() { - @Override - public Command handleDraft() { - return new GetDraftDatasetVersionCommand(req, ds); - } - - @Override - public Command handleSpecific(long major, long minor) { - return new GetSpecificPublishedDatasetVersionCommand(req, ds, major, minor); - } + @Override + public Command handleLatest() { + return new GetLatestAccessibleDatasetVersionCommand(req, ds); + } - @Override - public Command handleLatestPublished() { - return new GetLatestPublishedDatasetVersionCommand(req, ds); - } - })); - if ( dsv == null || dsv.getId() == null ) { - throw new WrappedResponse( notFound("Dataset version " + versionNumber + " of dataset " + ds.getId() + " not found") ); + @Override + public Command handleDraft() { + return new GetDraftDatasetVersionCommand(req, ds); + } + + @Override + public Command handleSpecific(long major, long minor) { + return new GetSpecificPublishedDatasetVersionCommand(req, ds, major, minor); + } + + @Override + public Command handleLatestPublished() { + return new GetLatestPublishedDatasetVersionCommand(req, ds); + } + })); + if (dsv == null || dsv.getId() == null) { + throw new WrappedResponse(notFound("Dataset version " + versionNumber + " of dataset " + ds.getId() + " not found")); } if (dsv.isReleased()&& uriInfo!=null) { MakeDataCountLoggingServiceBean.MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, ds); @@ -2547,14 +2556,14 @@ public Response getLocksForDataset(@PathParam("identifier") String id, @QueryPar Dataset dataset = null; try { dataset = findDatasetOrDie(id); - Set locks; + Set locks; if (lockType == null) { locks = dataset.getLocks(); } else { // request for a specific type lock: DatasetLock lock = dataset.getLockFor(lockType); - locks = new HashSet<>(); + locks = new HashSet<>(); if (lock != null) { locks.add(lock); } @@ -2564,9 +2573,9 @@ public Response getLocksForDataset(@PathParam("identifier") String id, @QueryPar } catch (WrappedResponse wr) { return wr.getResponse(); - } - } - + } + } + @DELETE @Path("{identifier}/locks") public Response deleteLocks(@PathParam("identifier") String id, @QueryParam("type") DatasetLock.Reason lockType) { @@ -2639,7 +2648,7 @@ public Response lockDataset(@PathParam("identifier") String id, @PathParam("type AuthenticatedUser user = findAuthenticatedUserOrDie(); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "This API end point can be used by superusers only."); - } + } Dataset dataset = findDatasetOrDie(id); DatasetLock lock = dataset.getLockFor(lockType); if (lock != null) { @@ -2732,7 +2741,7 @@ public Response getMakeDataCountCitations(@PathParam("id") String idSupplied) { Dataset dataset = findDatasetOrDie(idSupplied); JsonArrayBuilder datasetsCitations = Json.createArrayBuilder(); List externalCitations = datasetExternalCitationsService.getDatasetExternalCitationsByDataset(dataset); - for (DatasetExternalCitations citation : externalCitations ){ + for (DatasetExternalCitations citation : externalCitations) { JsonObjectBuilder candidateObj = Json.createObjectBuilder(); /** * In the future we can imagine storing and presenting more @@ -2743,9 +2752,9 @@ public Response getMakeDataCountCitations(@PathParam("id") String idSupplied) { */ candidateObj.add("citationUrl", citation.getCitedByUrl()); datasetsCitations.add(candidateObj); - } - return ok(datasetsCitations); - + } + return ok(datasetsCitations); + } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -2761,20 +2770,20 @@ public Response getMakeDataCountMetricCurrentMonth(@PathParam("id") String idSup @GET @Path("{identifier}/storagesize") - public Response getStorageSize(@PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + public Response getStorageSize(@PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached, + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + return response(req -> ok(MessageFormat.format(BundleUtil.getStringFromBundle("datasets.api.datasize.storage"), - execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), includeCached,GetDatasetStorageSizeCommand.Mode.STORAGE, null))))); + execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), includeCached, GetDatasetStorageSizeCommand.Mode.STORAGE, null))))); } @GET @Path("{identifier}/versions/{versionId}/downloadsize") - public Response getDownloadSize(@PathParam("identifier") String dvIdtf, @PathParam("versionId") String version, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + public Response getDownloadSize(@PathParam("identifier") String dvIdtf, @PathParam("versionId") String version, + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + return response(req -> ok(MessageFormat.format(BundleUtil.getStringFromBundle("datasets.api.datasize.download"), - execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), false, GetDatasetStorageSizeCommand.Mode.DOWNLOAD, getDatasetVersionOrDie(req, version , findDatasetOrDie(dvIdtf), uriInfo, headers)))))); + execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), false, GetDatasetStorageSizeCommand.Mode.DOWNLOAD, getDatasetVersionOrDie(req, version, findDatasetOrDie(dvIdtf), uriInfo, headers)))))); } @GET @@ -2898,7 +2907,7 @@ public Response getFileStore(@PathParam("identifier") String dvIdtf, } catch (WrappedResponse ex) { return error(Response.Status.NOT_FOUND, "No such dataset"); } - + return response(req -> ok(dataset.getEffectiveStorageDriverId())); } @@ -2918,9 +2927,9 @@ public Response setFileStore(@PathParam("identifier") String dvIdtf, if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -2936,14 +2945,14 @@ public Response setFileStore(@PathParam("identifier") String dvIdtf, } } return error(Response.Status.BAD_REQUEST, - "No Storage Driver found for : " + storageDriverLabel); + "No Storage Driver found for : " + storageDriverLabel); } @DELETE @Path("{identifier}/storageDriver") public Response resetFileStore(@PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + // Superuser-only: AuthenticatedUser user; try { @@ -2954,9 +2963,9 @@ public Response resetFileStore(@PathParam("identifier") String dvIdtf, if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -2971,8 +2980,8 @@ public Response resetFileStore(@PathParam("identifier") String dvIdtf, @GET @Path("{identifier}/curationLabelSet") public Response getCurationLabelSet(@PathParam("identifier") String dvIdtf, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + try { AuthenticatedUser user = findAuthenticatedUserOrDie(); if (!user.isSuperuser()) { @@ -2981,24 +2990,24 @@ public Response getCurationLabelSet(@PathParam("identifier") String dvIdtf, } catch (WrappedResponse wr) { return wr.getResponse(); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { return ex.getResponse(); } - + return response(req -> ok(dataset.getEffectiveCurationLabelSetName())); } - + @PUT @Path("{identifier}/curationLabelSet") public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, @QueryParam("name") String curationLabelSet, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + // Superuser-only: AuthenticatedUser user; try { @@ -3009,9 +3018,9 @@ public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -3033,12 +3042,12 @@ public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, return error(Response.Status.BAD_REQUEST, "No Such Curation Label Set"); } - + @DELETE @Path("{identifier}/curationLabelSet") public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { - + // Superuser-only: AuthenticatedUser user; try { @@ -3049,15 +3058,15 @@ public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { return ex.getResponse(); } - + dataset.setCurationLabelSetName(SystemConfig.DEFAULTCURATIONLABELSET); datasetService.merge(dataset); return ok("Curation Label Set reset to default: " + SystemConfig.DEFAULTCURATIONLABELSET); @@ -3066,16 +3075,16 @@ public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, @GET @Path("{identifier}/allowedCurationLabels") public Response getAllowedCurationLabels(@PathParam("identifier") String dvIdtf, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { AuthenticatedUser user = null; try { user = findAuthenticatedUserOrDie(); } catch (WrappedResponse wr) { return wr.getResponse(); } - - Dataset dataset; - + + Dataset dataset; + try { dataset = findDatasetOrDie(dvIdtf); } catch (WrappedResponse ex) { @@ -3088,7 +3097,7 @@ public Response getAllowedCurationLabels(@PathParam("identifier") String dvIdtf, return error(Response.Status.FORBIDDEN, "You are not permitted to view the allowed curation labels for this dataset."); } } - + @GET @Path("{identifier}/timestamps") @Produces(MediaType.APPLICATION_JSON) @@ -3118,6 +3127,7 @@ public Response getTimestamps(@PathParam("identifier") String id) { if (dataset.getLastExportTime() != null) { timestamps.add("lastMetadataExportTime", formatter.format(dataset.getLastExportTime().toInstant().atZone(ZoneId.systemDefault()))); + } if (dataset.getMostRecentMajorVersionReleaseDate() != null) { @@ -3129,11 +3139,11 @@ public Response getTimestamps(@PathParam("identifier") String id) { timestamps.add("hasStaleIndex", (dataset.getModificationTime() != null && (dataset.getIndexTime() == null || (dataset.getIndexTime().compareTo(dataset.getModificationTime()) <= 0))) ? true - : false); + : false); timestamps.add("hasStalePermissionIndex", (dataset.getPermissionModificationTime() != null && (dataset.getIndexTime() == null || (dataset.getIndexTime().compareTo(dataset.getModificationTime()) <= 0))) ? true - : false); + : false); } // More detail if you can see a draft if (canSeeDraft) { @@ -3162,6 +3172,129 @@ public Response getTimestamps(@PathParam("identifier") String id) { } + @POST + @Path("{id}/addglobusFiles") + @Consumes(MediaType.MULTIPART_FORM_DATA) + public Response addGlobusFilesToDataset(@PathParam("id") String datasetId, + @FormDataParam("jsonData") String jsonData, + @Context UriInfo uriInfo, + @Context HttpHeaders headers + ) throws IOException, ExecutionException, InterruptedException { + + logger.info(" ==== (api addGlobusFilesToDataset) jsonData ====== " + jsonData); + + if (!systemConfig.isHTTPUpload()) { + return error(Response.Status.SERVICE_UNAVAILABLE, BundleUtil.getStringFromBundle("file.api.httpDisabled")); + } + + // ------------------------------------- + // (1) Get the user from the API key + // ------------------------------------- + AuthenticatedUser authUser; + try { + authUser = findAuthenticatedUserOrDie(); + } catch (WrappedResponse ex) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") + ); + } + + // ------------------------------------- + // (2) Get the Dataset Id + // ------------------------------------- + Dataset dataset; + + try { + dataset = findDatasetOrDie(datasetId); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + + //------------------------------------ + // (2b) Make sure dataset does not have package file + // -------------------------------------- + + for (DatasetVersion dv : dataset.getVersions()) { + if (dv.isHasPackageFile()) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.api.alreadyHasPackageFile") + ); + } + } + + + String lockInfoMessage = "Globus Upload API started "; + DatasetLock lock = datasetService.addDatasetLock(dataset.getId(), DatasetLock.Reason.GlobusUpload, + (authUser).getId(), lockInfoMessage); + if (lock != null) { + dataset.addLock(lock); + } else { + logger.log(Level.WARNING, "Failed to lock the dataset (dataset id={0})", dataset.getId()); + } + + + ApiToken token = authSvc.findApiTokenByUser(authUser); + + if(uriInfo != null) { + logger.info(" ==== (api uriInfo.getRequestUri()) jsonData ====== " + uriInfo.getRequestUri().toString()); + } + + + String requestUrl = headers.getRequestHeader("origin").get(0); + + if(requestUrl.contains("localhost")){ + requestUrl = "http://localhost:8080"; + } + + // Async Call + globusService.globusUpload(jsonData, token, dataset, requestUrl, authUser); + + return ok("Async call to Globus Upload started "); + + } + + @POST + @Path("{id}/deleteglobusRule") + @Consumes(MediaType.MULTIPART_FORM_DATA) + public Response deleteglobusRule(@PathParam("id") String datasetId,@FormDataParam("jsonData") String jsonData + ) throws IOException, ExecutionException, InterruptedException { + + + logger.info(" ==== (api deleteglobusRule) jsonData ====== " + jsonData); + + + if (!systemConfig.isHTTPUpload()) { + return error(Response.Status.SERVICE_UNAVAILABLE, BundleUtil.getStringFromBundle("file.api.httpDisabled")); + } + + // ------------------------------------- + // (1) Get the user from the API key + // ------------------------------------- + User authUser; + try { + authUser = findUserOrDie(); + } catch (WrappedResponse ex) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") + ); + } + + // ------------------------------------- + // (2) Get the Dataset Id + // ------------------------------------- + Dataset dataset; + + try { + dataset = findDatasetOrDie(datasetId); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + + // Async Call + globusService.globusDownload(jsonData, dataset, authUser); + + return ok("Async call to Globus Download started"); + + } + + /** * Add multiple Files to an existing Dataset * @@ -3201,6 +3334,9 @@ public Response addFilesToDataset(@PathParam("id") String idSupplied, return wr.getResponse(); } + dataset.getLocks().forEach(dl -> { + logger.info(dl.toString()); + }); //------------------------------------ // (2a) Make sure dataset does not have package file @@ -3230,10 +3366,10 @@ public Response addFilesToDataset(@PathParam("id") String idSupplied, return addFileHelper.addFiles(jsonData, dataset, authUser); } - - /** + + /** * API to find curation assignments and statuses - * + * * @return * @throws WrappedResponse */ diff --git a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java index 07215cb919e..c9eb3638b90 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstance.java @@ -11,6 +11,8 @@ import edu.harvard.iq.dataverse.EjbDataverseEngine; import edu.harvard.iq.dataverse.GuestbookResponse; import java.util.List; +import java.util.logging.Logger; + import edu.harvard.iq.dataverse.dataaccess.OptionalAccessService; import javax.faces.context.FacesContext; import javax.ws.rs.core.HttpHeaders; @@ -22,6 +24,7 @@ */ public class DownloadInstance { + private static final Logger logger = Logger.getLogger(DownloadInstance.class.getCanonicalName()); /* private ByteArrayOutputStream outStream = null; @@ -122,6 +125,7 @@ public Boolean checkIfServiceSupportedAndSetConverter(String serviceArg, String for (OptionalAccessService dataService : servicesAvailable) { if (dataService != null) { + logger.fine("Checking service: " + dataService.getServiceName()); if (serviceArg.equals("variables")) { // Special case for the subsetting parameter (variables=): if ("subset".equals(dataService.getServiceName())) { @@ -149,6 +153,7 @@ public Boolean checkIfServiceSupportedAndSetConverter(String serviceArg, String return true; } String argValuePair = serviceArg + "=" + serviceArgValue; + logger.fine("Comparing: " + argValuePair + " and " + dataService.getServiceArguments()); if (argValuePair.startsWith(dataService.getServiceArguments())) { conversionParam = serviceArg; conversionParamValue = serviceArgValue; diff --git a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java index 6bde8b5b07a..01f627ea23b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/DownloadInstanceWriter.java @@ -27,9 +27,12 @@ import edu.harvard.iq.dataverse.engine.command.Command; import edu.harvard.iq.dataverse.engine.command.exception.CommandException; import edu.harvard.iq.dataverse.engine.command.impl.CreateGuestbookResponseCommand; +import edu.harvard.iq.dataverse.globus.GlobusServiceBean; import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean; import edu.harvard.iq.dataverse.makedatacount.MakeDataCountLoggingServiceBean.MakeDataCountEntry; import edu.harvard.iq.dataverse.util.FileUtil; +import edu.harvard.iq.dataverse.util.SystemConfig; + import java.io.File; import java.io.FileInputStream; import java.net.URI; @@ -59,6 +62,10 @@ public class DownloadInstanceWriter implements MessageBodyWriter clazz, Type type, Annotation[] throw new NotFoundException("Datafile " + dataFile.getId() + ": Failed to locate and/or open physical file."); } + + boolean redirectSupported = false; + String auxiliaryTag = null; + String auxiliaryType = null; + String auxiliaryFileName = null; // Before we do anything else, check if this download can be handled // by a redirect to remote storage (only supported on S3, as of 5.4): if (storageIO.downloadRedirectEnabled()) { @@ -101,10 +113,8 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] // for a saved original; but CANNOT if it is a column subsetting // request (must be streamed in real time locally); or a format // conversion that hasn't been cached and saved on S3 yet. - boolean redirectSupported = true; - String auxiliaryTag = null; - String auxiliaryType = null; - String auxiliaryFileName = null; + redirectSupported = true; + if ("imageThumb".equals(di.getConversionParam())) { @@ -112,7 +122,7 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] int requestedSize = 0; if (!"".equals(di.getConversionParamValue())) { try { - requestedSize = new Integer(di.getConversionParamValue()); + requestedSize = Integer.parseInt(di.getConversionParamValue()); } catch (java.lang.NumberFormatException ex) { // it's ok, the default size will be used. } @@ -177,39 +187,52 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] redirectSupported = false; } } - - if (redirectSupported) { - // definitely close the (potentially still open) input stream, - // since we are not going to use it. The S3 documentation in particular - // emphasizes that it is very important not to leave these - // lying around un-closed, since they are going to fill - // up the S3 connection pool! - storageIO.closeInputStream(); - // [attempt to] redirect: - String redirect_url_str; - try { - redirect_url_str = storageIO.generateTemporaryDownloadUrl(auxiliaryTag, auxiliaryType, auxiliaryFileName); - } catch (IOException ioex) { - logger.warning("Unable to generate downloadURL for " + dataFile.getId() + ": " + auxiliaryTag); - //Setting null will let us try to get the file/aux file w/o redirecting - redirect_url_str = null; + } + String redirect_url_str=null; + + if (redirectSupported) { + // definitely close the (potentially still open) input stream, + // since we are not going to use it. The S3 documentation in particular + // emphasizes that it is very important not to leave these + // lying around un-closed, since they are going to fill + // up the S3 connection pool! + storageIO.closeInputStream(); + // [attempt to] redirect: + try { + redirect_url_str = storageIO.generateTemporaryDownloadUrl(auxiliaryTag, auxiliaryType, auxiliaryFileName); + } catch (IOException ioex) { + logger.warning("Unable to generate downloadURL for " + dataFile.getId() + ": " + auxiliaryTag); + //Setting null will let us try to get the file/aux file w/o redirecting + redirect_url_str = null; + } + } + + if (systemConfig.isGlobusFileDownload() && systemConfig.getGlobusStoresList() + .contains(DataAccess.getStorageDriverFromIdentifier(dataFile.getStorageIdentifier()))) { + if (di.getConversionParam() != null) { + if (di.getConversionParam().equals("format")) { + + if ("GlobusTransfer".equals(di.getConversionParamValue())) { + redirect_url_str = globusService.getGlobusAppUrlForDataset(dataFile.getOwner(), false, dataFile); + } } + } + if (redirect_url_str!=null) { - logger.fine("Data Access API: direct S3 url: " + redirect_url_str); - + logger.fine("Data Access API: redirect url: " + redirect_url_str); URI redirect_uri; try { redirect_uri = new URI(redirect_url_str); - } catch (URISyntaxException|NullPointerException ex) { - logger.info("Data Access API: failed to create S3 redirect url (" + redirect_url_str + ")"); + } catch (URISyntaxException ex) { + logger.info("Data Access API: failed to create redirect url (" + redirect_url_str + ")"); redirect_uri = null; } if (redirect_uri != null) { // increment the download count, if necessary: if (di.getGbr() != null && !(isThumbnailDownload(di) || isPreprocessedMetadataDownload(di))) { try { - logger.fine("writing guestbook response, for an S3 download redirect."); + logger.fine("writing guestbook response, for a download redirect."); Command cmd = new CreateGuestbookResponseCommand(di.getDataverseRequestService().getDataverseRequest(), di.getGbr(), di.getGbr().getDataFile().getOwner()); di.getCommand().submit(cmd); MakeDataCountEntry entry = new MakeDataCountEntry(di.getRequestUriInfo(), di.getRequestHttpHeaders(), di.getDataverseRequestService(), di.getGbr().getDataFile()); @@ -220,7 +243,7 @@ public void writeTo(DownloadInstance di, Class clazz, Type type, Annotation[] // finally, issue the redirect: Response response = Response.seeOther(redirect_uri).build(); - logger.fine("Issuing redirect to the file location on S3."); + logger.fine("Issuing redirect to the file location."); throw new RedirectionException(response); } throw new ServiceUnavailableException(); diff --git a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java index 4df567815d5..5c0f3a49f76 100644 --- a/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/authorization/providers/builtin/DataverseUserPage.java @@ -523,6 +523,13 @@ public void displayNotification() { userNotification.setTheObject(datasetVersionService.find(userNotification.getObjectId())); break; + case GLOBUSUPLOADCOMPLETED: + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + case GLOBUSDOWNLOADCOMPLETED: + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + userNotification.setTheObject(datasetService.find(userNotification.getObjectId())); + break; + case CHECKSUMIMPORT: userNotification.setTheObject(datasetVersionService.find(userNotification.getObjectId())); break; diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java index acb441267ee..90e4a54dbe8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java @@ -35,16 +35,14 @@ import java.nio.channels.ReadableByteChannel; import java.nio.channels.WritableByteChannel; import java.nio.file.Path; +import java.util.HashMap; import java.util.Iterator; import java.util.List; +import java.util.Map; import java.util.regex.Matcher; import java.util.regex.Pattern; -//import org.apache.commons.httpclient.Header; -//import org.apache.commons.httpclient.methods.GetMethod; - - /** * * @author Leonid Andreev @@ -81,7 +79,10 @@ public StorageIO(T dvObject, DataAccessRequest req, String driverId) { protected boolean isReadAccess = false; protected boolean isWriteAccess = false; - + //A public store is one in which files may be accessible outside Dataverse and therefore accessible without regard to Dataverse's access controls related to restriction and embargoes. + //Currently, this is just used to warn users at upload time rather than disable restriction/embargo. + static protected Map driverPublicAccessMap = new HashMap(); + public boolean canRead() { return isReadAccess; } @@ -590,12 +591,21 @@ public String generateTemporaryDownloadUrl(String auxiliaryTag, String auxiliary throw new UnsupportedDataAccessOperationException("Direct download not implemented for this storage type"); } + + public static boolean isPublicStore(String driverId) { + //Read once and cache + if(!driverPublicAccessMap.containsKey(driverId)) { + driverPublicAccessMap.put(driverId, Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".public"))); + } + return driverPublicAccessMap.get(driverId); + } + public static String getDriverPrefix(String driverId) { return driverId+ DataAccess.SEPARATOR; } public static boolean isDirectUploadEnabled(String driverId) { - return Boolean.getBoolean(System.getProperty("dataverse.files." + driverId + ".download-redirect", "false")); + return Boolean.parseBoolean(System.getProperty("dataverse.files." + driverId + ".upload-redirect")); } //Check that storageIdentifier is consistent with store's config diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java index 2e5cebf47d6..b1725b040a3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java @@ -874,7 +874,7 @@ public String getSwiftContainerName() { } return null; } - + //https://gist.github.com/ishikawa/88599 public static String toHexString(byte[] bytes) { Formatter formatter = new Formatter(); diff --git a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java index 4dc8ea2de21..7683aab7dfa 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataset/DatasetUtil.java @@ -37,6 +37,7 @@ import edu.harvard.iq.dataverse.datasetutility.FileSizeChecker; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; import edu.harvard.iq.dataverse.license.License; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.StringUtil; import static edu.harvard.iq.dataverse.util.json.JsonPrinter.json; import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; @@ -457,7 +458,7 @@ public static List getDatasetSummaryFields(DatasetVersion datasetV return datasetFields; } - public static boolean isAppropriateStorageDriver(Dataset dataset){ + public static boolean isRsyncAppropriateStorageDriver(Dataset dataset){ // ToDo - rsync was written before multiple store support and currently is hardcoded to use the DataAccess.S3 store. // When those restrictions are lifted/rsync can be configured per store, this test should check that setting // instead of testing for the 's3" store, @@ -477,16 +478,16 @@ public static boolean isAppropriateStorageDriver(Dataset dataset){ public static String getDownloadSize(DatasetVersion dsv, boolean original) { return FileSizeChecker.bytesToHumanReadable(getDownloadSizeNumeric(dsv, original)); } - + public static Long getDownloadSizeNumeric(DatasetVersion dsv, boolean original) { return getDownloadSizeNumericBySelectedFiles(dsv.getFileMetadatas(), original); } - + public static Long getDownloadSizeNumericBySelectedFiles(List fileMetadatas, boolean original) { long bytes = 0l; for (FileMetadata fileMetadata : fileMetadatas) { DataFile dataFile = fileMetadata.getDataFile(); - if (original && dataFile.isTabularData()) { + if (original && dataFile.isTabularData()) { bytes += dataFile.getOriginalFileSize() == null ? 0 : dataFile.getOriginalFileSize(); } else { bytes += dataFile.getFilesize(); diff --git a/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java b/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java index e97b66d325b..a6ca27050a1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java +++ b/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java @@ -774,7 +774,7 @@ private boolean runAddReplacePhase2(boolean tabIngest){ } } } - + msgt("step_090_notifyUser"); if (!this.step_090_notifyUser()){ return false; @@ -1375,7 +1375,7 @@ private boolean step_040_auto_checkForDuplicates(){ String fileType = fileToReplace.getOriginalFileFormat() != null ? fileToReplace.getOriginalFileFormat() : fileToReplace.getContentType(); if (!finalFileList.get(0).getContentType().equalsIgnoreCase(fileType)) { String friendlyType = fileToReplace.getOriginalFormatLabel() != null ? fileToReplace.getOriginalFormatLabel() : fileToReplace.getFriendlyType(); - + List errParams = Arrays.asList(friendlyType, finalFileList.get(0).getFriendlyType()); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java index 3e17cc638a4..12bb3fb6a0a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/FinalizeDatasetPublicationCommand.java @@ -37,6 +37,9 @@ import java.util.concurrent.Future; import org.apache.solr.client.solrj.SolrServerException; +import javax.ejb.EJB; +import javax.inject.Inject; + /** * @@ -48,7 +51,9 @@ public class FinalizeDatasetPublicationCommand extends AbstractPublishDatasetCommand { private static final Logger logger = Logger.getLogger(FinalizeDatasetPublicationCommand.class.getName()); - + + + /** * mirror field from {@link PublishDatasetCommand} of same name */ diff --git a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java index 3ed11602e85..33d8c2d0d54 100644 --- a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java +++ b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolHandler.java @@ -2,10 +2,8 @@ import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.Dataset; -import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.authorization.users.ApiToken; -import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.URLTokenUtil; import java.io.StringReader; @@ -106,8 +104,6 @@ public void setApiToken(ApiToken apiToken) { public String getExploreScript() { String toolUrl = this.getToolUrlWithQueryParams(); logger.fine("Exploring with " + toolUrl); - String msg = BundleUtil.getStringFromBundle("externaltools.enable.browser.popups"); - String script = "const newWin = window.open('" + toolUrl + "', target='_blank'); if (!newWin || newWin.closed || typeof newWin.closed == \"undefined\") {alert(\"" + msg + "\");}"; - return script; + return getScriptForUrl(toolUrl); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/AccessList.java b/src/main/java/edu/harvard/iq/dataverse/globus/AccessList.java new file mode 100644 index 00000000000..9a963000541 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/AccessList.java @@ -0,0 +1,33 @@ +package edu.harvard.iq.dataverse.globus; + +import java.util.ArrayList; + +public class AccessList { + private int length; + private String endpoint; + private ArrayList DATA; + + public void setDATA(ArrayList DATA) { + this.DATA = DATA; + } + + public void setEndpoint(String endpoint) { + this.endpoint = endpoint; + } + + public void setLength(int length) { + this.length = length; + } + + public String getEndpoint() { + return endpoint; + } + + public ArrayList getDATA() { + return DATA; + } + + public int getLength() { + return length; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/AccessToken.java b/src/main/java/edu/harvard/iq/dataverse/globus/AccessToken.java new file mode 100644 index 00000000000..877fc68e4a1 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/AccessToken.java @@ -0,0 +1,88 @@ +package edu.harvard.iq.dataverse.globus; + +import java.util.ArrayList; + +public class AccessToken implements java.io.Serializable { + + private String accessToken; + private String idToken; + private Long expiresIn; + private String resourceServer; + private String tokenType; + private String state; + private String scope; + private String refreshToken; + private ArrayList otherTokens; + + public String getAccessToken() { + return accessToken; + } + + String getIdToken() { + return idToken; + } + + Long getExpiresIn() { + return expiresIn; + } + + String getResourceServer() { + return resourceServer; + } + + String getTokenType() { + return tokenType; + } + + String getState() { + return state; + } + + String getScope() { + return scope; + } + + String getRefreshToken() { + return refreshToken; + } + + ArrayList getOtherTokens() { + return otherTokens; + } + + public void setAccessToken(String accessToken) { + this.accessToken = accessToken; + } + + public void setExpiresIn(Long expiresIn) { + this.expiresIn = expiresIn; + } + + public void setIdToken(String idToken) { + this.idToken = idToken; + } + + public void setOtherTokens(ArrayList otherTokens) { + this.otherTokens = otherTokens; + } + + public void setRefreshToken(String refreshToken) { + this.refreshToken = refreshToken; + } + + public void setResourceServer(String resourceServer) { + this.resourceServer = resourceServer; + } + + public void setScope(String scope) { + this.scope = scope; + } + + public void setState(String state) { + this.state = state; + } + + public void setTokenType(String tokenType) { + this.tokenType = tokenType; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/FileDetailsHolder.java b/src/main/java/edu/harvard/iq/dataverse/globus/FileDetailsHolder.java new file mode 100644 index 00000000000..0b8373cba09 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/FileDetailsHolder.java @@ -0,0 +1,29 @@ +package edu.harvard.iq.dataverse.globus; + +public class FileDetailsHolder { + + private String hash; + private String mime; + private String storageID; + + public FileDetailsHolder(String id, String hash, String mime) { + + this.storageID = id; + this.hash = hash; + this.mime = mime; + + } + + public String getStorageID() { + return this.storageID; + } + + public String getHash() { + return hash; + } + + public String getMime() { + return mime; + } + +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java new file mode 100644 index 00000000000..9d80c5cc280 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusServiceBean.java @@ -0,0 +1,1265 @@ +package edu.harvard.iq.dataverse.globus; + +import com.google.gson.FieldNamingPolicy; +import com.google.gson.GsonBuilder; +import edu.harvard.iq.dataverse.*; + +import javax.ejb.Asynchronous; +import javax.ejb.EJB; +import javax.ejb.Stateless; +import javax.ejb.TransactionAttribute; +import javax.ejb.TransactionAttributeType; +import javax.inject.Inject; +import javax.inject.Named; +import javax.json.Json; +import javax.json.JsonArray; +import javax.json.JsonArrayBuilder; +import javax.json.JsonObject; +import javax.json.JsonPatch; +import javax.servlet.http.HttpServletRequest; + +import static edu.harvard.iq.dataverse.util.json.JsonPrinter.json; +import static edu.harvard.iq.dataverse.util.json.JsonPrinter.toJsonArray; + +import java.io.*; + +import java.net.HttpURLConnection; +import java.net.MalformedURLException; +import java.net.URL; +import java.net.URLEncoder; +import java.sql.Timestamp; +import java.text.SimpleDateFormat; +import java.util.*; +import java.util.concurrent.CompletableFuture; +import java.util.concurrent.ExecutionException; +import java.util.concurrent.Executor; +import java.util.concurrent.Executors; +import java.util.logging.FileHandler; +import java.util.logging.Level; +import java.util.logging.Logger; +import java.util.stream.Collectors; +import java.util.stream.IntStream; + +import com.google.gson.Gson; +import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.dataaccess.DataAccess; +import edu.harvard.iq.dataverse.dataaccess.StorageIO; +import edu.harvard.iq.dataverse.settings.SettingsServiceBean; +import edu.harvard.iq.dataverse.util.FileUtil; +import edu.harvard.iq.dataverse.util.SystemConfig; +import edu.harvard.iq.dataverse.util.URLTokenUtil; +import edu.harvard.iq.dataverse.util.json.JsonUtil; + +@Stateless +@Named("GlobusServiceBean") +public class GlobusServiceBean implements java.io.Serializable { + + @EJB + protected DatasetServiceBean datasetSvc; + + @EJB + protected SettingsServiceBean settingsSvc; + + @Inject + DataverseSession session; + + @EJB + protected AuthenticationServiceBean authSvc; + + @EJB + EjbDataverseEngine commandEngine; + + @EJB + UserNotificationServiceBean userNotificationService; + + private static final Logger logger = Logger.getLogger(GlobusServiceBean.class.getCanonicalName()); + private static final SimpleDateFormat logFormatter = new SimpleDateFormat("yyyy-MM-dd'T'HH-mm-ss"); + + private String code; + private String userTransferToken; + private String state; + + public String getState() { + return state; + } + + public void setState(String state) { + this.state = state; + } + + public String getCode() { + return code; + } + + public void setCode(String code) { + this.code = code; + } + + public String getUserTransferToken() { + return userTransferToken; + } + + public void setUserTransferToken(String userTransferToken) { + this.userTransferToken = userTransferToken; + } + + ArrayList checkPermisions(AccessToken clientTokenUser, String directory, String globusEndpoint, + String principalType, String principal) throws MalformedURLException { + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access_list"); + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + ArrayList ids = new ArrayList(); + if (result.status == 200) { + AccessList al = parseJson(result.jsonResponse, AccessList.class, false); + + for (int i = 0; i < al.getDATA().size(); i++) { + Permissions pr = al.getDATA().get(i); + if ((pr.getPath().equals(directory + "/") || pr.getPath().equals(directory)) + && pr.getPrincipalType().equals(principalType) + && ((principal == null) || (principal != null && pr.getPrincipal().equals(principal)))) { + ids.add(pr.getId()); + } else { + logger.info(pr.getPath() + " === " + directory + " == " + pr.getPrincipalType()); + continue; + } + } + } + + return ids; + } + + public void updatePermision(AccessToken clientTokenUser, String directory, String principalType, String perm) + throws MalformedURLException { + if (directory != null && !directory.equals("")) { + directory = directory + "/"; + } + logger.info("Start updating permissions." + " Directory is " + directory); + String globusEndpoint = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusEndpoint, ""); + ArrayList rules = checkPermisions(clientTokenUser, directory, globusEndpoint, principalType, null); + logger.info("Size of rules " + rules.size()); + int count = 0; + while (count < rules.size()) { + logger.info("Start removing rules " + rules.get(count)); + Permissions permissions = new Permissions(); + permissions.setDATA_TYPE("access"); + permissions.setPermissions(perm); + permissions.setPath(directory); + + Gson gson = new GsonBuilder().create(); + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + + rules.get(count)); + logger.info("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + + rules.get(count)); + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "PUT", gson.toJson(permissions)); + if (result.status != 200) { + logger.warning("Cannot update access rule " + rules.get(count)); + } else { + logger.info("Access rule " + rules.get(count) + " was updated"); + } + count++; + } + } + + public void deletePermision(String ruleId, Logger globusLogger) throws MalformedURLException { + + if (ruleId.length() > 0) { + AccessToken clientTokenUser = getClientToken(); + globusLogger.info("Start deleting permissions."); + String globusEndpoint = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusEndpoint, ""); + + URL url = new URL( + "https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + ruleId); + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "DELETE", null); + if (result.status != 200) { + globusLogger.warning("Cannot delete access rule " + ruleId); + } else { + globusLogger.info("Access rule " + ruleId + " was deleted successfully"); + } + } + + } + + public int givePermission(String principalType, String principal, String perm, AccessToken clientTokenUser, + String directory, String globusEndpoint) throws MalformedURLException { + + ArrayList rules = checkPermisions(clientTokenUser, directory, globusEndpoint, principalType, principal); + + Permissions permissions = new Permissions(); + permissions.setDATA_TYPE("access"); + permissions.setPrincipalType(principalType); + permissions.setPrincipal(principal); + permissions.setPath(directory + "/"); + permissions.setPermissions(perm); + + Gson gson = new GsonBuilder().create(); + MakeRequestResponse result = null; + if (rules.size() == 0) { + logger.info("Start creating the rule"); + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access"); + result = makeRequest(url, "Bearer", clientTokenUser.getOtherTokens().get(0).getAccessToken(), "POST", + gson.toJson(permissions)); + + if (result.status == 400) { + logger.severe("Path " + permissions.getPath() + " is not valid"); + } else if (result.status == 409) { + logger.warning("ACL already exists or Endpoint ACL already has the maximum number of access rules"); + } + + return result.status; + } else { + logger.info("Start Updating the rule"); + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/access/" + + rules.get(0)); + result = makeRequest(url, "Bearer", clientTokenUser.getOtherTokens().get(0).getAccessToken(), "PUT", + gson.toJson(permissions)); + + if (result.status == 400) { + logger.severe("Path " + permissions.getPath() + " is not valid"); + } else if (result.status == 409) { + logger.warning("ACL already exists or Endpoint ACL already has the maximum number of access rules"); + } + logger.info("Result status " + result.status); + } + + return result.status; + } + + public boolean getSuccessfulTransfers(AccessToken clientTokenUser, String taskId) throws MalformedURLException { + + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint_manager/task/" + taskId + + "/successful_transfers"); + + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + + if (result.status == 200) { + logger.info(" SUCCESS ====== "); + return true; + } + return false; + } + + public GlobusTask getTask(AccessToken clientTokenUser, String taskId, Logger globusLogger) throws MalformedURLException { + + URL url = new URL("https://transfer.api.globusonline.org/v0.10/endpoint_manager/task/" + taskId); + + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + + GlobusTask task = null; + + if (result.status == 200) { + task = parseJson(result.jsonResponse, GlobusTask.class, false); + } + if (result.status != 200) { + globusLogger.warning("Cannot find information for the task " + taskId + " : Reason : " + + result.jsonResponse.toString()); + } + + return task; + } + + public AccessToken getClientToken() throws MalformedURLException { + String globusBasicToken = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusBasicToken, ""); + URL url = new URL( + "https://auth.globus.org/v2/oauth2/token?scope=openid+email+profile+urn:globus:auth:scope:transfer.api.globus.org:all&grant_type=client_credentials"); + + MakeRequestResponse result = makeRequest(url, "Basic", globusBasicToken, "POST", null); + AccessToken clientTokenUser = null; + if (result.status == 200) { + clientTokenUser = parseJson(result.jsonResponse, AccessToken.class, true); + } + return clientTokenUser; + } + + public AccessToken getAccessToken(HttpServletRequest origRequest, String globusBasicToken) + throws UnsupportedEncodingException, MalformedURLException { + String serverName = origRequest.getServerName(); + if (serverName.equals("localhost")) { + logger.severe("Changing localhost to utoronto"); + serverName = "utl-192-123.library.utoronto.ca"; + } + + String redirectURL = "https://" + serverName + "/globus.xhtml"; + + redirectURL = URLEncoder.encode(redirectURL, "UTF-8"); + + URL url = new URL("https://auth.globus.org/v2/oauth2/token?code=" + code + "&redirect_uri=" + redirectURL + + "&grant_type=authorization_code"); + logger.info(url.toString()); + + MakeRequestResponse result = makeRequest(url, "Basic", globusBasicToken, "POST", null); + AccessToken accessTokenUser = null; + + if (result.status == 200) { + logger.info("Access Token: \n" + result.toString()); + accessTokenUser = parseJson(result.jsonResponse, AccessToken.class, true); + logger.info(accessTokenUser.getAccessToken()); + } + + return accessTokenUser; + + } + + public MakeRequestResponse makeRequest(URL url, String authType, String authCode, String method, + String jsonString) { + String str = null; + HttpURLConnection connection = null; + int status = 0; + try { + connection = (HttpURLConnection) url.openConnection(); + // Basic + // NThjMGYxNDQtN2QzMy00ZTYzLTk3MmUtMjljNjY5YzJjNGJiOktzSUVDMDZtTUxlRHNKTDBsTmRibXBIbjZvaWpQNGkwWVVuRmQyVDZRSnc9 + logger.info(authType + " " + authCode); + connection.setRequestProperty("Authorization", authType + " " + authCode); + // connection.setRequestProperty("Content-Type", + // "application/x-www-form-urlencoded"); + connection.setRequestMethod(method); + if (jsonString != null) { + connection.setRequestProperty("Content-Type", "application/json"); + connection.setRequestProperty("Accept", "application/json"); + logger.info(jsonString); + connection.setDoOutput(true); + OutputStreamWriter wr = new OutputStreamWriter(connection.getOutputStream()); + wr.write(jsonString); + wr.flush(); + } + + status = connection.getResponseCode(); + logger.info("Status now " + status); + InputStream result = connection.getInputStream(); + if (result != null) { + logger.info("Result is not null"); + str = readResultJson(result).toString(); + logger.info("str is "); + logger.info(result.toString()); + } else { + logger.info("Result is null"); + str = null; + } + + logger.info("status: " + status); + } catch (IOException ex) { + logger.info("IO"); + logger.severe(ex.getMessage()); + logger.info(ex.getCause().toString()); + logger.info(ex.getStackTrace().toString()); + } finally { + if (connection != null) { + connection.disconnect(); + } + } + MakeRequestResponse r = new MakeRequestResponse(str, status); + return r; + + } + + private StringBuilder readResultJson(InputStream in) { + StringBuilder sb = null; + try { + + BufferedReader br = new BufferedReader(new InputStreamReader(in)); + sb = new StringBuilder(); + String line; + while ((line = br.readLine()) != null) { + sb.append(line + "\n"); + } + br.close(); + logger.info(sb.toString()); + } catch (IOException e) { + sb = null; + logger.severe(e.getMessage()); + } + return sb; + } + + private T parseJson(String sb, Class jsonParserClass, boolean namingPolicy) { + if (sb != null) { + Gson gson = null; + if (namingPolicy) { + gson = new GsonBuilder().setFieldNamingPolicy(FieldNamingPolicy.LOWER_CASE_WITH_UNDERSCORES).create(); + + } else { + gson = new GsonBuilder().create(); + } + T jsonClass = gson.fromJson(sb, jsonParserClass); + return jsonClass; + } else { + logger.severe("Bad respond from token rquest"); + return null; + } + } + + public String getDirectory(String datasetId) { + Dataset dataset = null; + String directory = null; + try { + dataset = datasetSvc.find(Long.parseLong(datasetId)); + if (dataset == null) { + logger.severe("Dataset not found " + datasetId); + return null; + } + String storeId = dataset.getStorageIdentifier(); + storeId.substring(storeId.indexOf("//") + 1); + directory = storeId.substring(storeId.indexOf("//") + 1); + logger.info(storeId); + logger.info(directory); + logger.info("Storage identifier:" + dataset.getIdentifierForFileStorage()); + return directory; + + } catch (NumberFormatException nfe) { + logger.severe(nfe.getMessage()); + + return null; + } + + } + + class MakeRequestResponse { + public String jsonResponse; + public int status; + + MakeRequestResponse(String jsonResponse, int status) { + this.jsonResponse = jsonResponse; + this.status = status; + } + + } + + private MakeRequestResponse findDirectory(String directory, AccessToken clientTokenUser, String globusEndpoint) + throws MalformedURLException { + URL url = new URL(" https://transfer.api.globusonline.org/v0.10/endpoint/" + globusEndpoint + "/ls?path=" + + directory + "/"); + + MakeRequestResponse result = makeRequest(url, "Bearer", + clientTokenUser.getOtherTokens().get(0).getAccessToken(), "GET", null); + logger.info("find directory status:" + result.status); + + return result; + } + + public boolean giveGlobusPublicPermissions(String datasetId) + throws UnsupportedEncodingException, MalformedURLException { + + String globusEndpoint = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusEndpoint, ""); + String globusBasicToken = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusBasicToken, ""); + if (globusEndpoint.equals("") || globusBasicToken.equals("")) { + return false; + } + AccessToken clientTokenUser = getClientToken(); + if (clientTokenUser == null) { + logger.severe("Cannot get client token "); + return false; + } + + String directory = getDirectory(datasetId); + logger.info(directory); + + MakeRequestResponse status = findDirectory(directory, clientTokenUser, globusEndpoint); + + if (status.status == 200) { + + /* + * FilesList fl = parseJson(status.jsonResponse, FilesList.class, false); + * ArrayList files = fl.getDATA(); if (files != null) { for (FileG file: + * files) { if (!file.getName().contains("cached") && + * !file.getName().contains(".thumb")) { int perStatus = + * givePermission("all_authenticated_users", "", "r", clientTokenUser, directory + * + "/" + file.getName(), globusEndpoint); logger.info("givePermission status " + * + perStatus + " for " + file.getName()); if (perStatus == 409) { + * logger.info("Permissions already exist or limit was reached for " + + * file.getName()); } else if (perStatus == 400) { + * logger.info("No file in Globus " + file.getName()); } else if (perStatus != + * 201) { logger.info("Cannot get permission for " + file.getName()); } } } } + */ + + int perStatus = givePermission("all_authenticated_users", "", "r", clientTokenUser, directory, + globusEndpoint); + logger.info("givePermission status " + perStatus); + if (perStatus == 409) { + logger.info("Permissions already exist or limit was reached"); + } else if (perStatus == 400) { + logger.info("No directory in Globus"); + } else if (perStatus != 201 && perStatus != 200) { + logger.info("Cannot give read permission"); + return false; + } + + } else if (status.status == 404) { + logger.info("There is no globus directory"); + } else { + logger.severe("Cannot find directory in globus, status " + status); + return false; + } + + return true; + } + + // Generates the URL to launch the Globus app + public String getGlobusAppUrlForDataset(Dataset d) { + return getGlobusAppUrlForDataset(d, true, null); + } + + public String getGlobusAppUrlForDataset(Dataset d, boolean upload, DataFile df) { + String localeCode = session.getLocaleCode(); + ApiToken apiToken = null; + User user = session.getUser(); + + if (user instanceof AuthenticatedUser) { + apiToken = authSvc.findApiTokenByUser((AuthenticatedUser) user); + + if ((apiToken == null) || (apiToken.getExpireTime().before(new Date()))) { + logger.fine("Created apiToken for user: " + user.getIdentifier()); + apiToken = authSvc.generateApiTokenForUser((AuthenticatedUser) user); + } + } + String storePrefix = ""; + String driverId = d.getEffectiveStorageDriverId(); + try { + storePrefix = DataAccess.getDriverPrefix(driverId); + } catch (Exception e) { + logger.warning("GlobusAppUrlForDataset: Failed to get storePrefix for " + driverId); + } + //Use URLTokenUtil for params currently in common with external tools. + URLTokenUtil tokenUtil = new URLTokenUtil(d, df, apiToken, localeCode); + String appUrl; + if (upload) { + appUrl = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost") + + "/upload?datasetPid={datasetPid}&siteUrl={siteUrl}&apiToken={apiToken}&datasetId={datasetId}&datasetVersion={datasetVersion}&dvLocale={localeCode}"; + } else { + if (df == null) { + appUrl = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost") + + "/download?datasetPid={datasetPid}&siteUrl={siteUrl}" + + ((apiToken != null) ? "&apiToken={apiToken}" : "") + + "&datasetId={datasetId}&datasetVersion={datasetVersion}&dvLocale={localeCode}"; + } else { + String rawStorageId = df.getStorageIdentifier(); + rawStorageId=rawStorageId.substring(rawStorageId.lastIndexOf(":")+1); + appUrl = settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusAppUrl, "http://localhost") + + "/download-file?datasetPid={datasetPid}&siteUrl={siteUrl}" + + ((apiToken != null) ? "&apiToken={apiToken}" : "") + + "&datasetId={datasetId}&datasetVersion={datasetVersion}&dvLocale={localeCode}&fileId={fileId}&storageIdentifier=" + + rawStorageId + "&fileName=" + df.getCurrentName(); + } + } + return tokenUtil.replaceTokensWithValues(appUrl) + "&storePrefix=" + storePrefix; + } + + public String getGlobusDownloadScript(Dataset dataset, ApiToken apiToken) { + return URLTokenUtil.getScriptForUrl(getGlobusAppUrlForDataset(dataset, false, null)); + + } + + @Asynchronous + @TransactionAttribute(TransactionAttributeType.REQUIRES_NEW) + public void globusUpload(String jsonData, ApiToken token, Dataset dataset, String httpRequestUrl, + AuthenticatedUser authUser) throws ExecutionException, InterruptedException, MalformedURLException { + + Integer countAll = 0; + Integer countSuccess = 0; + Integer countError = 0; + String logTimestamp = logFormatter.format(new Date()); + Logger globusLogger = Logger.getLogger( + "edu.harvard.iq.dataverse.upload.client.DatasetServiceBean." + "GlobusUpload" + logTimestamp); + String logFileName = "../logs" + File.separator + "globusUpload_id_" + dataset.getId() + "_" + logTimestamp + + ".log"; + FileHandler fileHandler; + boolean fileHandlerSuceeded; + try { + fileHandler = new FileHandler(logFileName); + globusLogger.setUseParentHandlers(false); + fileHandlerSuceeded = true; + } catch (IOException | SecurityException ex) { + Logger.getLogger(DatasetServiceBean.class.getName()).log(Level.SEVERE, null, ex); + return; + } + + if (fileHandlerSuceeded) { + globusLogger.addHandler(fileHandler); + } else { + globusLogger = logger; + } + + globusLogger.info("Starting an globusUpload "); + + String datasetIdentifier = dataset.getStorageIdentifier(); + + // ToDo - use DataAccess methods? + String storageType = datasetIdentifier.substring(0, datasetIdentifier.indexOf("://") + 3); + datasetIdentifier = datasetIdentifier.substring(datasetIdentifier.indexOf("://") + 3); + + Thread.sleep(5000); + + JsonObject jsonObject = null; + try (StringReader rdr = new StringReader(jsonData)) { + jsonObject = Json.createReader(rdr).readObject(); + } catch (Exception jpe) { + jpe.printStackTrace(); + logger.log(Level.SEVERE, "Error parsing dataset json. Json: {0}"); + } + logger.info("json: " + JsonUtil.prettyPrint(jsonObject)); + + String taskIdentifier = jsonObject.getString("taskIdentifier"); + + String ruleId = ""; + try { + ruleId = jsonObject.getString("ruleId"); + } catch (NullPointerException npe) { + logger.warning("NPE for jsonData object"); + } + + // globus task status check + GlobusTask task = globusStatusCheck(taskIdentifier, globusLogger); + String taskStatus = getTaskStatus(task); + + if (ruleId.length() > 0) { + deletePermision(ruleId, globusLogger); + } + + // If success, switch to an EditInProgress lock - do this before removing the + // GlobusUpload lock + // Keeping a lock through the add datafiles API call avoids a conflicting edit + // and keeps any open dataset page refreshing until the datafile appears + if (!(taskStatus.startsWith("FAILED") || taskStatus.startsWith("INACTIVE"))) { + datasetSvc.addDatasetLock(dataset, + new DatasetLock(DatasetLock.Reason.EditInProgress, authUser, "Completing Globus Upload")); + } + + DatasetLock gLock = dataset.getLockFor(DatasetLock.Reason.GlobusUpload); + if (gLock == null) { + logger.log(Level.WARNING, "No lock found for dataset"); + } else { + logger.log(Level.FINE, "Removing GlobusUpload lock " + gLock.getId()); + /* + * Note: This call to remove a lock only works immediately because it is in + * another service bean. Despite the removeDatasetLocks method having the + * REQUIRES_NEW transaction annotation, when the globusUpload method and that + * method were in the same bean (globusUpload was in the DatasetServiceBean to + * start), the globus lock was still seen in the API call initiated in the + * addFilesAsync method called within the globusUpload method. I.e. it appeared + * that the lock removal was not committed/visible outside this method until + * globusUpload itself ended. + */ + datasetSvc.removeDatasetLocks(dataset, DatasetLock.Reason.GlobusUpload); + } + + if (taskStatus.startsWith("FAILED") || taskStatus.startsWith("INACTIVE")) { + String comment = "Reason : " + taskStatus.split("#")[1] + "
Short Description : " + + taskStatus.split("#")[2]; + userNotificationService.sendNotification((AuthenticatedUser) authUser, new Timestamp(new Date().getTime()), + UserNotification.Type.GLOBUSUPLOADCOMPLETEDWITHERRORS, dataset.getId(), comment, true); + globusLogger.info("Globus task failed "); + + } else { + try { + // + + List inputList = new ArrayList(); + JsonArray filesJsonArray = jsonObject.getJsonArray("files"); + + if (filesJsonArray != null) { + + for (JsonObject fileJsonObject : filesJsonArray.getValuesAs(JsonObject.class)) { + + // storageIdentifier s3://gcs5-bucket1:1781cfeb8a7-748c270a227c from + // externalTool + String storageIdentifier = fileJsonObject.getString("storageIdentifier"); + String[] bits = storageIdentifier.split(":"); + String bucketName = bits[1].replace("/", ""); + String fileId = bits[bits.length - 1]; + + // fullpath s3://gcs5-bucket1/10.5072/FK2/3S6G2E/1781cfeb8a7-4ad9418a5873 + String fullPath = storageType + bucketName + "/" + datasetIdentifier + "/" + fileId; + String fileName = fileJsonObject.getString("fileName"); + + inputList.add(fileId + "IDsplit" + fullPath + "IDsplit" + fileName); + } + + // calculateMissingMetadataFields: checksum, mimetype + JsonObject newfilesJsonObject = calculateMissingMetadataFields(inputList, globusLogger); + JsonArray newfilesJsonArray = newfilesJsonObject.getJsonArray("files"); + + JsonArrayBuilder jsonDataSecondAPI = Json.createArrayBuilder(); + + for (JsonObject fileJsonObject : filesJsonArray.getValuesAs(JsonObject.class)) { + + countAll++; + String storageIdentifier = fileJsonObject.getString("storageIdentifier"); + String fileName = fileJsonObject.getString("fileName"); + String directoryLabel = fileJsonObject.getString("directoryLabel"); + String[] bits = storageIdentifier.split(":"); + String fileId = bits[bits.length - 1]; + + List newfileJsonObject = IntStream.range(0, newfilesJsonArray.size()) + .mapToObj(index -> ((JsonObject) newfilesJsonArray.get(index)).getJsonObject(fileId)) + .filter(Objects::nonNull).collect(Collectors.toList()); + + if (newfileJsonObject != null) { + if (!newfileJsonObject.get(0).getString("hash").equalsIgnoreCase("null")) { + JsonPatch path = Json.createPatchBuilder() + .add("/md5Hash", newfileJsonObject.get(0).getString("hash")).build(); + fileJsonObject = path.apply(fileJsonObject); + path = Json.createPatchBuilder() + .add("/mimeType", newfileJsonObject.get(0).getString("mime")).build(); + fileJsonObject = path.apply(fileJsonObject); + jsonDataSecondAPI.add(fileJsonObject); + countSuccess++; + } else { + globusLogger.info(fileName + + " will be skipped from adding to dataset by second API due to missing values "); + countError++; + } + } else { + globusLogger.info(fileName + + " will be skipped from adding to dataset by second API due to missing values "); + countError++; + } + } + + String newjsonData = jsonDataSecondAPI.build().toString(); + + globusLogger.info("Successfully generated new JsonData for Second API call"); + + String command = "curl -H \"X-Dataverse-key:" + token.getTokenString() + "\" -X POST " + + httpRequestUrl + "/api/datasets/:persistentId/addFiles?persistentId=doi:" + + datasetIdentifier + " -F jsonData='" + newjsonData + "'"; + System.out.println("*******====command ==== " + command); + + String output = addFilesAsync(command, globusLogger); + if (output.equalsIgnoreCase("ok")) { + // if(!taskSkippedFiles) + if (countError == 0) { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), UserNotification.Type.GLOBUSUPLOADCOMPLETED, + dataset.getId(), countSuccess + " files added out of " + countAll, true); + } else { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), + UserNotification.Type.GLOBUSUPLOADCOMPLETEDWITHERRORS, dataset.getId(), + countSuccess + " files added out of " + countAll, true); + } + globusLogger.info("Successfully completed api/datasets/:persistentId/addFiles call "); + } else { + globusLogger.log(Level.SEVERE, + "******* Error while executing api/datasets/:persistentId/add call ", command); + } + + } + + globusLogger.info("Files processed: " + countAll.toString()); + globusLogger.info("Files added successfully: " + countSuccess.toString()); + globusLogger.info("Files failures: " + countError.toString()); + globusLogger.info("Finished upload via Globus job."); + + if (fileHandlerSuceeded) { + fileHandler.close(); + } + + } catch (Exception e) { + logger.info("Exception from globusUpload call "); + e.printStackTrace(); + globusLogger.info("Exception from globusUpload call " + e.getMessage()); + datasetSvc.removeDatasetLocks(dataset, DatasetLock.Reason.EditInProgress); + } + } + } + + public String addFilesAsync(String curlCommand, Logger globusLogger) + throws ExecutionException, InterruptedException { + CompletableFuture addFilesFuture = CompletableFuture.supplyAsync(() -> { + try { + Thread.sleep(2000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + return (addFiles(curlCommand, globusLogger)); + }, executor).exceptionally(ex -> { + globusLogger.fine("Something went wrong : " + ex.getLocalizedMessage()); + ex.printStackTrace(); + return null; + }); + + String result = addFilesFuture.get(); + + return result; + } + + private String addFiles(String curlCommand, Logger globusLogger) { + ProcessBuilder processBuilder = new ProcessBuilder(); + Process process = null; + String line; + String status = ""; + + try { + globusLogger.info("Call to : " + curlCommand); + processBuilder.command("bash", "-c", curlCommand); + process = processBuilder.start(); + process.waitFor(); + + BufferedReader br = new BufferedReader(new InputStreamReader(process.getInputStream())); + + StringBuilder sb = new StringBuilder(); + while ((line = br.readLine()) != null) + sb.append(line); + globusLogger.info(" API Output : " + sb.toString()); + JsonObject jsonObject = null; + try (StringReader rdr = new StringReader(sb.toString())) { + jsonObject = Json.createReader(rdr).readObject(); + } catch (Exception jpe) { + jpe.printStackTrace(); + globusLogger.log(Level.SEVERE, "Error parsing dataset json."); + } + + status = jsonObject.getString("status"); + } catch (Exception ex) { + globusLogger.log(Level.SEVERE, + "******* Unexpected Exception while executing api/datasets/:persistentId/add call ", ex); + } + + return status; + } + + @Asynchronous + public void globusDownload(String jsonData, Dataset dataset, User authUser) throws MalformedURLException { + + String logTimestamp = logFormatter.format(new Date()); + Logger globusLogger = Logger.getLogger( + "edu.harvard.iq.dataverse.upload.client.DatasetServiceBean." + "GlobusDownload" + logTimestamp); + + String logFileName = "../logs" + File.separator + "globusDownload_id_" + dataset.getId() + "_" + logTimestamp + + ".log"; + FileHandler fileHandler; + boolean fileHandlerSuceeded; + try { + fileHandler = new FileHandler(logFileName); + globusLogger.setUseParentHandlers(false); + fileHandlerSuceeded = true; + } catch (IOException | SecurityException ex) { + Logger.getLogger(DatasetServiceBean.class.getName()).log(Level.SEVERE, null, ex); + return; + } + + if (fileHandlerSuceeded) { + globusLogger.addHandler(fileHandler); + } else { + globusLogger = logger; + } + + globusLogger.info("Starting an globusDownload "); + + JsonObject jsonObject = null; + try (StringReader rdr = new StringReader(jsonData)) { + jsonObject = Json.createReader(rdr).readObject(); + } catch (Exception jpe) { + jpe.printStackTrace(); + globusLogger.log(Level.SEVERE, "Error parsing dataset json. Json: {0}"); + } + + String taskIdentifier = jsonObject.getString("taskIdentifier"); + String ruleId = ""; + + try { + jsonObject.getString("ruleId"); + } catch (NullPointerException npe) { + + } + + // globus task status check + GlobusTask task = globusStatusCheck(taskIdentifier, globusLogger); + String taskStatus = getTaskStatus(task); + + if (ruleId.length() > 0) { + deletePermision(ruleId, globusLogger); + } + + if (taskStatus.startsWith("FAILED") || taskStatus.startsWith("INACTIVE")) { + String comment = "Reason : " + taskStatus.split("#")[1] + "
Short Description : " + + taskStatus.split("#")[2]; + userNotificationService.sendNotification((AuthenticatedUser) authUser, new Timestamp(new Date().getTime()), + UserNotification.Type.GLOBUSDOWNLOADCOMPLETEDWITHERRORS, dataset.getId(), comment, true); + globusLogger.info("Globus task failed during download process"); + } else { + boolean taskSkippedFiles = (task.getSkip_source_errors() == null) ? false : task.getSkip_source_errors(); + if (!taskSkippedFiles) { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), UserNotification.Type.GLOBUSDOWNLOADCOMPLETED, + dataset.getId()); + } else { + userNotificationService.sendNotification((AuthenticatedUser) authUser, + new Timestamp(new Date().getTime()), UserNotification.Type.GLOBUSDOWNLOADCOMPLETEDWITHERRORS, + dataset.getId(), ""); + } + } + } + + Executor executor = Executors.newFixedThreadPool(10); + + private GlobusTask globusStatusCheck(String taskId, Logger globusLogger) throws MalformedURLException { + boolean taskCompletion = false; + String status = ""; + GlobusTask task = null; + int pollingInterval = SystemConfig.getIntLimitFromStringOrDefault(settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusPollingInterval), 50); + do { + try { + globusLogger.info("checking globus transfer task " + taskId); + Thread.sleep(pollingInterval * 1000); + AccessToken clientTokenUser = getClientToken(); + // success = globusServiceBean.getSuccessfulTransfers(clientTokenUser, taskId); + task = getTask(clientTokenUser, taskId, globusLogger); + if (task != null) { + status = task.getStatus(); + if (status != null) { + // The task is in progress. + if (status.equalsIgnoreCase("ACTIVE")) { + if (task.getNice_status().equalsIgnoreCase("ok") + || task.getNice_status().equalsIgnoreCase("queued")) { + taskCompletion = false; + } else { + taskCompletion = true; + // status = "FAILED" + "#" + task.getNice_status() + "#" + + // task.getNice_status_short_description(); + } + } else { + // The task is either succeeded, failed or inactive. + taskCompletion = true; + // status = status + "#" + task.getNice_status() + "#" + + // task.getNice_status_short_description(); + } + } else { + // status = "FAILED"; + taskCompletion = true; + } + } else { + // status = "FAILED"; + taskCompletion = true; + } + } catch (Exception ex) { + ex.printStackTrace(); + } + + } while (!taskCompletion); + + globusLogger.info("globus transfer task completed successfully"); + return task; + } + + private String getTaskStatus(GlobusTask task) { + String status = null; + if (task != null) { + status = task.getStatus(); + if (status != null) { + // The task is in progress. + if (status.equalsIgnoreCase("ACTIVE")) { + status = "FAILED" + "#" + task.getNice_status() + "#" + task.getNice_status_short_description(); + } else { + // The task is either succeeded, failed or inactive. + status = status + "#" + task.getNice_status() + "#" + task.getNice_status_short_description(); + } + } else { + status = "FAILED"; + } + } else { + status = "FAILED"; + } + return status; + } + + public JsonObject calculateMissingMetadataFields(List inputList, Logger globusLogger) + throws InterruptedException, ExecutionException, IOException { + + List> hashvalueCompletableFutures = inputList.stream() + .map(iD -> calculateDetailsAsync(iD, globusLogger)).collect(Collectors.toList()); + + CompletableFuture allFutures = CompletableFuture + .allOf(hashvalueCompletableFutures.toArray(new CompletableFuture[hashvalueCompletableFutures.size()])); + + CompletableFuture> allCompletableFuture = allFutures.thenApply(future -> { + return hashvalueCompletableFutures.stream().map(completableFuture -> completableFuture.join()) + .collect(Collectors.toList()); + }); + + CompletableFuture completableFuture = allCompletableFuture.thenApply(files -> { + return files.stream().map(d -> json(d)).collect(toJsonArray()); + }); + + JsonArrayBuilder filesObject = (JsonArrayBuilder) completableFuture.get(); + + JsonObject output = Json.createObjectBuilder().add("files", filesObject).build(); + + return output; + + } + + private CompletableFuture calculateDetailsAsync(String id, Logger globusLogger) { + // logger.info(" calcualte additional details for these globus id ==== " + id); + + return CompletableFuture.supplyAsync(() -> { + try { + Thread.sleep(2000); + } catch (InterruptedException e) { + e.printStackTrace(); + } + try { + return (calculateDetails(id, globusLogger)); + } catch (InterruptedException | IOException e) { + e.printStackTrace(); + } + return null; + }, executor).exceptionally(ex -> { + return null; + }); + } + + private FileDetailsHolder calculateDetails(String id, Logger globusLogger) + throws InterruptedException, IOException { + int count = 0; + String checksumVal = ""; + InputStream in = null; + String fileId = id.split("IDsplit")[0]; + String fullPath = id.split("IDsplit")[1]; + String fileName = id.split("IDsplit")[2]; + + // ToDo: what if the file doesnot exists in s3 + // ToDo: what if checksum calculation failed + + do { + try { + StorageIO dataFileStorageIO = DataAccess.getDirectStorageIO(fullPath); + in = dataFileStorageIO.getInputStream(); + checksumVal = FileUtil.calculateChecksum(in, DataFile.ChecksumType.MD5); + count = 3; + } catch (IOException ioex) { + count = 3; + logger.info(ioex.getMessage()); + globusLogger.info("S3AccessIO: DataFile (fullPAth " + fullPath + + ") does not appear to be an S3 object associated with driver: "); + } catch (Exception ex) { + count = count + 1; + ex.printStackTrace(); + logger.info(ex.getMessage()); + Thread.sleep(5000); + } + + } while (count < 3); + + if (checksumVal.length() == 0) { + checksumVal = "NULL"; + } + + String mimeType = calculatemime(fileName); + globusLogger.info(" File Name " + fileName + " File Details " + fileId + " checksum = " + checksumVal + + " mimeType = " + mimeType); + return new FileDetailsHolder(fileId, checksumVal, mimeType); + // getBytes(in)+"" ); + // calculatemime(fileName)); + } + + public String calculatemime(String fileName) throws InterruptedException { + + String finalType = FileUtil.MIME_TYPE_UNDETERMINED_DEFAULT; + String type = FileUtil.determineFileTypeByNameAndExtension(fileName); + + if (type!=null && !type.isBlank()) { + if (FileUtil.useRecognizedType(finalType, type)) { + finalType = type; + } + } + + return finalType; + } + /* + * public boolean globusFinishTransfer(Dataset dataset, AuthenticatedUser user) + * throws MalformedURLException { + * + * logger.info("=====Tasklist == dataset id :" + dataset.getId()); String + * directory = null; + * + * try { + * + * List fileMetadatas = new ArrayList<>(); + * + * StorageIO datasetSIO = DataAccess.getStorageIO(dataset); + * + * + * + * DatasetVersion workingVersion = dataset.getEditVersion(); + * + * if (workingVersion.getCreateTime() != null) { + * workingVersion.setCreateTime(new Timestamp(new Date().getTime())); } + * + * directory = dataset.getAuthorityForFileStorage() + "/" + + * dataset.getIdentifierForFileStorage(); + * + * System.out.println("======= directory ==== " + directory + + * " ==== datasetId :" + dataset.getId()); Map checksumMapOld + * = new HashMap<>(); + * + * Iterator fmIt = workingVersion.getFileMetadatas().iterator(); + * + * while (fmIt.hasNext()) { FileMetadata fm = fmIt.next(); if (fm.getDataFile() + * != null && fm.getDataFile().getId() != null) { String chksum = + * fm.getDataFile().getChecksumValue(); if (chksum != null) { + * checksumMapOld.put(chksum, 1); } } } + * + * List dFileList = new ArrayList<>(); boolean update = false; for + * (S3ObjectSummary s3ObjectSummary : datasetSIO.listAuxObjects("")) { + * + * String s3ObjectKey = s3ObjectSummary.getKey(); + * + * + * String t = s3ObjectKey.replace(directory, ""); + * + * if (t.indexOf(".") > 0) { long totalSize = s3ObjectSummary.getSize(); String + * filePath = s3ObjectKey; String fileName = + * filePath.split("/")[filePath.split("/").length - 1]; String fullPath = + * datasetSIO.getStorageLocation() + "/" + fileName; + * + * logger.info("Full path " + fullPath); StorageIO dataFileStorageIO = + * DataAccess.getDirectStorageIO(fullPath); InputStream in = + * dataFileStorageIO.getInputStream(); + * + * String checksumVal = FileUtil.calculateChecksum(in, + * DataFile.ChecksumType.MD5); //String checksumVal = s3ObjectSummary.getETag(); + * logger.info("The checksum is " + checksumVal); if + * ((checksumMapOld.get(checksumVal) != null)) { logger.info("datasetId :" + + * dataset.getId() + "======= filename ==== " + filePath + + * " == file already exists "); } else if (filePath.contains("cached") || + * filePath.contains(".thumb")) { logger.info(filePath + " is ignored"); } else + * { update = true; logger.info("datasetId :" + dataset.getId() + + * "======= filename ==== " + filePath + " == new file "); try { + * + * DataFile datafile = new DataFile(DataFileServiceBean.MIME_TYPE_GLOBUS_FILE); + * //MIME_TYPE_GLOBUS datafile.setModificationTime(new Timestamp(new + * Date().getTime())); datafile.setCreateDate(new Timestamp(new + * Date().getTime())); datafile.setPermissionModificationTime(new Timestamp(new + * Date().getTime())); + * + * FileMetadata fmd = new FileMetadata(); + * + * + * fmd.setLabel(fileName); fmd.setDirectoryLabel(filePath.replace(directory, + * "").replace(File.separator + fileName, "")); + * + * fmd.setDataFile(datafile); + * + * datafile.getFileMetadatas().add(fmd); + * + * FileUtil.generateS3PackageStorageIdentifierForGlobus(datafile); + * logger.info("==== datasetId :" + dataset.getId() + "======= filename ==== " + * + filePath + " == added to datafile, filemetadata "); + * + * try { // We persist "SHA1" rather than "SHA-1". + * //datafile.setChecksumType(DataFile.ChecksumType.SHA1); + * datafile.setChecksumType(DataFile.ChecksumType.MD5); + * datafile.setChecksumValue(checksumVal); } catch (Exception cksumEx) { + * logger.info("==== datasetId :" + dataset.getId() + + * "======Could not calculate checksumType signature for the new file "); } + * + * datafile.setFilesize(totalSize); + * + * dFileList.add(datafile); + * + * } catch (Exception ioex) { logger.info("datasetId :" + dataset.getId() + + * "======Failed to process and/or save the file " + ioex.getMessage()); return + * false; + * + * } } } } if (update) { + * + * List filesAdded = new ArrayList<>(); + * + * if (dFileList != null && dFileList.size() > 0) { + * + * // Dataset dataset = version.getDataset(); + * + * for (DataFile dataFile : dFileList) { + * + * if (dataFile.getOwner() == null) { dataFile.setOwner(dataset); + * + * workingVersion.getFileMetadatas().add(dataFile.getFileMetadata()); + * dataFile.getFileMetadata().setDatasetVersion(workingVersion); + * dataset.getFiles().add(dataFile); + * + * } + * + * filesAdded.add(dataFile); + * + * } + * + * logger.info("==== datasetId :" + dataset.getId() + + * " ===== Done! Finished saving new files to the dataset."); } + * + * fileMetadatas.clear(); for (DataFile addedFile : filesAdded) { + * fileMetadatas.add(addedFile.getFileMetadata()); } filesAdded = null; + * + * if (workingVersion.isDraft()) { + * + * logger.info("Async: ==== datasetId :" + dataset.getId() + + * " ==== inside draft version "); + * + * Timestamp updateTime = new Timestamp(new Date().getTime()); + * + * workingVersion.setLastUpdateTime(updateTime); + * dataset.setModificationTime(updateTime); + * + * + * for (FileMetadata fileMetadata : fileMetadatas) { + * + * if (fileMetadata.getDataFile().getCreateDate() == null) { + * fileMetadata.getDataFile().setCreateDate(updateTime); + * fileMetadata.getDataFile().setCreator((AuthenticatedUser) user); } + * fileMetadata.getDataFile().setModificationTime(updateTime); } + * + * + * } else { logger.info("datasetId :" + dataset.getId() + + * " ==== inside released version "); + * + * for (int i = 0; i < workingVersion.getFileMetadatas().size(); i++) { for + * (FileMetadata fileMetadata : fileMetadatas) { if + * (fileMetadata.getDataFile().getStorageIdentifier() != null) { + * + * if (fileMetadata.getDataFile().getStorageIdentifier().equals(workingVersion. + * getFileMetadatas().get(i).getDataFile().getStorageIdentifier())) { + * workingVersion.getFileMetadatas().set(i, fileMetadata); } } } } + * + * + * } + * + * + * try { Command cmd; logger.info("Async: ==== datasetId :" + + * dataset.getId() + + * " ======= UpdateDatasetVersionCommand START in globus function "); cmd = new + * UpdateDatasetVersionCommand(dataset, new DataverseRequest(user, + * (HttpServletRequest) null)); ((UpdateDatasetVersionCommand) + * cmd).setValidateLenient(true); //new DataverseRequest(authenticatedUser, + * (HttpServletRequest) null) //dvRequestService.getDataverseRequest() + * commandEngine.submit(cmd); } catch (CommandException ex) { + * logger.log(Level.WARNING, "==== datasetId :" + dataset.getId() + + * "======CommandException updating DatasetVersion from batch job: " + + * ex.getMessage()); return false; } + * + * logger.info("==== datasetId :" + dataset.getId() + + * " ======= GLOBUS CALL COMPLETED SUCCESSFULLY "); + * + * //return true; } + * + * } catch (Exception e) { String message = e.getMessage(); + * + * logger.info("==== datasetId :" + dataset.getId() + + * " ======= GLOBUS CALL Exception ============== " + message); + * e.printStackTrace(); return false; //return + * error(Response.Status.INTERNAL_SERVER_ERROR, + * "Uploaded files have passed checksum validation but something went wrong while attempting to move the files into Dataverse. Message was '" + * + message + "'."); } + * + * String globusBasicToken = + * settingsSvc.getValueForKey(SettingsServiceBean.Key.GlobusBasicToken, ""); + * AccessToken clientTokenUser = getClientToken(globusBasicToken); + * updatePermision(clientTokenUser, directory, "identity", "r"); return true; } + * + */ +} diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/GlobusTask.java b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusTask.java new file mode 100644 index 00000000000..c2b01779f4a --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/GlobusTask.java @@ -0,0 +1,92 @@ +package edu.harvard.iq.dataverse.globus; + +public class GlobusTask { + + private String DATA_TYPE; + private String type; + private String status; + private String owner_id; + private String request_time; + private String task_id; + private String destination_endpoint_display_name; + private boolean skip_source_errors; + private String nice_status; + private String nice_status_short_description; + + public String getDestination_endpoint_display_name() { + return destination_endpoint_display_name; + } + + public void setDestination_endpoint_display_name(String destination_endpoint_display_name) { + this.destination_endpoint_display_name = destination_endpoint_display_name; + } + + public void setRequest_time(String request_time) { + this.request_time = request_time; + } + + public String getRequest_time() { + return request_time; + } + + public String getTask_id() { + return task_id; + } + + public void setTask_id(String task_id) { + this.task_id = task_id; + } + + public String getDATA_TYPE() { + return DATA_TYPE; + } + + public void setDATA_TYPE(String DATA_TYPE) { + this.DATA_TYPE = DATA_TYPE; + } + + public String getType() { + return type; + } + + public void setType(String type) { + this.type = type; + } + + public String getStatus() { + return status; + } + + public void setStatus(String status) { + this.status = status; + } + + public String getOwner_id() { + return owner_id; + } + + public void setOwner_id(String owner_id) { + this.owner_id = owner_id; + } + + public Boolean getSkip_source_errors() { + return skip_source_errors; + } + + public void setSkip_source_errors(Boolean skip_source_errors) { + this.skip_source_errors = skip_source_errors; + } + + public String getNice_status() { + return nice_status; + } + + public void setNice_status(String nice_status) { + this.nice_status = nice_status; + } + + public String getNice_status_short_description() { + return nice_status_short_description; + } + +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/globus/Permissions.java b/src/main/java/edu/harvard/iq/dataverse/globus/Permissions.java new file mode 100644 index 00000000000..b8bb5193fa4 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/globus/Permissions.java @@ -0,0 +1,58 @@ +package edu.harvard.iq.dataverse.globus; + +public class Permissions { + private String DATA_TYPE; + private String principal_type; + private String principal; + private String id; + private String path; + private String permissions; + + public void setPath(String path) { + this.path = path; + } + + public void setDATA_TYPE(String DATA_TYPE) { + this.DATA_TYPE = DATA_TYPE; + } + + public void setPermissions(String permissions) { + this.permissions = permissions; + } + + public void setPrincipal(String principal) { + this.principal = principal; + } + + public void setPrincipalType(String principalType) { + this.principal_type = principalType; + } + + public String getPath() { + return path; + } + + public String getDATA_TYPE() { + return DATA_TYPE; + } + + public String getPermissions() { + return permissions; + } + + public String getPrincipal() { + return principal; + } + + public String getPrincipalType() { + return principal_type; + } + + public void setId(String id) { + this.id = id; + } + + public String getId() { + return id; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java index c12b8f6e452..bb68152eeba 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/SettingsServiceBean.java @@ -457,6 +457,31 @@ Whether Harvesting (OAI) service is enabled * when the Distributor field (citation metadatablock) is set (true) */ ExportInstallationAsDistributorOnlyWhenNotSet, + + /** + * Basic Globus Token for Globus Application + */ + GlobusBasicToken, + /** + * GlobusEndpoint is Globus endpoint for Globus application + */ + GlobusEndpoint, + /** + * Comma separated list of Globus enabled stores + */ + GlobusStores, + /** Globus App URL + * + */ + GlobusAppUrl, + /** Globus Polling Interval how long in seconds Dataverse waits between checks on Globus upload status checks + * + */ + GlobusPollingInterval, + /**Enable single-file download/transfers for Globus + * + */ + GlobusSingleFileTransfer, /** * Optional external executables to run on the metadata for dataverses * and datasets being published; as an extra validation step, to diff --git a/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java index 4e97df82a8f..339de904f9e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java @@ -1189,7 +1189,7 @@ public static CreateDataFileResult createDataFiles(DatasetVersion version, Input } // end createDataFiles - private static boolean useRecognizedType(String suppliedContentType, String recognizedType) { + public static boolean useRecognizedType(String suppliedContentType, String recognizedType) { // is it any better than the type that was supplied to us, // if any? // This is not as trivial a task as one might expect... @@ -1621,32 +1621,33 @@ public static String getPublicDownloadUrl(String dataverseSiteUrl, String persis */ public static String getFileDownloadUrlPath(String downloadType, Long fileId, boolean gbRecordsWritten, Long fileMetadataId) { String fileDownloadUrl = "/api/access/datafile/" + fileId; - if (downloadType != null && downloadType.equals("bundle")) { - if (fileMetadataId == null) { - fileDownloadUrl = "/api/access/datafile/bundle/" + fileId; - } else { - fileDownloadUrl = "/api/access/datafile/bundle/" + fileId + "?fileMetadataId=" + fileMetadataId; - } - } - if (downloadType != null && downloadType.equals("original")) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=original"; - } - if (downloadType != null && downloadType.equals("RData")) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=RData"; - } - if (downloadType != null && downloadType.equals("var")) { - if (fileMetadataId == null) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata"; - } else { - fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata?fileMetadataId=" + fileMetadataId; + if (downloadType != null) { + switch(downloadType) { + case "original": + case"RData": + case "tab": + case "GlobusTransfer": + fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=" + downloadType; + break; + case "bundle": + if (fileMetadataId == null) { + fileDownloadUrl = "/api/access/datafile/bundle/" + fileId; + } else { + fileDownloadUrl = "/api/access/datafile/bundle/" + fileId + "?fileMetadataId=" + fileMetadataId; + } + break; + case "var": + if (fileMetadataId == null) { + fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata"; + } else { + fileDownloadUrl = "/api/access/datafile/" + fileId + "/metadata?fileMetadataId=" + fileMetadataId; + } + break; + } + } - } - if (downloadType != null && downloadType.equals("tab")) { - fileDownloadUrl = "/api/access/datafile/" + fileId + "?format=tab"; - } if (gbRecordsWritten) { - if (downloadType != null && ((downloadType.equals("original") || downloadType.equals("RData") || downloadType.equals("tab")) || - ((downloadType.equals("var") || downloadType.equals("bundle") ) && fileMetadataId != null))) { + if (fileDownloadUrl.contains("?")) { fileDownloadUrl += "&gbrecs=true"; } else { fileDownloadUrl += "?gbrecs=true"; @@ -1784,10 +1785,10 @@ public static void validateDataFileChecksum(DataFile dataFile) throws IOExceptio StorageIO storage = dataFile.getStorageIO(); InputStream in = null; - + try { storage.open(DataAccessOption.READ_ACCESS); - + if (!dataFile.isTabularData()) { in = storage.getInputStream(); } else { @@ -1842,7 +1843,7 @@ public static void validateDataFileChecksum(DataFile dataFile) throws IOExceptio } finally { IOUtils.closeQuietly(in); } - // try again: + // try again: if (recalculatedChecksum.equals(dataFile.getChecksumValue())) { fixed = true; try { @@ -1853,7 +1854,7 @@ public static void validateDataFileChecksum(DataFile dataFile) throws IOExceptio } } } - + if (!fixed) { String info = BundleUtil.getStringFromBundle("dataset.publish.file.validation.error.wrongChecksumValue", Arrays.asList(dataFile.getId().toString())); logger.log(Level.INFO, info); diff --git a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java index 03ab6da1d31..e76e2c5696b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/MailUtil.java @@ -75,6 +75,38 @@ public static String getSubjectTextBasedOnNotification(UserNotification userNoti } catch (Exception e) { return BundleUtil.getStringFromBundle("notification.email.import.filesystem.subject", rootDvNameAsList); } + case GLOBUSUPLOADCOMPLETED: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompleted.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompleted.subject", rootDvNameAsList); + } + case GLOBUSDOWNLOADCOMPLETED: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompleted.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompleted.subject", rootDvNameAsList); + } + case GLOBUSUPLOADCOMPLETEDWITHERRORS: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompletedWithErrors.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.uploadCompletedWithErrors.subject", rootDvNameAsList); + } + case GLOBUSDOWNLOADCOMPLETEDWITHERRORS: + try { + DatasetVersion version = (DatasetVersion)objectOfNotification; + List dsNameAsList = Arrays.asList(version.getDataset().getDisplayName()); + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompletedWithErrors.subject", dsNameAsList); + } catch (Exception e) { + return BundleUtil.getStringFromBundle("notification.email.globus.downloadCompletedWithErrors.subject", rootDvNameAsList); + } case CHECKSUMIMPORT: return BundleUtil.getStringFromBundle("notification.email.import.checksum.subject", rootDvNameAsList); diff --git a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java index bd27405fae5..7abd0d02065 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java @@ -422,7 +422,7 @@ public static long getLongLimitFromStringOrDefault(String limitSetting, Long def if (limitSetting != null && !limitSetting.equals("")) { try { - limit = new Long(limitSetting); + limit = Long.valueOf(limitSetting); } catch (NumberFormatException nfe) { limit = null; } @@ -431,12 +431,12 @@ public static long getLongLimitFromStringOrDefault(String limitSetting, Long def return limit != null ? limit : defaultValue; } - static int getIntLimitFromStringOrDefault(String limitSetting, Integer defaultValue) { + public static int getIntLimitFromStringOrDefault(String limitSetting, Integer defaultValue) { Integer limit = null; if (limitSetting != null && !limitSetting.equals("")) { try { - limit = new Integer(limitSetting); + limit = Integer.valueOf(limitSetting); } catch (NumberFormatException nfe) { limit = null; } @@ -579,7 +579,7 @@ public Integer getSearchHighlightFragmentSize() { } return null; } - + public long getTabularIngestSizeLimit() { // This method will return the blanket ingestable size limit, if // set on the system. I.e., the universal limit that applies to all @@ -856,7 +856,14 @@ public enum FileUploadMethods { * Traditional Dataverse file handling, which tends to involve users * uploading and downloading files using a browser or APIs. */ - NATIVE("native/http"); + NATIVE("native/http"), + + /** + * Upload through Globus of large files + */ + + GLOBUS("globus") + ; private final String text; @@ -896,7 +903,9 @@ public enum FileDownloadMethods { * go through Glassfish. */ RSYNC("rsal/rsync"), - NATIVE("native/http"); + NATIVE("native/http"), + GLOBUS("globus") + ; private final String text; private FileDownloadMethods(final String text) { @@ -984,15 +993,19 @@ public boolean isPublicInstall(){ } public boolean isRsyncUpload(){ - return getUploadMethodAvailable(SystemConfig.FileUploadMethods.RSYNC.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.RSYNC.toString(), true); } - + + public boolean isGlobusUpload(){ + return getMethodAvailable(FileUploadMethods.GLOBUS.toString(), true); + } + // Controls if HTTP upload is enabled for both GUI and API. public boolean isHTTPUpload(){ - return getUploadMethodAvailable(SystemConfig.FileUploadMethods.NATIVE.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.NATIVE.toString(), true); } - public boolean isRsyncOnly(){ + public boolean isRsyncOnly(){ String downloadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.DownloadMethods); if(downloadMethods == null){ return false; @@ -1005,26 +1018,37 @@ public boolean isRsyncOnly(){ return false; } else { return Arrays.asList(uploadMethods.toLowerCase().split("\\s*,\\s*")).size() == 1 && uploadMethods.toLowerCase().equals(SystemConfig.FileUploadMethods.RSYNC.toString()); - } + } } public boolean isRsyncDownload() { - String downloadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.DownloadMethods); - return downloadMethods !=null && downloadMethods.toLowerCase().contains(SystemConfig.FileDownloadMethods.RSYNC.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.RSYNC.toString(), false); } public boolean isHTTPDownload() { - String downloadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.DownloadMethods); - logger.warning("Download Methods:" + downloadMethods); - return downloadMethods !=null && downloadMethods.toLowerCase().contains(SystemConfig.FileDownloadMethods.NATIVE.toString()); + return getMethodAvailable(SystemConfig.FileUploadMethods.NATIVE.toString(), false); + } + + public boolean isGlobusDownload() { + return getMethodAvailable(FileUploadMethods.GLOBUS.toString(), false); } - private Boolean getUploadMethodAvailable(String method){ - String uploadMethods = settingsService.getValueForKey(SettingsServiceBean.Key.UploadMethods); - if (uploadMethods==null){ + public boolean isGlobusFileDownload() { + return (isGlobusDownload() && settingsService.isTrueForKey(SettingsServiceBean.Key.GlobusSingleFileTransfer, false)); + } + + public List getGlobusStoresList() { + String globusStores = settingsService.getValueForKey(SettingsServiceBean.Key.GlobusStores, ""); + return Arrays.asList(globusStores.split("\\s*,\\s*")); + } + + private Boolean getMethodAvailable(String method, boolean upload) { + String methods = settingsService.getValueForKey( + upload ? SettingsServiceBean.Key.UploadMethods : SettingsServiceBean.Key.DownloadMethods); + if (methods == null) { return false; } else { - return Arrays.asList(uploadMethods.toLowerCase().split("\\s*,\\s*")).contains(method); + return Arrays.asList(methods.toLowerCase().split("\\s*,\\s*")).contains(method); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/util/URLTokenUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/URLTokenUtil.java index 78280cd0f0f..b3d5f9d6b74 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/URLTokenUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/URLTokenUtil.java @@ -55,13 +55,26 @@ public URLTokenUtil(DataFile dataFile, ApiToken apiToken, FileMetadata fileMetad * @param apiToken The apiToken can be null */ public URLTokenUtil(Dataset dataset, ApiToken apiToken, String localeCode) { + this(dataset, null, apiToken, localeCode); + } + + /** + * Dataset level + * + * @param dataset Required. + * @param datafile Optional. + * @param apiToken Optional The apiToken can be null + * @localeCode Optional + * + */ + public URLTokenUtil(Dataset dataset, DataFile datafile, ApiToken apiToken, String localeCode) { if (dataset == null) { String error = "A Dataset is required."; logger.warning("Error in URLTokenUtil constructor: " + error); throw new IllegalArgumentException(error); } this.dataset = dataset; - this.dataFile = null; + this.dataFile = datafile; this.fileMetadata = null; this.apiToken = apiToken; this.localeCode = localeCode; @@ -112,7 +125,7 @@ public String replaceTokensWithValues(String url) { String token = matcher.group(1); ReservedWord reservedWord = ReservedWord.fromString(token); String tValue = getTokenValue(token); - logger.info("Replacing " + reservedWord.toString() + " with " + tValue + " in " + newUrl); + logger.fine("Replacing " + reservedWord.toString() + " with " + tValue + " in " + newUrl); newUrl = newUrl.replace(reservedWord.toString(), tValue); } return newUrl; @@ -171,6 +184,12 @@ private String getTokenValue(String value) { } throw new IllegalArgumentException("Cannot replace reserved word: " + value); } + + public static String getScriptForUrl(String url) { + String msg = BundleUtil.getStringFromBundle("externaltools.enable.browser.popups"); + String script = "const newWin = window.open('" + url + "', target='_blank'); if (!newWin || newWin.closed || typeof newWin.closed == \"undefined\") {alert(\"" + msg + "\");}"; + return script; + } public enum ReservedWord { diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index 76865f860e1..e088122419d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -1,6 +1,23 @@ package edu.harvard.iq.dataverse.util.json; import edu.harvard.iq.dataverse.*; +import edu.harvard.iq.dataverse.AuxiliaryFile; +import edu.harvard.iq.dataverse.ControlledVocabularyValue; +import edu.harvard.iq.dataverse.DataFile; +import edu.harvard.iq.dataverse.DataFileTag; +import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.DatasetDistributor; +import edu.harvard.iq.dataverse.DatasetFieldType; +import edu.harvard.iq.dataverse.DatasetField; +import edu.harvard.iq.dataverse.DatasetFieldCompoundValue; +import edu.harvard.iq.dataverse.DatasetFieldValue; +import edu.harvard.iq.dataverse.DatasetLock; +import edu.harvard.iq.dataverse.DatasetVersion; +import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.DataverseContact; +import edu.harvard.iq.dataverse.DataverseFacet; +import edu.harvard.iq.dataverse.DataverseTheme; +import edu.harvard.iq.dataverse.api.Datasets; import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.authorization.groups.impl.maildomain.MailDomainGroup; import edu.harvard.iq.dataverse.authorization.providers.builtin.BuiltinUser; @@ -19,6 +36,7 @@ import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.dataset.DatasetUtil; import edu.harvard.iq.dataverse.license.License; +import edu.harvard.iq.dataverse.globus.FileDetailsHolder; import edu.harvard.iq.dataverse.privateurl.PrivateUrl; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.DatasetFieldWalker; @@ -60,7 +78,7 @@ public class JsonPrinter { @EJB static DatasetFieldServiceBean datasetFieldService; - + public static void injectSettingsService(SettingsServiceBean ssb, DatasetFieldServiceBean dfsb) { settingsService = ssb; datasetFieldService = dfsb; @@ -329,6 +347,14 @@ public static JsonObjectBuilder json(Dataset ds) { return bld; } + public static JsonObjectBuilder json(FileDetailsHolder ds) { + return Json.createObjectBuilder().add(ds.getStorageID() , + Json.createObjectBuilder() + .add("id", ds.getStorageID() ) + .add("hash", ds.getHash()) + .add("mime",ds.getMime())); + } + public static JsonObjectBuilder json(DatasetVersion dsv) { JsonObjectBuilder bld = jsonObjectBuilder() .add("id", dsv.getId()).add("datasetId", dsv.getDataset().getId()) @@ -471,7 +497,7 @@ public static JsonObjectBuilder json(MetadataBlock block, List fie blockBld.add("name", block.getName()); final JsonArrayBuilder fieldsArray = Json.createArrayBuilder(); - Map cvocMap = (datasetFieldService==null) ? new HashMap() :datasetFieldService.getCVocConf(false); + Map cvocMap = (datasetFieldService==null) ? new HashMap() :datasetFieldService.getCVocConf(false); DatasetFieldWalker.walk(fields, settingsService, cvocMap, new DatasetFieldsToJson(fieldsArray)); blockBld.add("fields", fieldsArray); @@ -687,7 +713,7 @@ public void startField(DatasetField f) { objectStack.peek().add("multiple", typ.isAllowMultiples()); objectStack.peek().add("typeClass", typeClassString(typ)); } - + @Override public void addExpandedValuesArray(DatasetField f) { // Invariant: all values are multiple. Diffrentiation between multiple and single is done at endField. @@ -708,7 +734,7 @@ public void endField(DatasetField f) { f.getDatasetFieldType().isAllowMultiples() ? expandedValues : expandedValues.get(0)); } - + valueArrStack.peek().add(jsonField); } } @@ -724,7 +750,7 @@ public void externalVocabularyValue(DatasetFieldValue dsfv, JsonObject cvocEntry } } } - + @Override public void primitiveValue(DatasetFieldValue dsfv) { if (dsfv.getValue() != null) { diff --git a/src/main/java/propertyFiles/Bundle.properties b/src/main/java/propertyFiles/Bundle.properties index 6934f3a57ac..26a20c40cb3 100644 --- a/src/main/java/propertyFiles/Bundle.properties +++ b/src/main/java/propertyFiles/Bundle.properties @@ -232,8 +232,16 @@ notification.access.revoked.datafile=You have been removed from a role in {0}. notification.checksumfail=One or more files in your upload failed checksum validation for dataset {1}. Please re-run the upload script. If the problem persists, please contact support. notification.ingest.completed=Your Dataset {2} has one or more tabular files that completed the tabular ingest process. These files will be available for download in their original formats and other formats for enhanced archival purposes after you publish the dataset. The archival .tab files are displayed in the file table. Please see the guides for more information about ingest and support for tabular files. notification.ingest.completedwitherrors=Your Dataset {2} has one or more tabular files that have been uploaded successfully but are not supported for tabular ingest. After you publish the dataset, these files will not have additional archival features. Please see the guides for more information about ingest and support for tabular files.

Files with incomplete ingest:{5} -notification.mail.import.filesystem=Dataset {2} ({0}/dataset.xhtml?persistentId={1}) has been successfully uploaded and verified. -notification.import.filesystem=Dataset {1} has been successfully uploaded and verified. +notification.mail.import.filesystem=Globus transfer to Dataset {2} ({0}/dataset.xhtml?persistentId={1}) was successful. File(s) have been uploaded and verified. +notification.mail.globus.upload.completed=Globus transfer to Dataset {2} was successful. File(s) have been uploaded and verified.

{3}
+notification.mail.globus.download.completed=Globus transfer of file(s) from the dataset {2} was successful.

{3}
+notification.mail.globus.upload.completedWithErrors=Globus transfer to Dataset {2} is complete with errors.

{3}
+notification.mail.globus.download.completedWithErrors=Globus transfer from the dataset {2} is complete with errors.

{3}
+notification.import.filesystem=Globus transfer to Dataset {1} was successful. File(s) have been uploaded and verified. +notification.globus.upload.completed=Globus transfer to Dataset {1} was successful. File(s) have been uploaded and verified. +notification.globus.download.completed=Globus transfer from the dataset {1} was successful. +notification.globus.upload.completedWithErrors=Globus transfer to Dataset {1} is complete with errors. +notification.globus.download.completedWithErrors=Globus transfer from the dataset {1} is complete with errors. notification.import.checksum={1}, dataset had file checksums added via a batch job. removeNotification=Remove Notification @@ -784,7 +792,10 @@ notification.email.apiTokenGenerated=Hello {0} {1},\n\nAPI Token has been genera notification.email.apiTokenGenerated.subject=API Token was generated notification.email.datasetWasMentioned=Hello {0},

The {1} has just been notified that the {2}, {4}, {5} "{8}" in this repository. notification.email.datasetWasMentioned.subject={0}: A Dataset Relationship has been reported! - +notification.email.globus.uploadCompleted.subject={0}: Files uploaded successfully via Globus and verified +notification.email.globus.downloadCompleted.subject={0}: Files downloaded successfully via Globus +notification.email.globus.uploadCompletedWithErrors.subject={0}: Uploaded files via Globus with errors +notification.email.globus.downloadCompletedWithErrors.subject={0}: Downloaded files via Globus with errors # dataverse.xhtml dataverse.name=Dataverse Name @@ -1541,6 +1552,9 @@ dataset.message.submit.remind.draft=When ready for sharing, please submit it dataset.message.publish.remind.draft.filePage=When ready for sharing, please go to the dataset page to publish it so that others can see these changes. dataset.message.submit.remind.draft.filePage=When ready for sharing, please go to the dataset page to submit it for review. dataset.message.publishSuccess=This dataset has been published. +dataset.message.publishGlobusFailure.details=Could not publish Globus data. +dataset.message.publishGlobusFailure=Error with publishing data. +dataset.message.GlobusError=Cannot go to Globus. dataset.message.only.authenticatedUsers=Only authenticated users may release Datasets. dataset.message.deleteSuccess=This dataset has been deleted. dataset.message.bulkFileUpdateSuccess=The selected files have been updated. @@ -1557,8 +1571,8 @@ dataset.message.deleteFailure=This dataset draft could not be deleted. dataset.message.deaccessionFailure=This dataset could not be deaccessioned. dataset.message.createFailure=The dataset could not be created. dataset.message.termsFailure=The dataset terms could not be updated. -dataset.message.label.fileAccess=File Access -dataset.message.publicInstall=Files are stored on a publicly accessible storage server. +dataset.message.label.fileAccess=Publicly-accessible storage +dataset.message.publicInstall=Files in this dataset may be readable outside Dataverse, restricted and embargoed access are disabled dataset.metadata.publicationDate=Publication Date dataset.metadata.publicationDate.tip=The publication date of a Dataset. dataset.metadata.citationDate=Citation Date @@ -1650,6 +1664,13 @@ file.fromHTTP=Upload with HTTP via your browser file.fromDropbox=Upload from Dropbox file.fromDropbox.tip=Select files from Dropbox. file.fromRsync=Upload with rsync + SSH via Data Capture Module (DCM) +file.fromGlobus.tip=Upload files via Globus transfer. This method is recommended for large file transfers. (Using it will cancel any other types of uploads in progress on this page.) +file.fromGlobusAfterCreate.tip=File upload via Globus transfer will be enabled after this dataset is created. +file.fromGlobus=Upload with Globus +file.finishGlobus=Globus Transfer has finished +file.downloadFromGlobus=Download through Globus +file.globus.transfer=Globus Transfer +file.globus.of=of: file.api.httpDisabled=File upload via HTTP is not available for this installation of Dataverse. file.api.alreadyHasPackageFile=File upload via HTTP disabled since this dataset already contains a package file. file.replace.original=Original File @@ -1721,6 +1742,8 @@ file.rsyncUpload.httpUploadDisabledDueToRsyncFileExisting=HTTP upload is disable file.rsyncUpload.httpUploadDisabledDueToRsyncFileExistingAndPublished=HTTP upload is disabled for this dataset because you have already uploaded files via rsync and published the dataset. file.rsyncUpload.rsyncUploadDisabledDueFileUploadedViaHttp=Upload with rsync + SSH is disabled for this dataset because you have already uploaded files via HTTP. If you would like to switch to rsync upload, then you must first remove all uploaded files from this dataset. Once this dataset is published, the chosen upload method is permanently locked in. file.rsyncUpload.rsyncUploadDisabledDueFileUploadedViaHttpAndPublished=Upload with rsync + SSH is disabled for this dataset because you have already uploaded files via HTTP and published the dataset. +file.globusUpload.inProgressMessage.summary=Globus Transfer in Progress +file.globusUpload.inProgressMessage.details=This dataset is locked while the data files are being transferred and verified. Large transfers may take significant time. You can check transfer status at https://app.globus.org/activity. file.metaData.checksum.copy=Click to copy file.metaData.dataFile.dataTab.unf=UNF file.metaData.dataFile.dataTab.variables=Variables diff --git a/src/main/webapp/dataset-license-terms.xhtml b/src/main/webapp/dataset-license-terms.xhtml index f51c899f479..1cbf297bf89 100644 --- a/src/main/webapp/dataset-license-terms.xhtml +++ b/src/main/webapp/dataset-license-terms.xhtml @@ -242,13 +242,13 @@ or !empty termsOfUseAndAccess.contactForAccess or !empty termsOfUseAndAccess.sizeOfCollection or !empty termsOfUseAndAccess.studyCompletion)}">
- +  
-
+
-
+
-
+
-
+
@@ -442,7 +447,7 @@ -
  • +
  • @@ -452,7 +457,7 @@
    • -
  • +
  • - + +
    • diff --git a/src/main/webapp/file-download-button-fragment.xhtml b/src/main/webapp/file-download-button-fragment.xhtml index 53dc1bc11b1..ac1ec525b44 100644 --- a/src/main/webapp/file-download-button-fragment.xhtml +++ b/src/main/webapp/file-download-button-fragment.xhtml @@ -56,6 +56,33 @@
  • #{bundle['file.accessBtn.header.download']}
    • + + + +
  • + + + + #{bundle['file.globus.of']} #{fileMetadata.dataFile.friendlyType == 'Unknown' ? bundle['file.download.filetype.unknown'] : fileMetadata.dataFile.friendlyType} + + + + + GT: #{fileMetadata.dataFile.friendlyType == 'Unknown' ? bundle['file.download.filetype.unknown'] : fileMetadata.dataFile.friendlyType} + +
    • + +
  • +
  • + + + + + + + + + + + + + + + + + diff --git a/src/main/webapp/manage-templates.xhtml b/src/main/webapp/manage-templates.xhtml index 3187c3b339f..c9841ace8e8 100644 --- a/src/main/webapp/manage-templates.xhtml +++ b/src/main/webapp/manage-templates.xhtml @@ -183,6 +183,7 @@ +
    diff --git a/src/main/webapp/resources/css/structure.css b/src/main/webapp/resources/css/structure.css index 6a7dfe38a12..c184c46cee9 100644 --- a/src/main/webapp/resources/css/structure.css +++ b/src/main/webapp/resources/css/structure.css @@ -882,6 +882,8 @@ div.panel-body.read-terms{max-height:220px; overflow-y:scroll; width:100%; backg /* UPLOAD FILES */ #dragdropMsg {padding:20px;font-size:1.3em;color:#808080;text-align:center;} .dropin-btn-status.ui-icon {background: url("https://www.dropbox.com/static/images/widgets/dbx-saver-status.png") no-repeat;} +.globus-btn.ui-icon {background: url("https://docs.globus.org/images/home/transfer.png") no-repeat;background-size:contain;display:inline-block;} + /* VERSIONS */ div[id$="versionsTable"] th.col-select-width * {display:none;} diff --git a/src/main/webapp/template.xhtml b/src/main/webapp/template.xhtml index b1296b1c667..280d5ed05b9 100644 --- a/src/main/webapp/template.xhtml +++ b/src/main/webapp/template.xhtml @@ -63,7 +63,8 @@ - + + @@ -106,7 +107,8 @@ - + + diff --git a/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java b/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java index 7d638e8da62..01fb8aad6cf 100644 --- a/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java +++ b/src/test/java/edu/harvard/iq/dataverse/util/FileUtilTest.java @@ -341,6 +341,18 @@ public void testDetermineFileTypeByName() { fail("File does not exist: " + file.toPath().toString()); } } + + @Test + public void testDetermineFileTypeFromNameLocalFile() { + //Verify that name of the local file isn't used in determining the type (as we often use *.tmp when the real name has a different extension) + try { + File file = File.createTempFile("empty", "png"); + assertEquals("text/plain", FileUtil.determineFileType(file, "something.txt")); + } catch (IOException ex) { + Logger.getLogger(FileUtilTest.class.getName()).log(Level.SEVERE, null, ex); + } + + } // isThumbnailSuppported() has been moved from DataFileService to FileUtil: /**