diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md index cf982992c03..b297dfc4ee8 100644 --- a/.github/ISSUE_TEMPLATE/bug_report.md +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -1,11 +1,13 @@ --- name: Bug report -about: Did you encounter something unexpected or incorrect in the Dataverse software? We'd like to hear about it! +about: Did you encounter something unexpected or incorrect in the Dataverse software? + We'd like to hear about it! title: '' labels: '' assignees: '' --- + ` + `` + `` + `` + `` + `` + `` + +- Restart Solr instance (usually service solr start, depending on solr/OS) + +### Optional Upgrade Step: Reindex Linked Dataverse Collections + +Datasets that are part of linked dataverse collections will now be displayed in +their linking dataverse collections. In order to fix the display of collections +that have already been linked you must re-index the linked collections. This +query will provide a list of commands to re-index the effected collections: + +``` +select 'curl http://localhost:8080/api/admin/index/dataverses/' +|| tmp.dvid from (select distinct dataverse_id as dvid +from dataverselinkingdataverse) as tmp +``` + +The result of the query will be a list of re-index commands such as: + +`curl http://localhost:8080/api/admin/index/dataverses/633` + +where '633' is the id of the linked collection. + +### Optional Upgrade Step: Run File Detection on .eln Files + +Now that .eln files are recognized, you can run the [Redetect File Type](https://guides.dataverse.org/en/5.13/api/native-api.html#redetect-file-type) API on them to switch them from "unknown" to "ELN Archive". Afterward, you can reindex these files to make them appear in search facets. diff --git a/doc/release-notes/6656-file-uploads.md b/doc/release-notes/6656-file-uploads.md deleted file mode 100644 index a2430a5d0a8..00000000000 --- a/doc/release-notes/6656-file-uploads.md +++ /dev/null @@ -1 +0,0 @@ -new JVM option: dataverse.files.uploads diff --git a/doc/release-notes/7715-signed-urls-for-external-tools.md b/doc/release-notes/7715-signed-urls-for-external-tools.md deleted file mode 100644 index c2d3859c053..00000000000 --- a/doc/release-notes/7715-signed-urls-for-external-tools.md +++ /dev/null @@ -1,3 +0,0 @@ -# Improved Security for External Tools - -This release adds support for configuring external tools to use signed URLs to access the Dataverse API. This eliminates the need for tools to have access to the user's apiToken in order to access draft or restricted datasets and datafiles. Signed URLS can be transferred via POST or via a callback when triggering a tool via GET. \ No newline at end of file diff --git a/doc/release-notes/7844-codemeta.md b/doc/release-notes/7844-codemeta.md deleted file mode 100644 index 4d98c1f840f..00000000000 --- a/doc/release-notes/7844-codemeta.md +++ /dev/null @@ -1,14 +0,0 @@ -# Experimental CodeMeta Schema Support - -With this release, we are adding "experimental" (see note below) support for research software metadata deposits. - -By adding a metadata block for [CodeMeta](https://codemeta.github.io), we take another step extending the Dataverse -scope being a research data repository towards first class support of diverse F.A.I.R. objects, currently focusing -on research software and computational workflows. - -There is more work underway to make Dataverse installations around the world "research software ready". We hope -for feedback from installations on the new metadata block to optimize and lift it from the experimental stage. - -**Note:** like the metadata block for computational workflows before, this schema is flagged as "experimental". -"Experimental" means it's brand new, opt-in, and might need future tweaking based on experience of usage in the field. -These blocks are listed here: https://guides.dataverse.org/en/latest/user/appendix.html#experimental-metadata \ No newline at end of file diff --git a/doc/release-notes/7940-stop-harvest-in-progress b/doc/release-notes/7940-stop-harvest-in-progress deleted file mode 100644 index cb27a900f15..00000000000 --- a/doc/release-notes/7940-stop-harvest-in-progress +++ /dev/null @@ -1,4 +0,0 @@ -## Mechanism added for stopping a harvest in progress - -It is now possible for an admin to stop a long-running harvesting job. See [Harvesting Clients](https://guides.dataverse.org/en/latest/admin/harvestclients.html) guide for more information. - diff --git a/doc/release-notes/8239-geospatial-indexing.md b/doc/release-notes/8239-geospatial-indexing.md deleted file mode 100644 index 165cb9031ba..00000000000 --- a/doc/release-notes/8239-geospatial-indexing.md +++ /dev/null @@ -1,5 +0,0 @@ -Support for indexing the "Geographic Bounding Box" fields ("West Longitude", "East Longitude", "North Latitude", and "South Latitude") from the Geospatial metadata block has been added. - -Geospatial search is supported but only via API using two new parameters: `geo_point` and `geo_radius`. - -A Solr schema update is required. diff --git a/doc/release-notes/8671-sorting-licenses.md b/doc/release-notes/8671-sorting-licenses.md deleted file mode 100644 index 4ceb9ec056f..00000000000 --- a/doc/release-notes/8671-sorting-licenses.md +++ /dev/null @@ -1,7 +0,0 @@ -## License sorting - -Licenses as shown in the dropdown in UI can be now sorted by the superusers. See [Configuring Licenses](https://guides.dataverse.org/en/5.10/installation/config.html#configuring-licenses) section of the Installation Guide for reference. - -## Backward Incompatibilities - -License files are now required to contain the new "sortOrder" column. When attempting to create a new license without this field, an error would be returned. See [Configuring Licenses](https://guides.dataverse.org/en/5.10/installation/config.html#configuring-licenses) section of the Installation Guide for reference. \ No newline at end of file diff --git a/doc/release-notes/8838-cstr.md b/doc/release-notes/8838-cstr.md deleted file mode 100644 index d6bcd33f412..00000000000 --- a/doc/release-notes/8838-cstr.md +++ /dev/null @@ -1,13 +0,0 @@ -### CSRT PID Types Added to Related Publication ID Type field - -A persistent identifier, [CSRT](https://www.cstr.cn/search/specification/), is added to the Related Publication field's ID Type child field. For datasets published with CSRT IDs, Dataverse will also include them in the datasets' Schema.org metadata exports. - -The CSRT - -### Required Upgrade Steps - -Update the Citation metadata block: - -- `wget https://github.com/IQSS/dataverse/releases/download/v#.##/citation.tsv` -- `curl http://localhost:8080/api/admin/datasetfield/load -X POST --data-binary @citation.tsv -H "Content-type: text/tab-separated-values"` -- Add the updated citation.properties file to the appropriate directory \ No newline at end of file diff --git a/doc/release-notes/8840-improved-download-estimate.md b/doc/release-notes/8840-improved-download-estimate.md deleted file mode 100644 index cb264b7e683..00000000000 --- a/doc/release-notes/8840-improved-download-estimate.md +++ /dev/null @@ -1 +0,0 @@ -To improve performance, Dataverse estimates download counts. This release includes an update that makes the estimate more accurate. \ No newline at end of file diff --git a/doc/release-notes/8944-metadatablocks.md b/doc/release-notes/8944-metadatablocks.md deleted file mode 100644 index 35bb7808e59..00000000000 --- a/doc/release-notes/8944-metadatablocks.md +++ /dev/null @@ -1,5 +0,0 @@ -The API endpoint `/api/metadatablocks/{block_id}` has been extended to include the following fields: - -- `controlledVocabularyValues` - All possible values for fields with a controlled vocabulary. For example, the values "Agricultural Sciences", "Arts and Humanities", etc. for the "Subject" field. -- `isControlledVocabulary`: Whether or not this field has a controlled vocabulary. -- `multiple`: Whether or not the field supports multiple values. diff --git a/doc/release-notes/9005-replaceFiles-api-call b/doc/release-notes/9005-replaceFiles-api-call deleted file mode 100644 index d1a86efb745..00000000000 --- a/doc/release-notes/9005-replaceFiles-api-call +++ /dev/null @@ -1,3 +0,0 @@ -9005 - -Direct upload and out-of-band uploads can now be used to replace multiple files with one API call (complementing the prior ability to add multiple new files) diff --git a/doc/release-notes/9096-folder-upload.md b/doc/release-notes/9096-folder-upload.md deleted file mode 100644 index 0345cd6c334..00000000000 --- a/doc/release-notes/9096-folder-upload.md +++ /dev/null @@ -1 +0,0 @@ -Dataverse can now support upload of an entire folder tree of files and retain the relative paths of files as directory path metadata for the uploaded files, if the installation is configured with S3 direct upload. diff --git a/doc/release-notes/9117-file-type-detection.md b/doc/release-notes/9117-file-type-detection.md deleted file mode 100644 index 462eaace8ed..00000000000 --- a/doc/release-notes/9117-file-type-detection.md +++ /dev/null @@ -1,5 +0,0 @@ -NetCDF and HDF5 files are now detected based on their content rather than just their file extension. - -Both "classic" NetCDF 3 files and more modern NetCDF 4 files are detected based on content. - -Detection for HDF4 files is only done through the file extension ".hdf", as before. diff --git a/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv b/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv index fd1f0f27bc5..b07ea8c4fd1 100644 --- a/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv +++ b/doc/sphinx-guides/source/_static/admin/dataverse-external-tools.tsv @@ -1,5 +1,6 @@ Tool Type Scope Description Data Explorer explore file A GUI which lists the variables in a tabular data file allowing searching, charting and cross tabulation analysis. See the README.md file at https://github.com/scholarsportal/dataverse-data-explorer-v2 for the instructions on adding Data Explorer to your Dataverse. Whole Tale explore dataset A platform for the creation of reproducible research packages that allows users to launch containerized interactive analysis environments based on popular tools such as Jupyter and RStudio. Using this integration, Dataverse users can launch Jupyter and RStudio environments to analyze published datasets. For more information, see the `Whole Tale User Guide `_. -File Previewers explore file A set of tools that display the content of files - including audio, html, `Hypothes.is `_ annotations, images, PDF, text, video, tabular data, spreadsheets, GeoJSON, and ZipFiles - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers +Binder explore dataset Binder allows you to spin up custom computing environments in the cloud (including Jupyter notebooks) with the files from your dataset. `Installation instructions `_ are in the Data Exploration Lab girder_ythub project. +File Previewers explore file A set of tools that display the content of files - including audio, html, `Hypothes.is `_ annotations, images, PDF, text, video, tabular data, spreadsheets, GeoJSON, zip, and NcML files - allowing them to be viewed without downloading the file. The previewers can be run directly from github.io, so the only required step is using the Dataverse API to register the ones you want to use. Documentation, including how to optionally brand the previewers, and an invitation to contribute through github are in the README.md file. Initial development was led by the Qualitative Data Repository and the spreasdheet previewer was added by the Social Sciences and Humanities Open Cloud (SSHOC) project. https://github.com/gdcc/dataverse-previewers Data Curation Tool configure file A GUI for curating data by adding labels, groups, weights and other details to assist with informed reuse. See the README.md file at https://github.com/scholarsportal/Dataverse-Data-Curation-Tool for the installation instructions. diff --git a/doc/sphinx-guides/source/_static/api/ddi_dataset.xml b/doc/sphinx-guides/source/_static/api/ddi_dataset.xml index 014ebb8c581..679f82a3d8a 100644 --- a/doc/sphinx-guides/source/_static/api/ddi_dataset.xml +++ b/doc/sphinx-guides/source/_static/api/ddi_dataset.xml @@ -34,7 +34,8 @@ LastProducer1, FirstProducer1 LastProducer2, FirstProducer2 1003-01-01 - ProductionPlace + ProductionPlace One + ProductionPlace Two SoftwareName1 SoftwareName2 GrantInformationGrantNumber1 diff --git a/doc/sphinx-guides/source/_static/installation/files/root/external-tools/auxFileTool.json b/doc/sphinx-guides/source/_static/installation/files/root/external-tools/auxFileTool.json new file mode 100644 index 00000000000..b188520dabb --- /dev/null +++ b/doc/sphinx-guides/source/_static/installation/files/root/external-tools/auxFileTool.json @@ -0,0 +1,26 @@ +{ + "displayName": "AuxFileViewer", + "description": "Show an auxiliary file from a dataset file.", + "toolName": "auxPreviewer", + "scope": "file", + "types": [ + "preview" + ], + "toolUrl": "https://example.com/AuxFileViewer.html", + "toolParameters": { + "queryParameters": [ + { + "fileid": "{fileId}" + } + ] + }, + "requirements": { + "auxFilesExist": [ + { + "formatTag": "myFormatTag", + "formatVersion": "0.1" + } + ] + }, + "contentType": "application/foobar" +} diff --git a/doc/sphinx-guides/source/admin/harvestclients.rst b/doc/sphinx-guides/source/admin/harvestclients.rst index d1067a690e9..6d9be11a9b5 100644 --- a/doc/sphinx-guides/source/admin/harvestclients.rst +++ b/doc/sphinx-guides/source/admin/harvestclients.rst @@ -21,6 +21,8 @@ Clients are managed on the "Harvesting Clients" page accessible via the :doc:`da The process of creating a new, or editing an existing client, is largely self-explanatory. It is split into logical steps, in a way that allows the user to go back and correct the entries made earlier. The process is interactive and guidance text is provided. For example, the user is required to enter the URL of the remote OAI server. When they click *Next*, the application will try to establish a connection to the server in order to verify that it is working, and to obtain the information about the sets of metadata records and the metadata formats it supports. The choices offered to the user on the next page will be based on this extra information. If the application fails to establish a connection to the remote archive at the address specified, or if an invalid response is received, the user is given an opportunity to check and correct the URL they entered. +Note that as of 5.13, a new entry "Custom HTTP Header" has been added to the Step 1. of Create or Edit form. This optional field can be used to configure this client with a specific HTTP header to be added to every OAI request. This is to accommodate a (rare) use case where the remote server may require a special token of some kind in order to offer some content not available to other clients. Most OAI servers offer the same publicly-available content to all clients, so few admins will have a use for this feature. It is however on the very first, Step 1. screen in case the OAI server requires this token even for the "ListSets" and "ListMetadataFormats" requests, which need to be sent in the Step 2. of creating or editing a client. Multiple headers can be supplied separated by `\\n` - actual "backslash" and "n" characters, not a single "new line" character. + How to Stop a Harvesting Run in Progress ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/sphinx-guides/source/admin/integrations.rst b/doc/sphinx-guides/source/admin/integrations.rst index b29e51b581d..1888fd89761 100644 --- a/doc/sphinx-guides/source/admin/integrations.rst +++ b/doc/sphinx-guides/source/admin/integrations.rst @@ -116,6 +116,8 @@ Binder Researchers can launch Jupyter Notebooks, RStudio, and other computational environments by entering the DOI of a dataset in a Dataverse installation on https://mybinder.org +A Binder button can also be added to every dataset page to launch Binder from there. See :doc:`external-tools`. + Institutions can self host BinderHub. The Dataverse Project is one of the supported `repository providers `_. Renku diff --git a/doc/sphinx-guides/source/admin/metadataexport.rst b/doc/sphinx-guides/source/admin/metadataexport.rst index 78b8c8ce223..200c3a3e342 100644 --- a/doc/sphinx-guides/source/admin/metadataexport.rst +++ b/doc/sphinx-guides/source/admin/metadataexport.rst @@ -57,3 +57,13 @@ Downloading Metadata via API ---------------------------- The :doc:`/api/native-api` section of the API Guide explains how end users can download the metadata formats above via API. + +Exporter Configuration +---------------------- + +Two exporters - Schema.org JSONLD and OpenAire - use an algorithm to determine whether an author, or contact, name belongs to a person or organization. While the algorithm works well, there are cases in which it makes mistakes, usually inferring that an organization is a person. + +The Dataverse software implements two jvm-options that can be used to tune the algorithm: + +- :ref:`dataverse.personOrOrg.assumeCommaInPersonName` - boolean, default false. If true, Dataverse will assume any name without a comma must be an organization. This may be most useful for curated Dataverse instances that enforce the "family name, given name" convention. +- :ref:`dataverse.personOrOrg.orgPhraseArray` - a JsonArray of strings. Any name that contains one of the strings is assumed to be an organization. For example, "Project" is a word that is not otherwise associated with being an organization. diff --git a/doc/sphinx-guides/source/api/external-tools.rst b/doc/sphinx-guides/source/api/external-tools.rst index 4f6c9a8015c..eec9944338f 100644 --- a/doc/sphinx-guides/source/api/external-tools.rst +++ b/doc/sphinx-guides/source/api/external-tools.rst @@ -53,15 +53,21 @@ External tools must be expressed in an external tool manifest file, a specific J Examples of Manifests +++++++++++++++++++++ -Let's look at two examples of external tool manifests (one at the file level and one at the dataset level) before we dive into how they work. +Let's look at a few examples of external tool manifests (both at the file level and at the dataset level) before we dive into how they work. + +.. _tools-for-files: External Tools for Files ^^^^^^^^^^^^^^^^^^^^^^^^ -:download:`fabulousFileTool.json <../_static/installation/files/root/external-tools/fabulousFileTool.json>` is a file level both an "explore" tool and a "preview" tool that operates on tabular files: +:download:`fabulousFileTool.json <../_static/installation/files/root/external-tools/fabulousFileTool.json>` is a file level (both an "explore" tool and a "preview" tool) that operates on tabular files: .. literalinclude:: ../_static/installation/files/root/external-tools/fabulousFileTool.json +:download:`auxFileTool.json <../_static/installation/files/root/external-tools/auxFileTool.json>` is a file level preview tool that operates on auxiliary files associated with a data file (note the "requirements" section): + +.. literalinclude:: ../_static/installation/files/root/external-tools/auxFileTool.json + External Tools for Datasets ^^^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -113,6 +119,10 @@ Terminology allowedApiCalls httpMethod Which HTTP method the specified callback uses such as ``GET`` or ``POST``. allowedApiCalls timeOut For non-public datasets and datafiles, how many minutes the signed URLs given to the tool should be valid for. Must be an integer. + + requirements **Resources your tool needs to function.** For now, the only requirement you can specify is that one or more auxiliary files exist (see auxFilesExist in the :ref:`tools-for-files` example). Currently, requirements only apply to preview tools. If the requirements are not met, the preview tool is not shown. + + auxFilesExist **An array containing formatTag and formatVersion pairs** for each auxiliary file that your tool needs to download to function properly. For example, a required aux file could have a ``formatTag`` of "NcML" and a ``formatVersion`` of "1.0". See also :doc:`/developers/aux-file-support`. toolName A **name** of an external tool that is used to differentiate between external tools and also used in bundle.properties for localization in the Dataverse installation web interface. For example, the toolName for Data Explorer is ``explorer``. For the Data Curation Tool the toolName is ``dct``. This is an optional parameter in the manifest JSON file. =========================== ========== diff --git a/doc/sphinx-guides/source/api/metrics.rst b/doc/sphinx-guides/source/api/metrics.rst index 6a878d73a98..f1eb1f88c71 100755 --- a/doc/sphinx-guides/source/api/metrics.rst +++ b/doc/sphinx-guides/source/api/metrics.rst @@ -72,7 +72,7 @@ Return Formats There are a number of API calls that provide time series, information reported per item (e.g. per dataset, per file, by subject, by category, and by file Mimetype), or both (time series per item). Because these calls all report more than a single number, the API provides two optional formats for the return that can be selected by specifying an HTTP Accept Header for the desired format: -* application/json - a JSON array of objects. For time-series, the objects include key/values for the ``date`` and ``count`` for that month. For per-item calls, the objects include the item (e.g. for a subject), or it's id/pid (for a dataset or datafile). For timeseries per-item, the objects also include a date. In all cases, the response is a single array. +* application/json - a JSON array of objects. For time-series, the objects include key/values for the ``date`` and ``count`` for that month. For per-item calls, the objects include the item (e.g. for a subject), or it's id/pid (for a dataset or datafile (which may/may not not have a PID)). For timeseries per-item, the objects also include a date. In all cases, the response is a single array. * Example: ``curl -H 'Accept:application/json' https://demo.dataverse.org/api/info/metrics/downloads/monthly`` @@ -120,7 +120,7 @@ Example: ``curl https://demo.dataverse.org/api/info/metrics/makeDataCount/viewsT Endpoint Table -------------- -The following table lists the available metrics endpoints (not including the Make Data Counts endpoints a single dataset which are part of the :doc:`/api/native-api`) along with additional notes about them. +The following table lists the available metrics endpoints (not including the Make Data Counts endpoints for a single dataset which are part of the :doc:`/api/native-api`) along with additional notes about them. .. csv-table:: Metrics Endpoints diff --git a/doc/sphinx-guides/source/api/native-api.rst b/doc/sphinx-guides/source/api/native-api.rst index 589b947f15e..3cd469e3883 100644 --- a/doc/sphinx-guides/source/api/native-api.rst +++ b/doc/sphinx-guides/source/api/native-api.rst @@ -552,6 +552,8 @@ You should expect an HTTP 200 ("OK") response and JSON indicating the database I .. note:: Only a Dataverse installation account with superuser permissions is allowed to include files when creating a dataset via this API. Adding files this way only adds their file metadata to the database, you will need to manually add the physical files to the file system. +.. _api-import-dataset: + Import a Dataset into a Dataverse Collection ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -728,13 +730,12 @@ The fully expanded example above (without environment variables) looks like this curl -H "X-Dataverse-key:$API_TOKEN" https://demo.dataverse.org/api/datasets/:persistentId/versions/:draft?persistentId=doi:10.5072/FK2/J8SJZB - -|CORS| Show the dataset whose id is passed: +|CORS| Show the dataset whose database id is passed: .. code-block:: bash export SERVER_URL=https://demo.dataverse.org - export ID=408730 + export ID=24 curl $SERVER_URL/api/datasets/$ID @@ -742,7 +743,7 @@ The fully expanded example above (without environment variables) looks like this .. code-block:: bash - curl https://demo.dataverse.org/api/datasets/408730 + curl https://demo.dataverse.org/api/datasets/24 The dataset id can be extracted from the response retrieved from the API which uses the persistent identifier (``/api/datasets/:persistentId/?persistentId=$PERSISTENT_IDENTIFIER``). @@ -1511,6 +1512,38 @@ The fully expanded example above (without environment variables) looks like this curl -H X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X POST https://demo.dataverse.org/api/datasets/:persistentId/add?persistentId=doi:10.5072/FK2/J8SJZB -F 'jsonData={"description":"A remote image.","storageIdentifier":"trsa://themes/custom/qdr/images/CoreTrustSeal-logo-transparent.png","checksumType":"MD5","md5Hash":"509ef88afa907eaf2c17c1c8d8fde77e","label":"testlogo.png","fileName":"testlogo.png","mimeType":"image/png"}' +.. _cleanup-storage-api: + +Cleanup storage of a Dataset +~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This is an experimental feature and should be tested on your system before using it in production. +Also, make sure that your backups are up-to-date before using this on production servers. +It is advised to first call this method with the ``dryrun`` parameter set to ``true`` before actually deleting the files. +This will allow you to manually inspect the files that would be deleted if that parameter is set to ``false`` or is omitted (a list of the files that would be deleted is provided in the response). + +If your Dataverse installation has been configured to support direct uploads, or in some other situations, +you could end up with some files in the storage of a dataset that are not linked to that dataset directly. Most commonly, this could +happen when an upload fails in the middle of a transfer, i.e. if a user does a UI direct upload and leaves the page without hitting cancel or save, +Dataverse doesn't know and doesn't clean up the files. Similarly in the direct upload API, if the final /addFiles call isn't done, the files are abandoned. + +All the files stored in the Dataset storage location that are not in the file list of that Dataset (and follow the naming pattern of the dataset files) can be removed, as shown in the example below. + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export PERSISTENT_ID=doi:10.5072/FK2/J8SJZB + export DRYRUN=true + + curl -H "X-Dataverse-key: $API_TOKEN" -X GET "$SERVER_URL/api/datasets/:persistentId/cleanStorage?persistentId=$PERSISTENT_ID&dryrun=$DRYRUN" + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H X-Dataverse-key: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx -X GET https://demo.dataverse.org/api/datasets/:persistentId/cleanStorage?persistentId=doi:10.5072/FK2/J8SJZB&dryrun=true + Adding Files To a Dataset via Other Tools ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -2058,6 +2091,77 @@ The response is a JSON object described in the :doc:`/api/external-tools` sectio Files ----- +Get JSON Representation of a File +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: Files can be accessed using persistent identifiers. This is done by passing the constant ``:persistentId`` where the numeric id of the file is expected, and then passing the actual persistent id as a query parameter with the name ``persistentId``. + +Example: Getting the file whose DOI is *10.5072/FK2/J8SJZB*: + +.. code-block:: bash + + export SERVER_URL=https://demo.dataverse.org + export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/J8SJZB + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + + curl -H "X-Dataverse-key:$API_TOKEN" $SERVER_URL/api/files/:persistentId/?persistentId=$PERSISTENT_IDENTIFIER + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" https://demo.dataverse.org/api/files/:persistentId/?persistentId=doi:10.5072/FK2/J8SJZB + +You may get its draft version of an unpublished file if you pass an api token with view draft permissions: + +.. code-block:: bash + + export SERVER_URL=https://demo.dataverse.org + export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/J8SJZB + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + + curl -H "X-Dataverse-key:$API_TOKEN" $SERVER/api/files/:persistentId/?persistentId=$PERSISTENT_IDENTIFIER + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" https://demo.dataverse.org/api/files/:persistentId/?persistentId=doi:10.5072/FK2/J8SJZB + + +|CORS| Show the file whose id is passed: + +.. code-block:: bash + + export SERVER_URL=https://demo.dataverse.org + export ID=408730 + + curl $SERVER_URL/api/file/$ID + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl https://demo.dataverse.org/api/files/408730 + +You may get its draft version of an published file if you pass an api token with view draft permissions and use the draft path parameter: + +.. code-block:: bash + + export SERVER_URL=https://demo.dataverse.org + export PERSISTENT_IDENTIFIER=doi:10.5072/FK2/J8SJZB + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + + curl -H "X-Dataverse-key:$API_TOKEN" $SERVER/api/files/:persistentId/draft/?persistentId=$PERSISTENT_IDENTIFIER + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" https://demo.dataverse.org/api/files/:persistentId/draft/?persistentId=doi:10.5072/FK2/J8SJZB + +The file id can be extracted from the response retrieved from the API which uses the persistent identifier (``/api/datasets/:persistentId/?persistentId=$PERSISTENT_IDENTIFIER``). + Adding Files ~~~~~~~~~~~~ @@ -2255,6 +2359,47 @@ Currently the following methods are used to detect file types: - The file extension (e.g. ".ipybn") is used, defined in a file called ``MimeTypeDetectionByFileExtension.properties``. - The file name (e.g. "Dockerfile") is used, defined in a file called ``MimeTypeDetectionByFileName.properties``. +.. _extractNcml: + +Extract NcML +~~~~~~~~~~~~ + +As explained in the :ref:`netcdf-and-hdf5` section of the User Guide, when those file types are uploaded, an attempt is made to extract an NcML file from them and store it as an auxiliary file. + +This happens automatically but superusers can also manually trigger this NcML extraction process with the API endpoint below. + +Note that "true" will be returned if an NcML file was created. "false" will be returned if there was an error or if the NcML file already exists (check server.log for details). + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export ID=24 + + curl -H "X-Dataverse-key:$API_TOKEN" -X POST "$SERVER_URL/api/files/$ID/extractNcml" + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X POST "https://demo.dataverse.org/api/files/24/extractNcml + +A curl example using a PID: + +.. code-block:: bash + + export API_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx + export SERVER_URL=https://demo.dataverse.org + export PERSISTENT_ID=doi:10.5072/FK2/AAA000 + + curl -H "X-Dataverse-key:$API_TOKEN" -X POST "$SERVER_URL/api/files/:persistentId/extractNcml?persistentId=$PERSISTENT_ID" + +The fully expanded example above (without environment variables) looks like this: + +.. code-block:: bash + + curl -H "X-Dataverse-key:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" -X POST "https://demo.dataverse.org/api/files/:persistentId/extractNcml?persistentId=doi:10.5072/FK2/AAA000" + Replacing Files ~~~~~~~~~~~~~~~ @@ -3296,7 +3441,8 @@ The following optional fields are supported: - archiveDescription: What the name suggests. If not supplied, will default to "This Dataset is harvested from our partners. Clicking the link will take you directly to the archival source of the data." - set: The OAI set on the remote server. If not supplied, will default to none, i.e., "harvest everything". - style: Defaults to "default" - a generic OAI archive. (Make sure to use "dataverse" when configuring harvesting from another Dataverse installation). - +- customHeaders: This can be used to configure this client with a specific HTTP header that will be added to every OAI request. This is to accommodate a use case where the remote server requires this header to supply some form of a token in order to offer some content not available to other clients. See the example below. Multiple headers can be supplied separated by `\\n` - actual "backslash" and "n" characters, not a single "new line" character. + Generally, the API will accept the output of the GET version of the API for an existing client as valid input, but some fields will be ignored. For example, as of writing this there is no way to configure a harvesting schedule via this API. An example JSON file would look like this:: @@ -3308,6 +3454,7 @@ An example JSON file would look like this:: "archiveUrl": "https://zenodo.org", "archiveDescription": "Moissonné depuis la collection LMOPS de l'entrepôt Zenodo. En cliquant sur ce jeu de données, vous serez redirigé vers Zenodo.", "metadataFormat": "oai_dc", + "customHeaders": "x-oai-api-key: xxxyyyzzz", "set": "user-lmops" } diff --git a/doc/sphinx-guides/source/conf.py b/doc/sphinx-guides/source/conf.py index 590eee4bd9d..736d86cacf5 100755 --- a/doc/sphinx-guides/source/conf.py +++ b/doc/sphinx-guides/source/conf.py @@ -66,9 +66,9 @@ # built documents. # # The short X.Y version. -version = '5.12.1' +version = '5.13' # The full version, including alpha/beta/rc tags. -release = '5.12.1' +release = '5.13' # The language for content autogenerated by Sphinx. Refer to documentation # for a list of supported languages. diff --git a/doc/sphinx-guides/source/developers/documentation.rst b/doc/sphinx-guides/source/developers/documentation.rst index b20fd112533..c89ed6e3b75 100755 --- a/doc/sphinx-guides/source/developers/documentation.rst +++ b/doc/sphinx-guides/source/developers/documentation.rst @@ -22,6 +22,8 @@ That's it! Thank you for your contribution! Your pull request will be added manu Please see https://github.com/IQSS/dataverse/pull/5857 for an example of a quick fix that was merged (the "Files changed" tab shows how a typo was fixed). +Preview your documentation changes which will be built automatically as part of your pull request in Github. It will show up as a check entitled: `docs/readthedocs.org:dataverse-guide — Read the Docs build succeeded!`. For example, this PR built to https://dataverse-guide--9249.org.readthedocs.build/en/9249/. + If you would like to read more about the Dataverse Project's use of GitHub, please see the :doc:`version-control` section. For bug fixes and features we request that you create an issue before making a pull request but this is not at all necessary for quick fixes to the documentation. .. _admin: https://github.com/IQSS/dataverse/tree/develop/doc/sphinx-guides/source/admin diff --git a/doc/sphinx-guides/source/index.rst b/doc/sphinx-guides/source/index.rst index cdc15ac50e0..f6eda53d718 100755 --- a/doc/sphinx-guides/source/index.rst +++ b/doc/sphinx-guides/source/index.rst @@ -6,7 +6,7 @@ Dataverse Documentation v. |version| ==================================== -These documentation guides are for the |version| version of Dataverse. To find guides belonging to previous versions, :ref:`guides_versions` has a list of all available versions. +These documentation guides are for the |version| version of Dataverse. To find guides belonging to previous or future versions, :ref:`guides_versions` has a list of all available versions. .. toctree:: :glob: diff --git a/doc/sphinx-guides/source/installation/config.rst b/doc/sphinx-guides/source/installation/config.rst index 6b71cc8a21d..8a42ca92b61 100644 --- a/doc/sphinx-guides/source/installation/config.rst +++ b/doc/sphinx-guides/source/installation/config.rst @@ -263,6 +263,153 @@ As for the "Remote only" authentication mode, it means that: - ``:DefaultAuthProvider`` has been set to use the desired authentication provider - The "builtin" authentication provider has been disabled (:ref:`api-toggle-auth-provider`). Note that disabling the "builtin" authentication provider means that the API endpoint for converting an account from a remote auth provider will not work. Converting directly from one remote authentication provider to another (i.e. from GitHub to Google) is not supported. Conversion from remote is always to "builtin". Then the user initiates a conversion from "builtin" to remote. Note that longer term, the plan is to permit multiple login options to the same Dataverse installation account per https://github.com/IQSS/dataverse/issues/3487 (so all this talk of conversion will be moot) but for now users can only use a single login option, as explained in the :doc:`/user/account` section of the User Guide. In short, "remote only" might work for you if you only plan to use a single remote authentication provider such that no conversion between remote authentication providers will be necessary. +.. _database-persistence: + +Database Persistence +-------------------- + +The Dataverse software uses a PostgreSQL database to store objects users create. +You can configure basic and advanced settings for the PostgreSQL database connection with the help of +MicroProfile Config API. + +Basic Database Settings ++++++++++++++++++++++++ + +1. Any of these settings can be set via system properties (see :ref:`jvm-options` starting at :ref:`dataverse.db.name`), environment variables or other + MicroProfile Config mechanisms supported by the app server. + `See Payara docs for supported sources `_. +2. Remember to protect your secrets. For passwords, use an environment variable (bare minimum), a password alias named the same + as the key (OK) or use the "dir config source" of Payara (best). + + Alias creation example: + + .. code-block:: shell + + echo "AS_ADMIN_ALIASPASSWORD=changeme" > /tmp/p.txt + asadmin create-password-alias --passwordfile /tmp/p.txt dataverse.db.password + rm /tmp/p.txt + +3. Environment variables follow the key, replacing any dot, colon, dash, etc. into an underscore "_" and all uppercase + letters. Example: ``dataverse.db.host`` -> ``DATAVERSE_DB_HOST`` + +.. list-table:: + :widths: 15 60 25 + :header-rows: 1 + :align: left + + * - MPCONFIG Key + - Description + - Default + * - dataverse.db.host + - The PostgreSQL server to connect to. + - ``localhost`` + * - dataverse.db.port + - The PostgreSQL server port to connect to. + - ``5432`` + * - dataverse.db.user + - The PostgreSQL user name to connect with. + - | ``dataverse`` + | (installer sets to ``dvnapp``) + * - dataverse.db.password + - The PostgreSQL users password to connect with. + + **Please note the safety advisory above.** + - *No default* + * - dataverse.db.name + - The PostgreSQL database name to use for the Dataverse installation. + - | ``dataverse`` + | (installer sets to ``dvndb``) + * - dataverse.db.parameters + - Connection parameters, such as ``sslmode=require``. See `Postgres JDBC docs `_ + Note: you don't need to provide the initial "?". + - *Empty string* + +Advanced Database Settings +++++++++++++++++++++++++++ + +The following options are useful in many scenarios. You might be interested in debug output during development or +monitoring performance in production. + +You can find more details within the Payara docs: + +- `User Guide: Connection Pool Configuration `_ +- `Tech Doc: Advanced Connection Pool Configuration `_. + +Connection Validation +^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 15 60 25 + :header-rows: 1 + :align: left + + * - MPCONFIG Key + - Description + - Default + * - dataverse.db.is-connection-validation-required + - ``true``: Validate connections, allow server to reconnect in case of failure. + - false + * - dataverse.db.connection-validation-method + - | The method of connection validation: + | ``table|autocommit|meta-data|custom-validation``. + - *Empty string* + * - dataverse.db.validation-table-name + - The name of the table used for validation if the validation method is set to ``table``. + - *Empty string* + * - dataverse.db.validation-classname + - The name of the custom class used for validation if the ``validation-method`` is set to ``custom-validation``. + - *Empty string* + * - dataverse.db.validate-atmost-once-period-in-seconds + - Specifies the time interval in seconds between successive requests to validate a connection at most once. + - ``0`` (disabled) + +Connection & Statement Leaks +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 15 60 25 + :header-rows: 1 + :align: left + + * - MPCONFIG Key + - Description + - Default + * - dataverse.db.connection-leak-timeout-in-seconds + - Specify timeout when connections count as "leaked". + - ``0`` (disabled) + * - dataverse.db.connection-leak-reclaim + - If enabled, leaked connection will be reclaimed by the pool after connection leak timeout occurs. + - ``false`` + * - dataverse.db.statement-leak-timeout-in-seconds + - Specifiy timeout when statements should be considered to be "leaked". + - ``0`` (disabled) + * - dataverse.db.statement-leak-reclaim + - If enabled, leaked statement will be reclaimed by the pool after statement leak timeout occurs. + - ``false`` + +Logging & Slow Performance +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +.. list-table:: + :widths: 15 60 25 + :header-rows: 1 + :align: left + + * - MPCONFIG Key + - Description + - Default + * - dataverse.db.statement-timeout-in-seconds + - Timeout property of a connection to enable termination of abnormally long running queries. + - ``-1`` (disabled) + * - dataverse.db.slow-query-threshold-in-seconds + - SQL queries that exceed this time in seconds will be logged. + - ``-1`` (disabled) + * - dataverse.db.log-jdbc-calls + - When set to true, all JDBC calls will be logged allowing tracing of all JDBC interactions including SQL. + - ``false`` + + + .. _file-storage: File Storage: Using a Local Filesystem and/or Swift and/or Object Stores and/or Trusted Remote Stores @@ -288,7 +435,9 @@ To support multiple stores, a Dataverse installation now requires an id, type, a Out of the box, a Dataverse installation is configured to use local file storage in the 'file' store by default. You can add additional stores and, as a superuser, configure specific Dataverse collections to use them (by editing the 'General Information' for the Dataverse collection as described in the :doc:`/admin/dataverses-datasets` section). -Note that the "\-Ddataverse.files.directory", if defined, continues to control where temporary files are stored (in the /temp subdir of that directory), independent of the location of any 'file' store defined above. +Note that the "\-Ddataverse.files.directory", if defined, continues to control where temporary files are stored +(in the /temp subdir of that directory), independent of the location of any 'file' store defined above. +(See also the option reference: :ref:`dataverse.files.directory`) If you wish to change which store is used by default, you'll need to delete the existing default storage driver and set a new one using jvm options. @@ -299,6 +448,8 @@ If you wish to change which store is used by default, you'll need to delete the It is also possible to set maximum file upload size limits per store. See the :ref:`:MaxFileUploadSizeInBytes` setting below. +.. _storage-files-dir: + File Storage ++++++++++++ @@ -1517,13 +1668,26 @@ protocol, host, and port number and should not include a trailing slash. - We are absolutely aware that it's confusing to have both ``dataverse.fqdn`` and ``dataverse.siteUrl``. https://github.com/IQSS/dataverse/issues/6636 is about resolving this confusion. - .. _dataverse.files.directory: dataverse.files.directory +++++++++++++++++++++++++ -This is how you configure the path Dataverse uses for temporary files. (File store specific dataverse.files.\.directory options set the permanent data storage locations.) +Please provide an absolute path to a directory backed by some mounted file system. This directory is used for a number +of purposes: + +1. ``/temp`` after uploading, data is temporarily stored here for ingest and/or before + shipping to the final storage destination. +2. ``/sword`` a place to store uploads via the :doc:`../api/sword` before transfer + to final storage location and/or ingest. +3. ``/googlecloudkey.json`` used with :ref:`Google Cloud Configuration` for BagIt exports. + This location is deprecated and might be refactored into a distinct setting in the future. +4. The experimental DCM feature for :doc:`../developers/big-data-support` is able to trigger imports for externally + uploaded files in a directory tree at ``//`` + under certain conditions. This directory may also be used by file stores for :ref:`permanent file storage `, + but this is controlled by other, store-specific settings. + +Defaults to ``/tmp/dataverse``. Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_FILES_DIRECTORY``. .. _dataverse.files.uploads: @@ -1544,6 +1708,8 @@ dataverse.auth.password-reset-timeout-in-minutes Users have 60 minutes to change their passwords by default. You can adjust this value here. +.. _dataverse.db.name: + dataverse.db.name +++++++++++++++++ @@ -1553,6 +1719,8 @@ Defaults to ``dataverse`` (but the installer sets it to ``dvndb``). Can also be set via *MicroProfile Config API* sources, e.g. the environment variable ``DATAVERSE_DB_NAME``. +See also :ref:`database-persistence`. + dataverse.db.user +++++++++++++++++ @@ -1848,8 +2016,6 @@ By default, download URLs to files will be included in Schema.org JSON-LD output ``./asadmin create-jvm-options '-Ddataverse.files.hide-schema-dot-org-download-urls=true'`` -Please note that there are other reasons why download URLs may not be included for certain files such as if a guestbook entry is required or if the file is restricted. - For more on Schema.org JSON-LD, see the :doc:`/admin/metadataexport` section of the Admin Guide. .. _useripaddresssourceheader: @@ -1879,6 +2045,27 @@ This setting is useful in cases such as running your Dataverse installation behi "HTTP_FORWARDED", "HTTP_VIA", "REMOTE_ADDR" + +.. _dataverse.personOrOrg.assumeCommaInPersonName: + +dataverse.personOrOrg.assumeCommaInPersonName ++++++++++++++++++++++++++++++++++++++++++++++ + +Please note that this setting is experimental. + +The Schema.org metadata and OpenAIRE exports and the Schema.org metadata included in DatasetPages try to infer whether each entry in the various fields (e.g. Author, Contributor) is a Person or Organization. If you are sure that +users are following the guidance to add people in the recommended family name, given name order, with a comma, you can set this true to always assume entries without a comma are for Organizations. The default is false. + +.. _dataverse.personOrOrg.orgPhraseArray: + +dataverse.personOrOrg.orgPhraseArray +++++++++++++++++++++++++++++++++++++ + +Please note that this setting is experimental. + +The Schema.org metadata and OpenAIRE exports and the Schema.org metadata included in DatasetPages try to infer whether each entry in the various fields (e.g. Author, Contributor) is a Person or Organization. +If you have examples where an orgization name is being inferred to belong to a person, you can use this setting to force it to be recognized as an organization. +The value is expected to be a JsonArray of strings. Any name that contains one of the strings is assumed to be an organization. For example, "Project" is a word that is not otherwise associated with being an organization. .. _dataverse.api.signature-secret: @@ -3235,7 +3422,7 @@ For example: ``curl -X PUT -d "This content needs to go through an additional review by the Curation Team before it can be published." http://localhost:8080/api/admin/settings/:DatasetMetadataValidationFailureMsg`` - + :ExternalValidationAdminOverride ++++++++++++++++++++++++++++++++ diff --git a/doc/sphinx-guides/source/installation/prep.rst b/doc/sphinx-guides/source/installation/prep.rst index c491659cd56..abb4349d3ad 100644 --- a/doc/sphinx-guides/source/installation/prep.rst +++ b/doc/sphinx-guides/source/installation/prep.rst @@ -79,15 +79,24 @@ System Requirements Hardware Requirements +++++++++++++++++++++ -A basic Dataverse installation runs fine on modest hardware. For example, as of this writing the test installation at http://phoenix.dataverse.org is backed by a single virtual machine with two 2.8 GHz processors, 8 GB of RAM and 50 GB of disk. +A basic Dataverse installation runs fine on modest hardware. For example, in the recent past we had a test instance backed by a single virtual machine with two 2.8 GHz processors, 8 GB of RAM and 50 GB of disk. In contrast, before we moved it to the Amazon Cloud, the production installation at https://dataverse.harvard.edu was backed by six servers with two Intel Xeon 2.53 Ghz CPUs and either 48 or 64 GB of RAM. The three servers with 48 GB of RAM run were web frontends running Glassfish 4 and Apache and were load balanced by a hardware device. The remaining three servers with 64 GB of RAM were the primary and backup database servers and a server dedicated to running Rserve. Multiple TB of storage were mounted from a SAN via NFS. -Currently, the Harvard Dataverse Repository is served by four AWS server nodes: two "m4.4xlarge" instances (64GB/16 vCPU) as web frontends, one 32GB/8 vCPU ("m4.2xlarge") instance for the Solr search engine, and one 16GB/4 vCPU ("m4.xlarge") instance for R. The PostgreSQL database is served by Amazon RDS, and physical files are stored on Amazon S3. +Currently, the Harvard Dataverse Repository is served by four AWS server nodes -The Dataverse Software installation script will attempt to give your app server the right amount of RAM based on your system. +- two instances for web frontends running Payara fronted by Apache ("m4.4xlarge" with 64 GB RAM and 16 vCPUs) -Experimentation and testing with various hardware configurations is encouraged, or course, but do reach out as explained in the :doc:`intro` as needed for assistance. + - these are sitting behind an AWS ELB load balancer + +- one instance for the Solr search engine ("m4.2xlarge" with 32 GB RAM and 8 vCPUs) +- one instance for R ("m4.xlarge" instances with 16 GB RAM and 4 vCPUs) + +The PostgreSQL database is served by Amazon RDS. + +Physical files are stored on Amazon S3. The primary bucket is replicated in real-time to a secondary bucket, which is backed up to Glacier. Deleted files are kept around on the secondary bucket for a little while for convenient recovery. In addition, we use a backup script mentioned under :doc:`/admin/backups`. + +Experimentation and testing with various hardware configurations is encouraged, or course. Note that the installation script will attempt to give your app server (the web frontend) the right amount of RAM based on your system. Software Requirements +++++++++++++++++++++ diff --git a/doc/sphinx-guides/source/user/appendix.rst b/doc/sphinx-guides/source/user/appendix.rst index d6009edc9c9..7d60054ae17 100755 --- a/doc/sphinx-guides/source/user/appendix.rst +++ b/doc/sphinx-guides/source/user/appendix.rst @@ -39,6 +39,9 @@ Unlike supported metadata, experimental metadata is not enabled by default in a - `CodeMeta Software Metadata `__: based on the `CodeMeta Software Metadata Schema, version 2.0 `__ (`see .tsv version `__) - `Computational Workflow Metadata `__ (`see .tsv version `__): adapted from `Bioschemas Computational Workflow Profile, version 1.0 `__ and `Codemeta `__. +Please note: these custom metadata schemas are not included in the Solr schema for indexing by default, you will need +to add them as necessary for your custom metadata blocks. See "Update the Solr Schema" in :doc:`../admin/metadatacustomization`. + See Also ~~~~~~~~ diff --git a/doc/sphinx-guides/source/user/dataset-management.rst b/doc/sphinx-guides/source/user/dataset-management.rst index 8043e7ffbb7..31dd7f9cf78 100755 --- a/doc/sphinx-guides/source/user/dataset-management.rst +++ b/doc/sphinx-guides/source/user/dataset-management.rst @@ -184,11 +184,32 @@ File Handling Certain file types in the Dataverse installation are supported by additional functionality, which can include downloading in different formats, previews, file-level metadata preservation, file-level data citation; and exploration through data visualization and analysis. See the sections below for information about special functionality for specific file types. +.. _file-previews: + File Previews ------------- Dataverse installations can add previewers for common file types uploaded by their research communities. The previews appear on the file page. If a preview tool for a specific file type is available, the preview will be created and will display automatically, after terms have been agreed to or a guestbook entry has been made, if necessary. File previews are not available for restricted files unless they are being accessed using a Private URL. See also :ref:`privateurl`. +Previewers are available for the following file types: + +- Text +- PDF +- Tabular (CSV, Excel, etc., see :doc:`tabulardataingest/index`) +- Code (R, etc.) +- Images (PNG, GIF, JPG) +- Audio (MP3, MPEG, WAV, OGG, M4A) +- Video (MP4, OGG, Quicktime) +- Zip (preview and extract/download) +- HTML +- GeoJSON +- NetCDF/HDF5 (NcML format) +- Hypothes.is + +Additional file types will be added to the `dataverse-previewers `_ repo before they are listed above so please check there for the latest information or to request (or contribute!) an additional file previewer. + +Installation of previewers is explained in the :doc:`/admin/external-tools` section of in the Admin Guide. + Tabular Data Files ------------------ @@ -306,6 +327,22 @@ Astronomy (FITS) Metadata found in the header section of `Flexible Image Transport System (FITS) files `_ are automatically extracted by the Dataverse Software, aggregated and displayed in the Astronomy Domain-Specific Metadata of the Dataset that the file belongs to. This FITS file metadata, is therefore searchable and browsable (facets) at the Dataset-level. +.. _geojson: + +GeoJSON +------- + +A map will be shown as a preview of GeoJSON files when the previewer has been enabled (see :ref:`file-previews`). See also a `video demo `_ of the GeoJSON previewer by its author, Kaitlin Newson. + +.. _netcdf-and-hdf5: + +NetCDF and HDF5 +--------------- + +For NetCDF and HDF5 files, an attempt will be made to extract metadata in NcML_ (XML) format and save it as an auxiliary file. (See also :doc:`/developers/aux-file-support` in the Developer Guide.) A previewer for these NcML files is available (see :ref:`file-previews`). + +.. _NcML: https://docs.unidata.ucar.edu/netcdf-java/current/userguide/ncml_overview.html + Compressed Files ---------------- diff --git a/doc/sphinx-guides/source/versions.rst b/doc/sphinx-guides/source/versions.rst index e0a344de9a1..4badeabef40 100755 --- a/doc/sphinx-guides/source/versions.rst +++ b/doc/sphinx-guides/source/versions.rst @@ -4,9 +4,11 @@ Dataverse Software Documentation Versions ========================================= -This list provides a way to refer to the documentation for previous versions of the Dataverse Software. In order to learn more about the updates delivered from one version to another, visit the `Releases `__ page in our GitHub repo. +This list provides a way to refer to the documentation for previous and future versions of the Dataverse Software. In order to learn more about the updates delivered from one version to another, visit the `Releases `__ page in our GitHub repo. -- 5.12.1 +- `develop Git branch `__ +- 5.13 +- `5.12.1 `__ - `5.12 `__ - `5.11.1 `__ - `5.11 `__ diff --git a/downloads/download.sh b/downloads/download.sh index 9d90e75b925..a0475f4cbe5 100755 --- a/downloads/download.sh +++ b/downloads/download.sh @@ -1,5 +1,5 @@ #!/bin/sh -curl -L -O https://s3.eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/6.2022.1/payara-6.2022.1.zip +curl -L -O https://s3.eu-west-1.amazonaws.com/payara.fish/Payara+Downloads/6.2023.2/payara-6.2023.2.zip curl -L -O https://archive.apache.org/dist/lucene/solr/8.11.1/solr-8.11.1.tgz curl -L -O https://search.maven.org/remotecontent?filepath=org/jboss/weld/weld-osgi-bundle/2.2.10.Final/weld-osgi-bundle-2.2.10.Final-glassfish4.jar curl -s -L http://sourceforge.net/projects/schemaspy/files/schemaspy/SchemaSpy%205.0.0/schemaSpy_5.0.0.jar/download > schemaSpy_5.0.0.jar diff --git a/modules/dataverse-parent/pom.xml b/modules/dataverse-parent/pom.xml index 601e5c54d0c..6231a5efc16 100644 --- a/modules/dataverse-parent/pom.xml +++ b/modules/dataverse-parent/pom.xml @@ -1,5 +1,5 @@ + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 edu.harvard.iq @@ -130,7 +130,7 @@ - 5.12.1 + 6.0.0 11 UTF-8 @@ -147,26 +147,26 @@ -Duser.timezone=${project.timezone} -Dfile.encoding=${project.build.sourceEncoding} -Duser.language=${project.language} -Duser.region=${project.region} - 6.2022.2 - 42.5.0 + 6.2023.2 + 42.5.4 8.11.1 - 1.12.290 - 0.177.0 + 1.12.419 + 0.190.0 - 1.7.35 + 2.0.6 2.11.0 1.2 3.12.0 - 1.21 - 4.5.13 - 4.4.14 + 1.22 + 4.5.14 + 4.4.16 - 5.0.0-RC2 + 5.0.0 - 1.15.0 + 1.17.6 2.10.1 4.13.1 @@ -174,21 +174,23 @@ ${junit.jupiter.version} 2.28.2 - 9.3 + 10.8.0 - 3.8.1 - 3.2.2 + 3.11.0 + 3.3.0 3.3.2 - 3.2.0 - 3.0.0-M1 - 3.0.0-M5 - 3.0.0-M5 + 3.5.0 + 3.1.0 + 3.0.0-M9 + 3.0.0-M9 3.3.0 - 3.1.2 + 3.2.1 + + 3.2.1 - 0.40.2 + 0.41.0 @@ -347,44 +349,4 @@ - - - ct - - - 5.2022.4 - - - - - - - io.github.git-commit-id - git-commit-id-maven-plugin - 5.0.0 - - - retrieve-git-details - - revision - - initialize - - - - ${project.basedir}/../../.git - UTC - 8 - false - - - - - - - - diff --git a/modules/dataverse-parent/pom.xml.versionsBackup b/modules/dataverse-parent/pom.xml.versionsBackup new file mode 100644 index 00000000000..601e5c54d0c --- /dev/null +++ b/modules/dataverse-parent/pom.xml.versionsBackup @@ -0,0 +1,390 @@ + + 4.0.0 + + edu.harvard.iq + dataverse-parent + ${revision} + pom + + dataverse-parent + https://dataverse.org + + + ../../pom.xml + ../../scripts/zipdownload + ../container-base + + + + + + + + + + fish.payara.api + payara-bom + ${payara.version} + pom + import + + + com.amazonaws + aws-java-sdk-bom + ${aws.version} + pom + import + + + com.google.cloud + google-cloud-bom + ${google.cloud.version} + pom + import + + + + + + org.postgresql + postgresql + ${postgresql.version} + + + commons-logging + commons-logging + ${commons.logging.version} + + + org.apache.commons + commons-lang3 + ${commons.lang3.version} + + + + org.apache.httpcomponents + httpclient + ${apache.httpcomponents.client.version} + + + + org.apache.httpcomponents + httpmime + ${apache.httpcomponents.client.version} + + + + org.apache.httpcomponents + httpcore + ${apache.httpcomponents.core.version} + + + + + commons-io + commons-io + ${commons.io.version} + + + + org.apache.commons + commons-compress + ${commons.compress.version} + + + + + org.slf4j + slf4j-api + ${slf4j.version} + + + org.slf4j + slf4j-jdk14 + ${slf4j.version} + + + org.slf4j + jcl-over-slf4j + ${slf4j.version} + + + org.slf4j + jul-to-slf4j + ${slf4j.version} + + + + + + org.testcontainers + testcontainers-bom + ${testcontainers.version} + import + pom + + + + + + + 5.12.1 + + 11 + UTF-8 + -Xdoclint:none + + + UTC + en + US + + -Duser.timezone=${project.timezone} -Dfile.encoding=${project.build.sourceEncoding} -Duser.language=${project.language} -Duser.region=${project.region} + + + 6.2022.2 + 42.5.0 + 8.11.1 + 1.12.290 + 0.177.0 + + + 1.7.35 + 2.11.0 + 1.2 + 3.12.0 + 1.21 + 4.5.13 + 4.4.14 + + + 5.0.0-RC2 + + + 1.15.0 + 2.10.1 + + 4.13.1 + 5.7.0 + ${junit.jupiter.version} + 2.28.2 + + 9.3 + + + 3.8.1 + 3.2.2 + 3.3.2 + 3.2.0 + 3.0.0-M1 + 3.0.0-M5 + 3.0.0-M5 + 3.3.0 + 3.1.2 + + + 0.40.2 + + + + + central + Central Repository + https://repo.maven.apache.org/maven2 + default + + false + + + never + + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + ${maven-compiler-plugin.version} + + + org.apache.maven.plugins + maven-jar-plugin + ${maven-jar-plugin.version} + + + org.apache.maven.plugins + maven-war-plugin + ${maven-war-plugin.version} + + + org.apache.maven.plugins + maven-dependency-plugin + ${maven-dependency-plugin.version} + + + org.apache.maven.plugins + maven-install-plugin + ${maven-install-plugin.version} + + + org.apache.maven.plugins + maven-surefire-plugin + ${maven-surefire-plugin.version} + + + org.apache.maven.plugins + maven-failsafe-plugin + ${maven-failsafe-plugin.version} + + + org.apache.maven.plugins + maven-checkstyle-plugin + ${maven-checkstyle-plugin.version} + + + com.puppycrawl.tools + checkstyle + ${checkstyle.version} + + + + + io.fabric8 + docker-maven-plugin + ${fabric8-dmp.version} + + + + + + + + + payara-nexus-artifacts + Payara Nexus Artifacts + https://nexus.payara.fish/repository/payara-artifacts + + true + + + false + + + + + payara-patched-externals + Payara Patched Externals + https://raw.github.com/payara/Payara_PatchedProjects/master + + true + + + false + + + + central-repo + Central Repository + https://repo1.maven.org/maven2 + default + + + prime-repo + PrimeFaces Maven Repository + https://repository.primefaces.org + default + + + dataone.org + https://maven.dataone.org + + true + + + true + + + + unidata-all + Unidata All + https://artifacts.unidata.ucar.edu/repository/unidata-all/ + + + dvn.private + Local repository for hosting jars not available from network repositories. + file://${project.basedir}/local_lib + + + + oss-sonatype + oss-sonatype + + https://oss.sonatype.org/content/repositories/snapshots/ + + + true + + + + s01-oss-sonatype + s01-oss-sonatype + + https://s01.oss.sonatype.org/content/repositories/snapshots/ + + + true + + + + + + + + ct + + + 5.2022.4 + + + + + + + io.github.git-commit-id + git-commit-id-maven-plugin + 5.0.0 + + + retrieve-git-details + + revision + + initialize + + + + ${project.basedir}/../../.git + UTC + 8 + false + + + + + + + + + diff --git a/pom.xml b/pom.xml index c202c0b1056..5008ad40d51 100644 --- a/pom.xml +++ b/pom.xml @@ -1,6 +1,6 @@ + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 4.0.0 @@ -20,12 +20,61 @@ false 1.2.18.4 - 8.5.10 + 9.15.1 1.20.1 - 0.8.7 - 5.2.1 - 2.4.1 - 5.5.3 + 0.8.8 + 5.2.3 + 2.7.0 + 6.0.0-beta1 + 1.1.3 + 1.6.2 + 3.1 + 2.0.0-SNAPSHOT + 1.3.1 + 2.10.1 + 1.5.1 + 0.4 + 31.1-jre + 12.0.0 + 1.0.10 + 1.10.0 + 2.2 + 1.7 + 8.11.1 + 1.2.0 + 2012-10-25-generated + 8.1.1 + 5.0 + 6.0 + 1.8-5 + 1.8-5 + 1.4.0 + 4.1 + 6.0.0-SNAPSHOT + 1.0.3 + 1.15.4 + 6.3.1 + 1.15 + 0.10.4 + 1.10.0 + 8.3.3 + 10.7 + 1.0.1 + 1.4.9 + 7.1.2 + 2.1.1 + 4.3.0 + 1.1 + 2.2 + 3.24.2 + 2.9.1 + 5.3.0 + 1.5.1 + 3.8.7 + 3.2.1 + 1.2.8 + 4.3.0 + 2.4.0-b180830.0359 io.gdcc sword2-server - 2.0.0-SNAPSHOT + ${sword2.version} @@ -113,12 +162,12 @@ com.apicatalog titanium-json-ld - 1.3.1 + ${apicatalog.version} com.google.code.gson gson - 2.8.9 + ${gson.version} compile @@ -138,12 +187,12 @@ but it seemed better to not add another repo. --> org.everit.json org.everit.json.schema - 1.5.1 + ${everit.version} org.mindrot jbcrypt - 0.4 + ${jbcrypt.version} org.postgresql @@ -163,7 +212,7 @@ com.google.guava guava - 29.0-jre + ${guava.version} jar @@ -204,18 +253,18 @@ org.primefaces primefaces - 11.0.0 + ${primefaces.version} jakarta org.primefaces.themes all-themes - 1.0.10 + ${primefaces.themes.version} org.omnifaces omnifaces - 4.0-M13 + ${omnifaces.version} @@ -252,63 +301,63 @@ org.apache.commons commons-text - 1.10.0 + ${commons.text.version} org.apache.commons commons-math - 2.2 + ${commons.math.version} commons-validator commons-validator - 1.7 + ${commons.validator.version} org.apache.solr solr-solrj - 8.11.1 + ${solr.version} colt colt - 1.2.0 + ${colt.version} nom.tam.fits fits - 2012-10-25-generated + ${fits.version} net.handle handle - 8.1.1 + ${handle.version} edu.harvard.iq.dvn unf5 - 5.0 + ${unf5.version} org.dataverse unf - 6.0 + ${unf.version} org.nuiton.thirdparty REngine - 0.6-1 + ${rengine.version} org.nuiton.thirdparty Rserve - 0.6-1 + ${rserve.version} @@ -345,61 +394,61 @@ com.github.jai-imageio jai-imageio-core - 1.3.1 + ${imageio.version} org.ocpsoft.rewrite rewrite-servlet - 6.0.0-SNAPSHOT + ${rewrite.version} org.ocpsoft.rewrite rewrite-config-prettyfaces - 6.0.0-SNAPSHOT + ${rewrite.version} edu.ucsb.nceas ezid - 1.0.0 + ${nceas.version} jar org.jsoup jsoup - 1.15.3 + ${jsoup.version} io.searchbox jest - 0.1.7 + ${searchbox.version} commons-codec commons-codec - 1.15 + ${commons.codec.version} org.javaswift joss - 0.10.0 + ${javaswift.version} org.apache.commons commons-csv - 1.2 + ${apache.commons.version} com.github.scribejava scribejava-apis - 6.9.0 + ${scribejava.version} com.nimbusds oauth2-oidc-sdk - 9.41.1 + ${nimbusds.version} @@ -416,7 +465,7 @@ com.google.auto.service auto-service - 1.0-rc2 + ${auto.service.version} true jar @@ -433,7 +482,7 @@ com.mashape.unirest unirest-java - 1.4.9 + ${unirest.version} @@ -444,7 +493,7 @@ org.duracloud common - 7.1.1 + ${duracloud.version} org.slf4j @@ -459,7 +508,7 @@ org.duracloud storeclient - 7.1.1 + ${duracloud.version} org.slf4j @@ -490,7 +539,7 @@ org.apache.opennlp opennlp-tools - 1.9.1 + ${opennlp.version} com.google.cloud @@ -502,13 +551,13 @@ com.auth0 java-jwt - 3.19.1 + ${auth0.version} io.github.erdtman java-json-canonicalization - 1.1 + ${erdtman.version} edu.ucar @@ -538,31 +587,31 @@ org.hamcrest hamcrest-library - 2.2 + ${hamcrest.version} test org.assertj assertj-core - 3.20.2 + ${assertj.version} test org.xmlunit xmlunit-assertj3 - 2.8.2 + ${xmlunit.version} test - com.jayway.restassured + io.rest-assured rest-assured - 2.4.0 + ${restassured.version} test org.skyscreamer jsonassert - 1.5.0 + ${skyscreamer.version} test @@ -652,6 +701,26 @@ + + org.apache.maven.plugins + maven-enforcer-plugin + ${enforcer.version} + + + enforce-maven + + enforce + + + + + ${maven.version} + + + + + + org.apache.maven.plugins maven-compiler-plugin @@ -690,7 +759,7 @@ de.qaware.maven go-offline-maven-plugin - 1.2.1 + ${qaware.version} @@ -723,12 +792,12 @@ org.eluder.coveralls coveralls-maven-plugin - 4.3.0 + ${coveralls.version} javax.xml.bind jaxb-api - 2.3.1 + ${jaxb.api.version} diff --git a/pom.xml.versionsBackup b/pom.xml.versionsBackup new file mode 100644 index 00000000000..8e987ab6741 --- /dev/null +++ b/pom.xml.versionsBackup @@ -0,0 +1,823 @@ + + + 4.0.0 + + + edu.harvard.iq + dataverse-parent + ${revision} + modules/dataverse-parent + + + + dataverse + war + dataverse + + false + 1.2.18.4 + 8.5.10 + 1.20.1 + 0.8.7 + 5.2.1 + 2.4.1 + 5.5.3 + + + + + + + + org.apache.abdera + abdera-core + 1.1.3 + + + org.apache.abdera + abdera-i18n + 1.1.3 + + + + + + + + + + org.slf4j + slf4j-jdk14 + runtime + + + + org.passay + passay + 1.6.0 + + + + + commons-httpclient + commons-httpclient + 3.1 + + + + + io.gdcc + sword2-server + 2.0.0-SNAPSHOT + + + + org.apache.abdera + abdera-core + + + + + org.apache.abdera + abdera-i18n + + + + + com.amazonaws + aws-java-sdk-s3 + + + + com.apicatalog + titanium-json-ld + 1.3.1 + + + com.google.code.gson + gson + 2.8.9 + compile + + + + com.fasterxml.jackson.core + jackson-core + provided + + + + com.fasterxml.jackson.core + jackson-databind + provided + + + + org.everit.json + org.everit.json.schema + 1.5.1 + + + org.mindrot + jbcrypt + 0.4 + + + org.postgresql + postgresql + + + org.flywaydb + flyway-core + ${flyway.version} + + + + org.eclipse.persistence + org.eclipse.persistence.jpa + provided + + + com.google.guava + guava + 29.0-jre + jar + + + + + org.eclipse.microprofile.config + microprofile-config-api + provided + + + jakarta.platform + jakarta.jakartaee-api + provided + + + + + + org.eclipse.angus + angus-activation + provided + + + + + + org.eclipse.parsson + jakarta.json + provided + + + + + org.glassfish + jakarta.faces + provided + + + org.primefaces + primefaces + 11.0.0 + jakarta + + + org.primefaces.themes + all-themes + 1.0.10 + + + org.omnifaces + omnifaces + 4.0-M13 + + + + + jakarta.validation + jakarta.validation-api + provided + + + org.hibernate.validator + hibernate-validator + provided + + + + + + org.glassfish.expressly + expressly + provided + + + + commons-io + commons-io + + + + org.apache.commons + commons-lang3 + + + + + org.apache.commons + commons-text + 1.10.0 + + + org.apache.commons + commons-math + 2.2 + + + commons-validator + commons-validator + 1.7 + + + + org.apache.solr + solr-solrj + 8.11.1 + + + colt + colt + 1.2.0 + + + + nom.tam.fits + fits + 2012-10-25-generated + + + net.handle + handle + 8.1.1 + + + + edu.harvard.iq.dvn + unf5 + 5.0 + + + + org.dataverse + unf + 6.0 + + + + + org.nuiton.thirdparty + REngine + 0.6-1 + + + org.nuiton.thirdparty + Rserve + 0.6-1 + + + + org.apache.poi + poi + ${poi.version} + + + org.apache.poi + poi-ooxml + ${poi.version} + + + org.apache.poi + poi-scratchpad + ${poi.version} + + + org.openpreservation.jhove + jhove-core + ${jhove.version} + + + org.openpreservation.jhove + jhove-modules + ${jhove.version} + + + org.openpreservation.jhove + jhove-ext-modules + ${jhove.version} + + + + com.github.jai-imageio + jai-imageio-core + 1.3.1 + + + org.ocpsoft.rewrite + rewrite-servlet + 6.0.0-SNAPSHOT + + + org.ocpsoft.rewrite + rewrite-config-prettyfaces + 6.0.0-SNAPSHOT + + + edu.ucsb.nceas + ezid + 1.0.0 + jar + + + org.jsoup + jsoup + 1.15.3 + + + io.searchbox + jest + 0.1.7 + + + commons-codec + commons-codec + 1.15 + + + + org.javaswift + joss + 0.10.0 + + + org.apache.commons + commons-csv + 1.2 + + + + com.github.scribejava + scribejava-apis + 6.9.0 + + + + com.nimbusds + oauth2-oidc-sdk + 9.41.1 + + + + io.gdcc + xoai-data-provider + ${gdcc.xoai.version} + + + io.gdcc + xoai-service-provider + ${gdcc.xoai.version} + + + + com.google.auto.service + auto-service + 1.0-rc2 + true + jar + + + + org.glassfish.jersey.core + jersey-server + + + + org.glassfish.jersey.media + jersey-media-multipart + + + com.mashape.unirest + unirest-java + 1.4.9 + + + + org.apache.commons + commons-compress + + + + org.duracloud + common + 7.1.1 + + + org.slf4j + log4j-over-slf4j + + + ch.qos.logback + logback-classic + + + + + org.duracloud + storeclient + 7.1.1 + + + org.slf4j + log4j-over-slf4j + + + com.amazonaws + aws-java-sdk-sqs + + + ch.qos.logback + logback-classic + + + + + + org.apache.tika + tika-core + ${tika.version} + + + org.apache.tika + tika-parsers-standard-package + ${tika.version} + + + + org.apache.opennlp + opennlp-tools + 1.9.1 + + + com.google.cloud + google-cloud-storage + + + + + + com.auth0 + java-jwt + 3.19.1 + + + + io.github.erdtman + java-json-canonicalization + 1.1 + + + edu.ucar + cdm-core + ${netcdf.version} + + + + + org.junit.jupiter + junit-jupiter + ${junit.jupiter.version} + test + + + junit + junit + ${junit.version} + test + + + org.junit.vintage + junit-vintage-engine + ${junit.vintage.version} + test + + + org.hamcrest + hamcrest-library + 2.2 + test + + + org.assertj + assertj-core + 3.20.2 + test + + + org.xmlunit + xmlunit-assertj3 + 2.8.2 + test + + + io.restassured + rest-assured + 2.4.0 + test + + + org.skyscreamer + jsonassert + 1.5.0 + test + + + com.vaadin.external.google + android-json + + + + + org.testcontainers + testcontainers + test + + + org.testcontainers + junit-jupiter + test + + + org.testcontainers + postgresql + test + + + org.mockito + mockito-core + ${mockito.version} + test + + + org.mockito + mockito-junit-jupiter + ${mockito.version} + test + + + io.smallrye.config + smallrye-config + ${smallrye-mpconfig.version} + test + + + + + + + + src/main/java + + *.properties + **/*.properties + **/mime.types + **/*.R + + + + src/main/resources + + **/*.sql + **/*.xml + **/firstNames/*.* + **/*.xsl + **/services/* + + + + src/main/resources + + true + + **/*.properties + + + + + + org.apache.maven.plugins + maven-enforcer-plugin + 3.2.1 + + + enforce-maven + + enforce + + + + + 3.8.7 + + + + + + + + org.apache.maven.plugins + maven-compiler-plugin + + ${target.java.version} + + ${compilerArgument} + + + + org.apache.maven.plugins + maven-jar-plugin + + + + true + true + + + + + + org.apache.maven.plugins + maven-war-plugin + + true + false + + + true + true + + + + + + de.qaware.maven + go-offline-maven-plugin + 1.2.1 + + + + + + + org.jacoco + jacoco-maven-plugin + ${jacoco.version} + + ${basedir}/target/coverage-reports/jacoco-unit.exec + ${basedir}/target/coverage-reports/jacoco-unit.exec + + + + jacoco-initialize + + prepare-agent + + + + jacoco-site + package + + report + + + + + + org.eluder.coveralls + coveralls-maven-plugin + 4.3.0 + + + javax.xml.bind + jaxb-api + 2.3.1 + + + + + org.apache.maven.plugins + maven-surefire-plugin + + + + ${testsToExclude} + ${skipUnitTests} + + + + org.apache.maven.plugins + maven-checkstyle-plugin + + checkstyle.xml + UTF-8 + true + + + + + + + dev + + + + true + + + edu.harvard.iq.dataverse.NonEssentialTests + + + + all-unit-tests + + + + tc + + true + 9.6 + + + + + org.apache.maven.plugins + maven-failsafe-plugin + ${maven-failsafe-plugin.version} + + testcontainers + + ${postgresql.server.version} + + + + + + integration-test + verify + + + + + + + + + diff --git a/scripts/api/data-deposit/upload-file b/scripts/api/data-deposit/upload-file index 576603d9367..b3d08c50cac 100755 --- a/scripts/api/data-deposit/upload-file +++ b/scripts/api/data-deposit/upload-file @@ -7,7 +7,7 @@ if [ -z "$1" ]; then else EDIT_MEDIA_URL=$1 fi -curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H "Content-Disposition: filename=trees.zip" -H "Content-Type: application/zip" -H "Packaging: http://purl.org/net/sword/package/SimpleZip" -u $USERNAME:$PASSWORD $EDIT_MEDIA_URL \ +curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H "Content-Disposition: attachment;filename=trees.zip" -H "Content-Type: application/zip" -H "Packaging: http://purl.org/net/sword/package/SimpleZip" -u $USERNAME:$PASSWORD $EDIT_MEDIA_URL \ | xmllint -format - -#curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H "Content-Disposition: filename=trees.zip" -H "Content-Type: application/zip" -H "Packaging: http://purl.org/net/sword/package/SimpleZip" https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/edit-media/study/doi:10.5072/FK2/19 \ +#curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H "Content-Disposition: attachment;filename=trees.zip" -H "Content-Type: application/zip" -H "Packaging: http://purl.org/net/sword/package/SimpleZip" https://$USERNAME:$PASSWORD@$DVN_SERVER/dvn/api/data-deposit/v1/swordv2/edit-media/study/doi:10.5072/FK2/19 \ #| xmllint -format - diff --git a/scripts/api/data/metadatablocks/citation.tsv b/scripts/api/data/metadatablocks/citation.tsv index 1b1ff0ae819..be32bb7134e 100644 --- a/scripts/api/data/metadatablocks/citation.tsv +++ b/scripts/api/data/metadatablocks/citation.tsv @@ -43,7 +43,7 @@ producerURL URL The URL of the producer's website https:// url 39 #VALUE FALSE FALSE FALSE FALSE FALSE FALSE producer citation producerLogoURL Logo URL The URL of the producer's logo https:// url 40
FALSE FALSE FALSE FALSE FALSE FALSE producer citation productionDate Production Date The date when the data were produced (not distributed, published, or archived) YYYY-MM-DD date 41 TRUE FALSE FALSE TRUE FALSE FALSE citation - productionPlace Production Location The location where the data and any related materials were produced or collected text 42 FALSE FALSE FALSE FALSE FALSE FALSE citation + productionPlace Production Location The location where the data and any related materials were produced or collected text 42 TRUE FALSE TRUE TRUE FALSE FALSE citation contributor Contributor The entity, such as a person or organization, responsible for collecting, managing, or otherwise contributing to the development of the Dataset none 43 : FALSE FALSE TRUE FALSE FALSE FALSE citation http://purl.org/dc/terms/contributor contributorType Type Indicates the type of contribution made to the dataset text 44 #VALUE TRUE TRUE FALSE TRUE FALSE FALSE contributor citation contributorName Name The name of the contributor, e.g. the person's name or the name of an organization 1) FamilyName, GivenName or 2) Organization text 45 #VALUE TRUE FALSE FALSE TRUE FALSE FALSE contributor citation diff --git a/scripts/installer/install b/scripts/installer/install index 2208f014606..6439158cb8b 100755 --- a/scripts/installer/install +++ b/scripts/installer/install @@ -685,7 +685,7 @@ print "can publish datasets. Once you receive the account name and password, add print "as the following two JVM options:\n"; print "\t-Ddoi.username=...\n"; print "\t-Ddoi.password=...\n"; -print "and restart payara5\n"; +print "and restart payara6\n"; print "If this is a production Dataverse and you are planning to register datasets as \n"; print "\"real\", non-test DOIs or Handles, consult the \"Persistent Identifiers and Publishing Datasets\"\n"; print "section of the Installataion guide, on how to configure your Dataverse with the proper registration\n"; @@ -954,9 +954,9 @@ sub setup_appserver { # The JHOVE conf file has an absolute PATH of the JHOVE config schema file (uh, yeah...) # - so it may need to be readjusted here: - if ( $glassfish_dir ne "/usr/local/payara5" ) + if ( $glassfish_dir ne "/usr/local/payara6" ) { - system( "sed 's:/usr/local/payara5:$glassfish_dir:g' < " . $JHOVE_CONFIG_DIST . " > " . $glassfish_dir . "/glassfish/domains/domain1/config/" . $JHOVE_CONFIG); + system( "sed 's:/usr/local/payara6:$glassfish_dir:g' < " . $JHOVE_CONFIG_DIST . " > " . $glassfish_dir . "/glassfish/domains/domain1/config/" . $JHOVE_CONFIG); } else { diff --git a/scripts/installer/install.py b/scripts/installer/install.py index e9b9e985e80..0dce859d35e 100644 --- a/scripts/installer/install.py +++ b/scripts/installer/install.py @@ -252,7 +252,7 @@ # 1d. check java version java_version = subprocess.check_output(["java", "-version"], stderr=subprocess.STDOUT).decode() print("Found java version "+java_version) - if not re.search('(1.8|11)', java_version): + if not re.search('(1.8|17)', java_version): sys.exit("Dataverse requires OpenJDK 1.8 or 11. Please make sure it's in your PATH, and try again.") # 1e. check if the setup scripts - setup-all.sh, are available as well, maybe? diff --git a/scripts/search/tests/create-all-and-test b/scripts/search/tests/create-all-and-test index cc188eed4fc..eaa57530aa7 100755 --- a/scripts/search/tests/create-all-and-test +++ b/scripts/search/tests/create-all-and-test @@ -11,7 +11,7 @@ echo curl -s -X POST -H "Content-type:application/json" -d @scripts/search/tests/data/dataset-finch1.json "http://localhost:8080/api/dataverses/finches/datasets/?key=$FINCHKEY" echo "Uploading a file via the SWORD API" . scripts/search/assumptions -curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H 'Content-Disposition: filename=trees.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u $SPRUCEKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$FIRST_SPRUCE_DOI +curl -s --insecure --data-binary @scripts/search/data/binary/trees.zip -H 'Content-Disposition: attachment;filename=trees.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u $SPRUCEKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$FIRST_SPRUCE_DOI echo echo "Uploading a file via the native API" # echo $FIRST_FINCH_DOI # FIXME: Why is this empty? diff --git a/scripts/search/tests/create-saved-search-and-test b/scripts/search/tests/create-saved-search-and-test index ac540925caf..07c215f1963 100755 --- a/scripts/search/tests/create-saved-search-and-test +++ b/scripts/search/tests/create-saved-search-and-test @@ -7,7 +7,7 @@ scripts/search/create-psi-dvs > /tmp/psi-dvs1 curl -s -X POST -H "Content-type:application/json" -d @scripts/search/tests/data/dataset-mali1.json "http://localhost:8080/api/dataverses/psimali/datasets/?key=$PSIADMINKEY" >/dev/null curl -s -X POST -H "Content-type:application/json" -d @scripts/search/tests/data/dataset-mali2.json "http://localhost:8080/api/dataverses/psimali/datasets/?key=$PSIADMINKEY" >/dev/null WOMEN_IN_MALI_DOI=`curl -s --globoff "http://localhost:8080/api/search?key=$ADMINKEY&q=title:\"Women+in+Mali+dataset+1\"" | jq '.data.items[].global_id' | sed 's/"//g'` -curl -s --insecure --data-binary @scripts/search/data/binary/health.zip -H 'Content-Disposition: filename=health.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u $PSIADMINKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$WOMEN_IN_MALI_DOI >/dev/null +curl -s --insecure --data-binary @scripts/search/data/binary/health.zip -H 'Content-Disposition: attachment;filename=health.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u $PSIADMINKEY: https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$WOMEN_IN_MALI_DOI >/dev/null scripts/search/saved-search-setup curl -s -X PUT http://localhost:8080/api/admin/savedsearches/makelinks/all | jq . echo "Running verification tests (silence is golden)" diff --git a/scripts/search/tests/grant-authusers-add-on-root b/scripts/search/tests/grant-authusers-add-on-root index 08b245fa561..f8f298c77a5 100755 --- a/scripts/search/tests/grant-authusers-add-on-root +++ b/scripts/search/tests/grant-authusers-add-on-root @@ -1,5 +1,4 @@ #!/bin/sh -. scripts/search/export-keys +export ADMINKEY=`cat /tmp/setup-all.sh.out | grep apiToken| jq .data.apiToken | tr -d \"` OUTPUT=`curl -s -X POST -H "Content-type:application/json" -d "{\"assignee\": \":authenticated-users\",\"role\": \"fullContributor\"}" "http://localhost:8080/api/dataverses/root/assignments?key=$ADMINKEY"` echo $OUTPUT -echo $OUTPUT | jq ' .data | {assignee,_roleAlias}' diff --git a/scripts/search/tests/publish-dataverse-root b/scripts/search/tests/publish-dataverse-root index a4c1585d41a..20bafe1945e 100755 --- a/scripts/search/tests/publish-dataverse-root +++ b/scripts/search/tests/publish-dataverse-root @@ -1,6 +1,4 @@ #!/bin/sh -. scripts/search/export-keys +export ADMINKEY=`cat /tmp/setup-all.sh.out | grep apiToken| jq .data.apiToken | tr -d \"` OUTPUT=`cat /dev/null | curl -s --insecure -u $ADMINKEY: -X POST -H 'In-Progress: false' --data-binary @- https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit/dataverse/root` echo $OUTPUT -echo -echo $OUTPUT | xmllint -format - diff --git a/scripts/search/tests/upload-1000-files b/scripts/search/tests/upload-1000-files index a4c1d46f5b5..cf016fd1aeb 100755 --- a/scripts/search/tests/upload-1000-files +++ b/scripts/search/tests/upload-1000-files @@ -2,4 +2,4 @@ . scripts/search/export-keys . scripts/search/assumptions echo "Uploading 1000 files" -curl -s --insecure --data-binary @scripts/search/data/binary/1000files.zip -H 'Content-Disposition: filename=1000files.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u spruce:spruce https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$FIRST_SPRUCE_DOI +curl -s --insecure --data-binary @scripts/search/data/binary/1000files.zip -H 'Content-Disposition: attachment;filename=1000files.zip' -H 'Content-Type: application/zip' -H 'Packaging: http://purl.org/net/sword/package/SimpleZip' -u spruce:spruce https://localhost:8181/dvn/api/data-deposit/v1.1/swordv2/edit-media/study/$FIRST_SPRUCE_DOI diff --git a/scripts/vagrant/setup.sh b/scripts/vagrant/setup.sh index b378c78db37..b28a64bc7b8 100644 --- a/scripts/vagrant/setup.sh +++ b/scripts/vagrant/setup.sh @@ -19,8 +19,8 @@ cp /dataverse/conf/vagrant/etc/yum.repos.d/shibboleth.repo /etc/yum.repos.d #yum install -y shibboleth shibboleth-embedded-ds # java configuration et alia -dnf install -qy java-11-openjdk-devel httpd mod_ssl unzip -alternatives --set java /usr/lib/jvm/jre-11-openjdk/bin/java +dnf install -qy java-17-openjdk-devel httpd mod_ssl unzip +alternatives --set java /usr/lib/jvm/jre-17-openjdk/bin/java java -version # maven included in centos8 requires 1.8.0 - download binary instead @@ -28,10 +28,7 @@ wget -q https://archive.apache.org/dist/maven/maven-3/3.8.2/binaries/apache-mave tar xfz apache-maven-3.8.2-bin.tar.gz mkdir /opt/maven mv apache-maven-3.8.2/* /opt/maven/ -echo "export JAVA_HOME=/usr/lib/jvm/jre-openjdk" > /etc/profile.d/maven.sh -echo "export M2_HOME=/opt/maven" >> /etc/profile.d/maven.sh -echo "export MAVEN_HOME=/opt/maven" >> /etc/profile.d/maven.sh -echo "export PATH=/opt/maven/bin:${PATH}" >> /etc/profile.d/maven.sh +{ echo "export JAVA_HOME=/usr/lib/jvm/jre-openjdk" ; echo "export M2_HOME=/opt/maven" ; echo "export MAVEN_HOME=/opt/maven" ; echo "export PATH=/opt/maven/bin:${PATH}" } >> /etc/profile.d/maven.sh chmod 0755 /etc/profile.d/maven.sh # disable centos8 postgresql module and install postgresql13-server @@ -51,7 +48,7 @@ SOLR_USER=solr echo "Ensuring Unix user '$SOLR_USER' exists" useradd $SOLR_USER || : DOWNLOAD_DIR='/dataverse/downloads' -PAYARA_ZIP="$DOWNLOAD_DIR/payara-6.2022.1.zip" +PAYARA_ZIP="$DOWNLOAD_DIR/payara-6.2023.2.zip" SOLR_TGZ="$DOWNLOAD_DIR/solr-8.11.1.tgz" if [ ! -f $PAYARA_ZIP ] || [ ! -f $SOLR_TGZ ]; then echo "Couldn't find $PAYARA_ZIP or $SOLR_TGZ! Running download script...." diff --git a/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFile.java b/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFile.java index bbb844f87e8..d03ebbc6f7b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFile.java +++ b/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFile.java @@ -55,7 +55,10 @@ public class AuxiliaryFile implements Serializable { private String formatTag; private String formatVersion; - + + /** + * The application/entity that created the auxiliary file. + */ private String origin; private boolean isPublic; diff --git a/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFileServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFileServiceBean.java index 078da640b1b..8c96f98ce39 100644 --- a/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFileServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/AuxiliaryFileServiceBean.java @@ -70,9 +70,13 @@ public AuxiliaryFile save(AuxiliaryFile auxiliaryFile) { * @param type how to group the files such as "DP" for "Differentially * @param mediaType user supplied content type (MIME type) * Private Statistics". - * @return success boolean - returns whether the save was successful + * @param save boolean - true to save immediately, false to let the cascade + * do persist to the database. + * @return an AuxiliaryFile with an id when save=true (assuming no + * exceptions) or an AuxiliaryFile without an id that will be persisted + * later through the cascade. */ - public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile dataFile, String formatTag, String formatVersion, String origin, boolean isPublic, String type, MediaType mediaType) { + public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile dataFile, String formatTag, String formatVersion, String origin, boolean isPublic, String type, MediaType mediaType, boolean save) { StorageIO storageIO = null; AuxiliaryFile auxFile = new AuxiliaryFile(); @@ -114,7 +118,14 @@ public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile auxFile.setType(type); auxFile.setDataFile(dataFile); auxFile.setFileSize(storageIO.getAuxObjectSize(auxExtension)); - auxFile = save(auxFile); + if (save) { + auxFile = save(auxFile); + } else { + if (dataFile.getAuxiliaryFiles() == null) { + dataFile.setAuxiliaryFiles(new ArrayList<>()); + } + dataFile.getAuxiliaryFiles().add(auxFile); + } } catch (IOException ioex) { logger.severe("IO Exception trying to save auxiliary file: " + ioex.getMessage()); throw new InternalServerErrorException(); @@ -129,7 +140,11 @@ public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile } return auxFile; } - + + public AuxiliaryFile processAuxiliaryFile(InputStream fileInputStream, DataFile dataFile, String formatTag, String formatVersion, String origin, boolean isPublic, String type, MediaType mediaType) { + return processAuxiliaryFile(fileInputStream, dataFile, formatTag, formatVersion, origin, isPublic, type, mediaType, true); + } + public AuxiliaryFile lookupAuxiliaryFile(DataFile dataFile, String formatTag, String formatVersion) { Query query = em.createNamedQuery("AuxiliaryFile.lookupAuxiliaryFile"); diff --git a/src/main/java/edu/harvard/iq/dataverse/DOIDataCiteServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DOIDataCiteServiceBean.java index 9853f807d47..c7c3b525b9e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DOIDataCiteServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DOIDataCiteServiceBean.java @@ -187,14 +187,14 @@ public void deleteIdentifier(DvObject dvObject) throws IOException, HttpExceptio //ToDo - PidUtils currently has a DataCite API call that would get the status at DataCite for this identifier - that could be more accurate than assuming based on whether the dvObject has been published String idStatus = DRAFT; if(dvObject.isReleased()) { - idStatus = PUBLIC; + idStatus = PUBLIC; } if ( idStatus != null ) { switch ( idStatus ) { case RESERVED: case DRAFT: logger.log(Level.INFO, "Delete status is reserved.."); - //service only removes the identifier from the cache (since it was written before DOIs could be registered in draft state) + //service only removes the identifier from the cache (since it was written before DOIs could be registered in draft state) doiDataCiteRegisterService.deleteIdentifier(identifier); //So we call the deleteDraftIdentifier method below until things are refactored deleteDraftIdentifier(dvObject); @@ -217,8 +217,8 @@ public void deleteIdentifier(DvObject dvObject) throws IOException, HttpExceptio * deleted. */ private void deleteDraftIdentifier(DvObject dvObject) throws IOException { - - //ToDo - incorporate into DataCiteRESTfulClient + + //ToDo - incorporate into DataCiteRESTfulClient String baseUrl = systemConfig.getDataCiteRestApiUrlString(); String username = System.getProperty("doi.username"); String password = System.getProperty("doi.password"); @@ -240,8 +240,8 @@ private void deleteDraftIdentifier(DvObject dvObject) throws IOException { connection.setRequestProperty("Authorization", basicAuth); int status = connection.getResponseCode(); if(status!=HttpStatus.SC_NO_CONTENT) { - logger.warning("Incorrect Response Status from DataCite: " + status + " : " + connection.getResponseMessage()); - throw new HttpException("Status: " + status); + logger.warning("Incorrect Response Status from DataCite: " + status + " : " + connection.getResponseMessage()); + throw new HttpException("Status: " + status); } logger.fine("deleteDoi status for " + doi.asString() + ": " + status); } diff --git a/src/main/java/edu/harvard/iq/dataverse/DataCitation.java b/src/main/java/edu/harvard/iq/dataverse/DataCitation.java index e22b95479c2..aef9d9092d6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataCitation.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataCitation.java @@ -14,7 +14,6 @@ import java.io.OutputStream; import java.io.OutputStreamWriter; import java.io.Writer; -import java.text.ParseException; import java.text.SimpleDateFormat; import java.util.ArrayList; import java.util.Date; @@ -206,7 +205,7 @@ public String toString(boolean html, boolean anonymized) { } if (persistentId != null) { - // always show url format + // always show url format citationList.add(formatURL(persistentId.toURL().toString(), persistentId.toURL().toString(), html)); } citationList.add(formatString(publisher, html)); @@ -619,7 +618,7 @@ private void createEndNoteXML(XMLStreamWriter xmlw) throws XMLStreamException { } - public Map getDataCiteMetadata() { + public Map getDataCiteMetadata() { Map metadata = new HashMap<>(); String authorString = getAuthorsString(); @@ -637,9 +636,9 @@ public Map getDataCiteMetadata() { metadata.put("datacite.publisher", producerString); metadata.put("datacite.publicationyear", getYear()); return metadata; - } + } - + // helper methods private String formatString(String value, boolean escapeHtml) { return formatString(value, escapeHtml, ""); diff --git a/src/main/java/edu/harvard/iq/dataverse/DataCiteRESTfullClient.java b/src/main/java/edu/harvard/iq/dataverse/DataCiteRESTfullClient.java index 491f19ab36c..390d5047386 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataCiteRESTfullClient.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataCiteRESTfullClient.java @@ -203,22 +203,22 @@ public String inactiveDataset(String doi) { public static void main(String[] args) throws Exception { String doi = "10.5072/DVN/274533"; DataCiteRESTfullClient client = new DataCiteRESTfullClient("https://mds.test.datacite.org", "DATACITE_TEST_USERNAME", "DATACITE_TEST_PASSWORD"); -// System.out.println(client.getUrl(doi)); -// System.out.println(client.getMetadata(doi)); +// System.out.println(client.getUrl(doi)); +// System.out.println(client.getMetadata(doi)); // System.out.println(client.postMetadata(readAndClose("C:/Users/luopc/Desktop/datacite.xml", "utf-8"))); // System.out.println(client.postUrl("10.5072/000000001", "http://opendata.pku.edu.cn/dvn/dv/DAIM/faces/study/StudyPage.xhtml?globalId=hdl:TEST/10027&studyListingIndex=1_1acc4e9f23fa10b3cc0500d9eb5e")); // client.close(); -// String doi2 = "10.1/1.0003"; -// SimpleRESTfullClient client2 = new SimpleRESTfullClient("https://162.105.140.119:8443/mds", "PKULIB.IR", "luopengcheng","localhost.keystore"); -// System.out.println(client2.getUrl("10.1/1.0002")); -// System.out.println(client2.getUrl("10.1/1.0002")); -// System.out.println(client2.getMetadata(doi2)); -// client2.postUrl("10.1/1.0003", "http://ir.pku.edu.cn"); -// System.out.println(client2.postUrl("10.1/1.0008", "http://ir.pku.edu.cn")); -// System.out.println(client2.postMetadata(FileUtil.loadAsString(new File("C:/Users/luopc/Desktop/test/datacite-example-ResourceTypeGeneral_Collection-v3.0.xml"), "utf-8"))); -// System.out.println(client2.getMetadata("10.1/1.0007")); -// System.out.println(client2.inactiveDataSet("10.1/1.0007")); -// client2.close(); +// String doi2 = "10.1/1.0003"; +// SimpleRESTfullClient client2 = new SimpleRESTfullClient("https://162.105.140.119:8443/mds", "PKULIB.IR", "luopengcheng","localhost.keystore"); +// System.out.println(client2.getUrl("10.1/1.0002")); +// System.out.println(client2.getUrl("10.1/1.0002")); +// System.out.println(client2.getMetadata(doi2)); +// client2.postUrl("10.1/1.0003", "http://ir.pku.edu.cn"); +// System.out.println(client2.postUrl("10.1/1.0008", "http://ir.pku.edu.cn")); +// System.out.println(client2.postMetadata(FileUtil.loadAsString(new File("C:/Users/luopc/Desktop/test/datacite-example-ResourceTypeGeneral_Collection-v3.0.xml"), "utf-8"))); +// System.out.println(client2.getMetadata("10.1/1.0007")); +// System.out.println(client2.inactiveDataSet("10.1/1.0007")); +// client2.close(); } diff --git a/src/main/java/edu/harvard/iq/dataverse/DataFile.java b/src/main/java/edu/harvard/iq/dataverse/DataFile.java index 33b43b8f37f..66223c5aafd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataFile.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataFile.java @@ -51,8 +51,8 @@ * @author gdurand */ @NamedQueries({ - @NamedQuery( name="DataFile.removeFromDatasetVersion", - query="DELETE FROM FileMetadata f WHERE f.datasetVersion.id=:versionId and f.dataFile.id=:fileId"), + @NamedQuery( name="DataFile.removeFromDatasetVersion", + query="DELETE FROM FileMetadata f WHERE f.datasetVersion.id=:versionId and f.dataFile.id=:fileId"), @NamedQuery(name = "DataFile.findByCreatorId", query = "SELECT o FROM DataFile o WHERE o.creator.id=:creatorId"), @NamedQuery(name = "DataFile.findByReleaseUserId", @@ -64,9 +64,9 @@ }) @Entity @Table(indexes = {@Index(columnList="ingeststatus") - , @Index(columnList="checksumvalue") - , @Index(columnList="contenttype") - , @Index(columnList="restricted")}) + , @Index(columnList="checksumvalue") + , @Index(columnList="contenttype") + , @Index(columnList="restricted")}) public class DataFile extends DvObject implements Comparable { private static final Logger logger = Logger.getLogger(DatasetPage.class.getCanonicalName()); private static final long serialVersionUID = 1L; @@ -579,7 +579,7 @@ public FileMetadata getLatestPublishedFileMetadata() throws UnsupportedOperation if(fmd == null) { throw new UnsupportedOperationException("No published metadata version for DataFile " + this.getId()); } - + return fmd; } @@ -819,11 +819,11 @@ protected String toStringExtras() { FileMetadata fmd = getLatestFileMetadata(); return "label:" + (fmd!=null? fmd.getLabel() : "[no metadata]"); } - - @Override - public T accept( Visitor v ) { - return v.visit(this); - } + + @Override + public T accept( Visitor v ) { + return v.visit(this); + } @Override public String getDisplayName() { diff --git a/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java index b7b69198e31..f96297536c8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataFileServiceBean.java @@ -1017,9 +1017,9 @@ public Boolean isPreviouslyPublished(Long fileId){ } public void deleteFromVersion( DatasetVersion d, DataFile f ) { - em.createNamedQuery("DataFile.removeFromDatasetVersion") - .setParameter("versionId", d.getId()).setParameter("fileId", f.getId()) - .executeUpdate(); + em.createNamedQuery("DataFile.removeFromDatasetVersion") + .setParameter("versionId", d.getId()).setParameter("fileId", f.getId()) + .executeUpdate(); } /* @@ -1436,7 +1436,7 @@ public String generateDataFileIdentifier(DataFile datafile, GlobalIdServiceBean prepend = datafile.getOwner().getIdentifier() + "/"; } else { //If there's a shoulder prepend independent identifiers with it - prepend = settingsService.getValueForKey(SettingsServiceBean.Key.Shoulder, ""); + prepend = settingsService.getValueForKey(SettingsServiceBean.Key.Shoulder, ""); } switch (doiIdentifierType) { diff --git a/src/main/java/edu/harvard/iq/dataverse/Dataset.java b/src/main/java/edu/harvard/iq/dataverse/Dataset.java index 8afe4e3866e..2f196fc8a8e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Dataset.java +++ b/src/main/java/edu/harvard/iq/dataverse/Dataset.java @@ -6,6 +6,8 @@ import edu.harvard.iq.dataverse.license.License; import edu.harvard.iq.dataverse.makedatacount.DatasetExternalCitations; import edu.harvard.iq.dataverse.makedatacount.DatasetMetrics; +import edu.harvard.iq.dataverse.settings.JvmSettings; + import java.nio.file.Path; import java.nio.file.Paths; import java.sql.Timestamp; @@ -530,11 +532,8 @@ private Collection getCategoryNames() { @Deprecated public Path getFileSystemDirectory() { Path studyDir = null; - - String filesRootDirectory = System.getProperty("dataverse.files.directory"); - if (filesRootDirectory == null || filesRootDirectory.equals("")) { - filesRootDirectory = "/tmp/files"; - } + + String filesRootDirectory = JvmSettings.FILES_DIRECTORY.lookup(); if (this.getAlternativePersistentIndentifiers() != null && !this.getAlternativePersistentIndentifiers().isEmpty()) { for (AlternativePersistentIdentifier api : this.getAlternativePersistentIndentifiers()) { diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetField.java b/src/main/java/edu/harvard/iq/dataverse/DatasetField.java index c836a20893f..3b39935c1a8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetField.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetField.java @@ -393,7 +393,7 @@ public boolean isEmptyForDisplay() { private boolean isEmpty(boolean forDisplay) { if (datasetFieldType.isPrimitive()) { // primitive - List values = forDisplay ? getValues() : getValues_nondisplay(); + List values = forDisplay ? getValues() : getValues_nondisplay(); for (String value : values) { if (!StringUtils.isBlank(value) && !(forDisplay && DatasetField.NA_VALUE.equals(value))) { return false; diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java b/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java index 824b486a42d..d53573342ab 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetFieldType.java @@ -23,10 +23,10 @@ @NamedQueries({ @NamedQuery(name="DatasetFieldType.findByName", query= "SELECT dsfType FROM DatasetFieldType dsfType WHERE dsfType.name=:name"), - @NamedQuery(name = "DatasetFieldType.findAllFacetable", - query= "select dsfType from DatasetFieldType dsfType WHERE dsfType.facetable = true and dsfType.title != '' order by dsfType.id"), + @NamedQuery(name = "DatasetFieldType.findAllFacetable", + query= "select dsfType from DatasetFieldType dsfType WHERE dsfType.facetable = true and dsfType.title != '' order by dsfType.id"), @NamedQuery(name = "DatasetFieldType.findFacetableByMetadaBlock", - query= "select dsfType from DatasetFieldType dsfType WHERE dsfType.facetable = true and dsfType.title != '' and dsfType.metadataBlock.id = :metadataBlockId order by dsfType.id") + query= "select dsfType from DatasetFieldType dsfType WHERE dsfType.facetable = true and dsfType.title != '' and dsfType.metadataBlock.id = :metadataBlockId order by dsfType.id") }) @Entity @Table(indexes = {@Index(columnList="metadatablock_id"),@Index(columnList="parentdatasetfieldtype_id")}) @@ -308,7 +308,7 @@ public void setMetadataBlock(MetadataBlock metadataBlock) { private String uri; public String getUri() { - return uri; + return uri; } public JsonLDTerm getJsonLDTerm() { @@ -320,7 +320,7 @@ public JsonLDTerm getJsonLDTerm() { } public void setUri(String uri) { - this.uri=uri; + this.uri=uri; } /** diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java index d47d5284b1f..1834b263bba 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetPage.java @@ -946,7 +946,7 @@ public Set getFileIdsInVersionFromSolr(Long datasetVersionId, String patte try { queryResponse = solrClientService.getSolrClient().query(solrQuery); - } catch (HttpSolrClient.RemoteSolrException ex) { + } catch (org.apache.solr.client.solrj.impl.BaseHttpSolrClient.RemoteSolrException ex) { logger.fine("Remote Solr Exception: " + ex.getLocalizedMessage()); String msg = ex.getLocalizedMessage(); if (msg.contains(SearchFields.FILE_DELETED)) { @@ -1828,7 +1828,7 @@ public void updateOwnerDataverse() { GlobalId gid = dataset.getGlobalId(); dataset = new Dataset(); if(gid!=null) { - dataset.setGlobalId(gid); + dataset.setGlobalId(gid); } // initiate from scratch: (isolate the creation of a new dataset in its own method?) @@ -2045,10 +2045,10 @@ private String init(boolean initFull) { return permissionsWrapper.notAuthorized(); } //Wait until the create command before actually getting an identifier, except if we're using directUpload - //Need to assign an identifier prior to calls to requestDirectUploadUrl if direct upload is used. + //Need to assign an identifier prior to calls to requestDirectUploadUrl if direct upload is used. if ( isEmpty(dataset.getIdentifier()) && systemConfig.directUploadEnabled(dataset) ) { - CommandContext ctxt = commandEngine.getContext(); - GlobalIdServiceBean idServiceBean = GlobalIdServiceBean.getBean(ctxt); + CommandContext ctxt = commandEngine.getContext(); + GlobalIdServiceBean idServiceBean = GlobalIdServiceBean.getBean(ctxt); dataset.setIdentifier(ctxt.datasets().generateDatasetIdentifier(dataset, idServiceBean)); } dataverseTemplates.addAll(dataverseService.find(ownerId).getTemplates()); @@ -3809,31 +3809,31 @@ public String cancel() { } public void cancelCreate() { - //Stop any uploads in progress (so that uploadedFiles doesn't change) - uploadInProgress.setValue(false); + //Stop any uploads in progress (so that uploadedFiles doesn't change) + uploadInProgress.setValue(false); - logger.fine("Cancelling: " + newFiles.size() + " : " + uploadedFiles.size()); + logger.fine("Cancelling: " + newFiles.size() + " : " + uploadedFiles.size()); - //Files that have been finished and are now in the lower list on the page - for (DataFile newFile : newFiles.toArray(new DataFile[0])) { - FileUtil.deleteTempFile(newFile, dataset, ingestService); - } - logger.fine("Deleted newFiles"); + //Files that have been finished and are now in the lower list on the page + for (DataFile newFile : newFiles.toArray(new DataFile[0])) { + FileUtil.deleteTempFile(newFile, dataset, ingestService); + } + logger.fine("Deleted newFiles"); - //Files in the upload process but not yet finished - //ToDo - if files are added to uploadFiles after we access it, those files are not being deleted. With uploadInProgress being set false above, this should be a fairly rare race condition. - for (DataFile newFile : uploadedFiles.toArray(new DataFile[0])) { - FileUtil.deleteTempFile(newFile, dataset, ingestService); - } - logger.fine("Deleted uploadedFiles"); + //Files in the upload process but not yet finished + //ToDo - if files are added to uploadFiles after we access it, those files are not being deleted. With uploadInProgress being set false above, this should be a fairly rare race condition. + for (DataFile newFile : uploadedFiles.toArray(new DataFile[0])) { + FileUtil.deleteTempFile(newFile, dataset, ingestService); + } + logger.fine("Deleted uploadedFiles"); - try { - String alias = dataset.getOwner().getAlias(); - logger.info("alias: " + alias); - FacesContext.getCurrentInstance().getExternalContext().redirect("/dataverse.xhtml?alias=" + alias); - } catch (IOException ex) { - logger.info("Failed to issue a redirect to file download url."); - } + try { + String alias = dataset.getOwner().getAlias(); + logger.info("alias: " + alias); + FacesContext.getCurrentInstance().getExternalContext().redirect("/dataverse.xhtml?alias=" + alias); + } catch (IOException ex) { + logger.info("Failed to issue a redirect to file download url."); + } } private HttpClient getClient() { @@ -5494,7 +5494,7 @@ public List getCachedToolsForDataFile(Long fileId, ExternalTool.Ty return cachedTools; } DataFile dataFile = datafileService.find(fileId); - cachedTools = ExternalToolServiceBean.findExternalToolsByFile(externalTools, dataFile); + cachedTools = externalToolService.findExternalToolsByFile(externalTools, dataFile); cachedToolsByFileId.put(fileId, cachedTools); //add to map so we don't have to do the lifting again return cachedTools; } diff --git a/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java b/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java index be4cf82e527..0b97d98f91b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java +++ b/src/main/java/edu/harvard/iq/dataverse/DatasetVersion.java @@ -1,7 +1,8 @@ package edu.harvard.iq.dataverse; -import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.MarkupChecker; +import edu.harvard.iq.dataverse.util.PersonOrOrgUtil; +import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.DatasetFieldType.FieldType; import edu.harvard.iq.dataverse.branding.BrandingUtil; import edu.harvard.iq.dataverse.dataset.DatasetUtil; @@ -841,12 +842,26 @@ public String getDescriptionPlainText() { return MarkupChecker.stripAllTags(getDescription()); } - public List getDescriptionsPlainText() { - List plainTextDescriptions = new ArrayList<>(); + /* This method is (only) used in creating schema.org json-jd where Google requires a text description <5000 chars. + * + * @returns - a single string composed of all descriptions (joined with \n if more than one) truncated with a trailing '...' if >=5000 chars + */ + public String getDescriptionsPlainTextTruncated() { + List plainTextDescriptions = new ArrayList(); + for (String htmlDescription : getDescriptions()) { plainTextDescriptions.add(MarkupChecker.stripAllTags(htmlDescription)); } - return plainTextDescriptions; + String description = String.join("\n", plainTextDescriptions); + if (description.length() >= 5000) { + int endIndex = description.substring(0, 4997).lastIndexOf(" "); + if (endIndex == -1) { + //There are no spaces so just break anyway + endIndex = 4997; + } + description = description.substring(0, endIndex) + "..."; + } + return description; } /** @@ -1801,27 +1816,46 @@ public String getJsonLd() { for (DatasetAuthor datasetAuthor : this.getDatasetAuthors()) { JsonObjectBuilder author = Json.createObjectBuilder(); String name = datasetAuthor.getName().getDisplayValue(); + String identifierAsUrl = datasetAuthor.getIdentifierAsUrl(); DatasetField authorAffiliation = datasetAuthor.getAffiliation(); String affiliation = null; if (authorAffiliation != null) { - affiliation = datasetAuthor.getAffiliation().getDisplayValue(); + affiliation = datasetAuthor.getAffiliation().getValue(); } - // We are aware of "givenName" and "familyName" but instead of a person it might be an organization such as "Gallup Organization". - //author.add("@type", "Person"); - author.add("name", name); - // We are aware that the following error is thrown by https://search.google.com/structured-data/testing-tool - // "The property affiliation is not recognized by Google for an object of type Thing." - // Someone at Google has said this is ok. - // This logic could be moved into the `if (authorAffiliation != null)` block above. - if (!StringUtil.isEmpty(affiliation)) { - author.add("affiliation", affiliation); - } - String identifierAsUrl = datasetAuthor.getIdentifierAsUrl(); - if (identifierAsUrl != null) { - // It would be valid to provide an array of identifiers for authors but we have decided to only provide one. - author.add("@id", identifierAsUrl); - author.add("identifier", identifierAsUrl); + JsonObject entity = PersonOrOrgUtil.getPersonOrOrganization(name, false, (identifierAsUrl!=null)); + String givenName= entity.containsKey("givenName") ? entity.getString("givenName"):null; + String familyName= entity.containsKey("familyName") ? entity.getString("familyName"):null; + + if (entity.getBoolean("isPerson")) { + // Person + author.add("@type", "Person"); + if (givenName != null) { + author.add("givenName", givenName); + } + if (familyName != null) { + author.add("familyName", familyName); + } + if (!StringUtil.isEmpty(affiliation)) { + author.add("affiliation", Json.createObjectBuilder().add("@type", "Organization").add("name", affiliation)); + } + //Currently all possible identifier URLs are for people not Organizations + if(identifierAsUrl != null) { + author.add("sameAs", identifierAsUrl); + //Legacy - not sure if these are still useful + author.add("@id", identifierAsUrl); + author.add("identifier", identifierAsUrl); + + } + } else { + // Organization + author.add("@type", "Organization"); + if (!StringUtil.isEmpty(affiliation)) { + author.add("parentOrganization", Json.createObjectBuilder().add("@type", "Organization").add("name", affiliation)); + } } + // Both cases + author.add("name", entity.getString("fullName")); + //And add to the array authors.add(author); } JsonArray authorsArray = authors.build(); @@ -1858,16 +1892,8 @@ public String getJsonLd() { job.add("dateModified", this.getPublicationDateAsString()); job.add("version", this.getVersionNumber().toString()); - JsonArrayBuilder descriptionsArray = Json.createArrayBuilder(); - List descriptions = this.getDescriptionsPlainText(); - for (String description : descriptions) { - descriptionsArray.add(description); - } - /** - * In Dataverse 4.8.4 "description" was a single string but now it's an - * array. - */ - job.add("description", descriptionsArray); + String description = this.getDescriptionsPlainTextTruncated(); + job.add("description", description); /** * "keywords" - contains subject(s), datasetkeyword(s) and topicclassification(s) @@ -1891,11 +1917,16 @@ public String getJsonLd() { job.add("keywords", keywords); /** - * citation: (multiple) related publication citation and URLs, if - * present. + * citation: (multiple) related publication citation and URLs, if present. * - * In Dataverse 4.8.4 "citation" was an array of strings but now it's an - * array of objects. + * Schema.org allows text or a CreativeWork object. Google recommends text with + * either the full citation or the PID URL. This code adds an object if we have + * the citation text for the work and/or an entry in the URL field (i.e. + * https://doi.org/...) The URL is reported as the 'url' field while the + * citation text (which would normally include the name) is reported as 'name' + * since there doesn't appear to be a better field ('text', which was used + * previously, is the actual text of the creative work). + * */ List relatedPublications = getRelatedPublications(); if (!relatedPublications.isEmpty()) { @@ -1910,11 +1941,12 @@ public String getJsonLd() { JsonObjectBuilder citationEntry = Json.createObjectBuilder(); citationEntry.add("@type", "CreativeWork"); if (pubCitation != null) { - citationEntry.add("text", pubCitation); + citationEntry.add("name", pubCitation); } if (pubUrl != null) { citationEntry.add("@id", pubUrl); citationEntry.add("identifier", pubUrl); + citationEntry.add("url", pubUrl); } if (addToArray) { jsonArrayBuilder.add(citationEntry); @@ -1956,13 +1988,14 @@ public String getJsonLd() { job.add("license",DatasetUtil.getLicenseURI(this)); } + String installationBrandName = BrandingUtil.getInstallationBrandName(); + job.add("includedInDataCatalog", Json.createObjectBuilder() .add("@type", "DataCatalog") - .add("name", BrandingUtil.getRootDataverseCollectionName()) + .add("name", installationBrandName) .add("url", SystemConfig.getDataverseSiteUrlStatic()) ); - - String installationBrandName = BrandingUtil.getInstallationBrandName(); + /** * Both "publisher" and "provider" are included but they have the same * values. Some services seem to prefer one over the other. @@ -2011,7 +2044,7 @@ public String getJsonLd() { } fileObject.add("@type", "DataDownload"); fileObject.add("name", fileMetadata.getLabel()); - fileObject.add("fileFormat", fileMetadata.getDataFile().getContentType()); + fileObject.add("encodingFormat", fileMetadata.getDataFile().getContentType()); fileObject.add("contentSize", fileMetadata.getDataFile().getFilesize()); fileObject.add("description", fileMetadata.getDescription()); fileObject.add("@id", filePidUrlAsString); @@ -2020,10 +2053,8 @@ public String getJsonLd() { if (hideFilesBoolean != null && hideFilesBoolean.equals("true")) { // no-op } else { - if (FileUtil.isPubliclyDownloadable(fileMetadata)) { - String nullDownloadType = null; - fileObject.add("contentUrl", dataverseSiteUrl + FileUtil.getFileDownloadUrlPath(nullDownloadType, fileMetadata.getDataFile().getId(), false, fileMetadata.getId())); - } + String nullDownloadType = null; + fileObject.add("contentUrl", dataverseSiteUrl + FileUtil.getFileDownloadUrlPath(nullDownloadType, fileMetadata.getDataFile().getId(), false, fileMetadata.getId())); } fileArray.add(fileObject); } diff --git a/src/main/java/edu/harvard/iq/dataverse/Dataverse.java b/src/main/java/edu/harvard/iq/dataverse/Dataverse.java index 7443eab07dd..b686a4e5e40 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Dataverse.java +++ b/src/main/java/edu/harvard/iq/dataverse/Dataverse.java @@ -57,16 +57,16 @@ }) @Entity @Table(indexes = {@Index(columnList="defaultcontributorrole_id") - , @Index(columnList="defaulttemplate_id") - , @Index(columnList="alias") - , @Index(columnList="affiliation") - , @Index(columnList="dataversetype") - , @Index(columnList="facetroot") - , @Index(columnList="guestbookroot") - , @Index(columnList="metadatablockroot") - , @Index(columnList="templateroot") - , @Index(columnList="permissionroot") - , @Index(columnList="themeroot")}) + , @Index(columnList="defaulttemplate_id") + , @Index(columnList="alias") + , @Index(columnList="affiliation") + , @Index(columnList="dataversetype") + , @Index(columnList="facetroot") + , @Index(columnList="guestbookroot") + , @Index(columnList="metadatablockroot") + , @Index(columnList="templateroot") + , @Index(columnList="permissionroot") + , @Index(columnList="themeroot")}) public class Dataverse extends DvObjectContainer { public enum DataverseType { @@ -156,7 +156,7 @@ public String getIndexableCategoryName() { ///private String storageDriver=null; - // Note: We can't have "Remove" here, as there are role assignments that refer + // Note: We can't have "Remove" here, as there are role assignments that refer // to this role. So, adding it would mean violating a forign key contstraint. @OneToMany(cascade = {CascadeType.MERGE}, fetch = FetchType.LAZY, diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseFacet.java b/src/main/java/edu/harvard/iq/dataverse/DataverseFacet.java index 768c2308e50..76a2959d64b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseFacet.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseFacet.java @@ -24,16 +24,16 @@ * @author gdurand */ @NamedQueries({ - @NamedQuery( name="DataverseFacet.removeByOwnerId", - query="DELETE FROM DataverseFacet f WHERE f.dataverse.id=:ownerId"), + @NamedQuery( name="DataverseFacet.removeByOwnerId", + query="DELETE FROM DataverseFacet f WHERE f.dataverse.id=:ownerId"), @NamedQuery( name="DataverseFacet.findByDataverseId", query="select f from DataverseFacet f where f.dataverse.id = :dataverseId order by f.displayOrder") }) @Entity @Table(indexes = {@Index(columnList="dataverse_id") - , @Index(columnList="datasetfieldtype_id") - , @Index(columnList="displayorder")}) + , @Index(columnList="datasetfieldtype_id") + , @Index(columnList="displayorder")}) public class DataverseFacet implements Serializable { private static final long serialVersionUID = 1L; diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseFacetServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataverseFacetServiceBean.java index 5c77989f6d6..c97f7b142b5 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseFacetServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseFacetServiceBean.java @@ -42,14 +42,14 @@ public void delete(DataverseFacet dataverseFacet) { cache.invalidate(); } - public void deleteFacetsFor( Dataverse d ) { - em.createNamedQuery("DataverseFacet.removeByOwnerId") - .setParameter("ownerId", d.getId()) - .executeUpdate(); + public void deleteFacetsFor( Dataverse d ) { + em.createNamedQuery("DataverseFacet.removeByOwnerId") + .setParameter("ownerId", d.getId()) + .executeUpdate(); cache.invalidate(d.getId()); - } - + } + public DataverseFacet create(int displayOrder, DatasetFieldType fieldType, Dataverse ownerDv) { DataverseFacet dataverseFacet = new DataverseFacet(); diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseFeaturedDataverse.java b/src/main/java/edu/harvard/iq/dataverse/DataverseFeaturedDataverse.java index 39ad6ca9520..319f1abe3ce 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseFeaturedDataverse.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseFeaturedDataverse.java @@ -19,14 +19,14 @@ */ @NamedQueries({ - @NamedQuery( name="DataverseFeaturedDataverse.removeByOwnerId", - query="DELETE FROM DataverseFeaturedDataverse f WHERE f.dataverse.id=:ownerId") + @NamedQuery( name="DataverseFeaturedDataverse.removeByOwnerId", + query="DELETE FROM DataverseFeaturedDataverse f WHERE f.dataverse.id=:ownerId") }) @Entity @Table(indexes = {@Index(columnList="dataverse_id") - , @Index(columnList="featureddataverse_id") - , @Index(columnList="displayorder")}) + , @Index(columnList="featureddataverse_id") + , @Index(columnList="displayorder")}) public class DataverseFeaturedDataverse implements Serializable { private static final long serialVersionUID = 1L; diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseFieldTypeInputLevel.java b/src/main/java/edu/harvard/iq/dataverse/DataverseFieldTypeInputLevel.java index c4749be0cb3..cf1f9a26c94 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseFieldTypeInputLevel.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseFieldTypeInputLevel.java @@ -37,8 +37,8 @@ , uniqueConstraints={ @UniqueConstraint(columnNames={"dataverse_id", "datasetfieldtype_id"})} , indexes = {@Index(columnList="dataverse_id") - , @Index(columnList="datasetfieldtype_id") - , @Index(columnList="required")} + , @Index(columnList="datasetfieldtype_id") + , @Index(columnList="required")} ) @Entity public class DataverseFieldTypeInputLevel implements Serializable { diff --git a/src/main/java/edu/harvard/iq/dataverse/DataversePage.java b/src/main/java/edu/harvard/iq/dataverse/DataversePage.java index daf33f444d9..d356ad1e4fd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataversePage.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataversePage.java @@ -1199,35 +1199,35 @@ public List completeHostDataverseMenuList(String query) { } public Set> getStorageDriverOptions() { - HashMap drivers =new HashMap(); - drivers.putAll(DataAccess.getStorageDriverLabels()); - //Add an entry for the default (inherited from an ancestor or the system default) - drivers.put(getDefaultStorageDriverLabel(), DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER); - return drivers.entrySet(); + HashMap drivers =new HashMap(); + drivers.putAll(DataAccess.getStorageDriverLabels()); + //Add an entry for the default (inherited from an ancestor or the system default) + drivers.put(getDefaultStorageDriverLabel(), DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER); + return drivers.entrySet(); } public String getDefaultStorageDriverLabel() { - String storageDriverId = DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER; - Dataverse parent = dataverse.getOwner(); - boolean fromAncestor=false; - if(parent != null) { - storageDriverId = parent.getEffectiveStorageDriverId(); - //recurse dataverse chain to root and if any have a storagedriver set, fromAncestor is true - while(parent!=null) { - if(!parent.getStorageDriverId().equals(DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER)) { - fromAncestor=true; - break; - } - parent=parent.getOwner(); - } - } - String label = DataAccess.getStorageDriverLabelFor(storageDriverId); - if(fromAncestor) { - label = label + " " + BundleUtil.getStringFromBundle("dataverse.inherited"); - } else { - label = label + " " + BundleUtil.getStringFromBundle("dataverse.default"); - } - return label; + String storageDriverId = DataAccess.DEFAULT_STORAGE_DRIVER_IDENTIFIER; + Dataverse parent = dataverse.getOwner(); + boolean fromAncestor=false; + if(parent != null) { + storageDriverId = parent.getEffectiveStorageDriverId(); + //recurse dataverse chain to root and if any have a storagedriver set, fromAncestor is true + while(parent!=null) { + if(!parent.getStorageDriverId().equals(DataAccess.UNDEFINED_STORAGE_DRIVER_IDENTIFIER)) { + fromAncestor=true; + break; + } + parent=parent.getOwner(); + } + } + String label = DataAccess.getStorageDriverLabelFor(storageDriverId); + if(fromAncestor) { + label = label + " " + BundleUtil.getStringFromBundle("dataverse.inherited"); + } else { + label = label + " " + BundleUtil.getStringFromBundle("dataverse.default"); + } + return label; } public Set> getMetadataLanguages() { diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java index 7194a1ef31e..4096e0d7fa2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseServiceBean.java @@ -253,13 +253,13 @@ public Dataverse findByAlias(String anAlias) { } } - public boolean hasData( Dataverse dv ) { - TypedQuery amountQry = em.createNamedQuery("Dataverse.ownedObjectsById", Long.class) - .setParameter("id", dv.getId()); - - return (amountQry.getSingleResult()>0); - } - + public boolean hasData( Dataverse dv ) { + TypedQuery amountQry = em.createNamedQuery("Dataverse.ownedObjectsById", Long.class) + .setParameter("id", dv.getId()); + + return (amountQry.getSingleResult()>0); + } + public boolean isRootDataverseExists() { long count = em.createQuery("SELECT count(dv) FROM Dataverse dv WHERE dv.owner.id=null", Long.class).getSingleResult(); return (count == 1); diff --git a/src/main/java/edu/harvard/iq/dataverse/DataverseSession.java b/src/main/java/edu/harvard/iq/dataverse/DataverseSession.java index e8d76e1825e..8b2e8f8aad9 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DataverseSession.java +++ b/src/main/java/edu/harvard/iq/dataverse/DataverseSession.java @@ -42,7 +42,7 @@ public class DataverseSession implements Serializable{ @EJB BuiltinUserServiceBean usersSvc; - + @EJB ActionLogServiceBean logSvc; @@ -137,11 +137,11 @@ public void setUser(User aUser) { return; } FacesContext context = FacesContext.getCurrentInstance(); - // Log the login/logout and Change the session id if we're using the UI and have - // a session, versus an API call with no session - (i.e. /admin/submitToArchive() - // which sets the user in the session to pass it through to the underlying command) + // Log the login/logout and Change the session id if we're using the UI and have + // a session, versus an API call with no session - (i.e. /admin/submitToArchive() + // which sets the user in the session to pass it through to the underlying command) // TODO: reformat to remove tabs etc. - if(context != null) { + if(context != null) { logSvc.log( new ActionLogRecord(ActionLogRecord.ActionType.SessionManagement,(aUser==null) ? "logout" : "login") .setUserIdentifier((aUser!=null) ? aUser.getIdentifier() : (user!=null ? user.getIdentifier() : "") )); diff --git a/src/main/java/edu/harvard/iq/dataverse/DvObject.java b/src/main/java/edu/harvard/iq/dataverse/DvObject.java index 998750ad01d..3ad60f3d889 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DvObject.java +++ b/src/main/java/edu/harvard/iq/dataverse/DvObject.java @@ -23,7 +23,7 @@ @NamedQuery(name = "DvObject.checkExists", query = "SELECT count(o) from DvObject o WHERE o.id=:id"), @NamedQuery(name = "DvObject.ownedObjectsById", - query="SELECT COUNT(obj) FROM DvObject obj WHERE obj.owner.id=:id"), + query="SELECT COUNT(obj) FROM DvObject obj WHERE obj.owner.id=:id"), @NamedQuery(name = "DvObject.findByGlobalId", query = "SELECT o FROM DvObject o WHERE o.identifier=:identifier and o.authority=:authority and o.protocol=:protocol and o.dtype=:dtype"), @@ -45,10 +45,10 @@ // child tables). Tested, appears to be working properly. -- L.A. Nov. 4 2014 @Inheritance(strategy=InheritanceType.JOINED) @Table(indexes = {@Index(columnList="dtype") - , @Index(columnList="owner_id") - , @Index(columnList="creator_id") - , @Index(columnList="releaseuser_id")}, - uniqueConstraints = {@UniqueConstraint(columnNames = {"authority,protocol,identifier"}),@UniqueConstraint(columnNames = {"owner_id,storageidentifier"})}) + , @Index(columnList="owner_id") + , @Index(columnList="creator_id") + , @Index(columnList="releaseuser_id")}, + uniqueConstraints = {@UniqueConstraint(columnNames = {"authority,protocol,identifier"}),@UniqueConstraint(columnNames = {"owner_id,storageidentifier"})}) public abstract class DvObject extends DataverseEntity implements java.io.Serializable { public static final String DATAVERSE_DTYPE_STRING = "Dataverse"; diff --git a/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java b/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java index a322a25103e..22adde09010 100644 --- a/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java +++ b/src/main/java/edu/harvard/iq/dataverse/DvObjectContainer.java @@ -12,7 +12,7 @@ */ @MappedSuperclass public abstract class DvObjectContainer extends DvObject { - + public static final String UNDEFINED_METADATA_LANGUAGE_CODE = "undefined"; //Used in dataverse.xhtml as a non-null selection option value (indicating inheriting the default) @@ -20,11 +20,11 @@ public abstract class DvObjectContainer extends DvObject { public void setOwner(Dataverse owner) { super.setOwner(owner); } - - @Override - public Dataverse getOwner() { - return super.getOwner()!=null ? (Dataverse)super.getOwner() : null; - } + + @Override + public Dataverse getOwner() { + return super.getOwner()!=null ? (Dataverse)super.getOwner() : null; + } protected abstract boolean isPermissionRoot(); diff --git a/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java b/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java index 770e31550f0..fa4a3ede971 100644 --- a/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/EditDatafilesPage.java @@ -33,6 +33,7 @@ import edu.harvard.iq.dataverse.ingest.IngestUtil; import edu.harvard.iq.dataverse.license.LicenseServiceBean; import edu.harvard.iq.dataverse.search.IndexServiceBean; +import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.settings.Setting; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.FileUtil; @@ -2427,10 +2428,8 @@ public boolean isTemporaryPreviewAvailable(String fileSystemId, String mimeType) return false; } - String filesRootDirectory = System.getProperty("dataverse.files.directory"); - if (filesRootDirectory == null || filesRootDirectory.isEmpty()) { - filesRootDirectory = "/tmp/files"; - } + // Retrieve via MPCONFIG. Has sane default /tmp/dataverse from META-INF/microprofile-config.properties + String filesRootDirectory = JvmSettings.FILES_DIRECTORY.lookup(); String fileSystemName = filesRootDirectory + "/temp/" + fileSystemId; diff --git a/src/main/java/edu/harvard/iq/dataverse/EjbDataverseEngine.java b/src/main/java/edu/harvard/iq/dataverse/EjbDataverseEngine.java index 0e565ce5dde..d5f41531475 100644 --- a/src/main/java/edu/harvard/iq/dataverse/EjbDataverseEngine.java +++ b/src/main/java/edu/harvard/iq/dataverse/EjbDataverseEngine.java @@ -374,7 +374,7 @@ public void addCommand(Command command) { logger.fine("Exception logging command stack(" + instance + "): " + e.getMessage()); } } - commandsCalled.push(command); + commandsCalled.push(command); } diff --git a/src/main/java/edu/harvard/iq/dataverse/FeaturedDataverseServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/FeaturedDataverseServiceBean.java index d4d701cb02f..03d9de23f54 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FeaturedDataverseServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/FeaturedDataverseServiceBean.java @@ -83,11 +83,11 @@ public void delete(DataverseFeaturedDataverse dataverseFeaturedDataverse) { em.remove(em.merge(dataverseFeaturedDataverse)); } - public void deleteFeaturedDataversesFor( Dataverse d ) { - em.createNamedQuery("DataverseFeaturedDataverse.removeByOwnerId") - .setParameter("ownerId", d.getId()) - .executeUpdate(); - } + public void deleteFeaturedDataversesFor( Dataverse d ) { + em.createNamedQuery("DataverseFeaturedDataverse.removeByOwnerId") + .setParameter("ownerId", d.getId()) + .executeUpdate(); + } public void create(int diplayOrder, Long featuredDataverseId, Long dataverseId) { DataverseFeaturedDataverse dataverseFeaturedDataverse = new DataverseFeaturedDataverse(); @@ -101,7 +101,7 @@ public void create(int diplayOrder, Long featuredDataverseId, Long dataverseId) dataverseFeaturedDataverse.setFeaturedDataverse(featuredDataverse); em.persist(dataverseFeaturedDataverse); - } + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java index 441395bd918..e0b7acefc0f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/FileDownloadServiceBean.java @@ -357,9 +357,9 @@ public void downloadDirectDatafileCitationXML(FileMetadata fileMetadata) { } public void downloadCitationXML(FileMetadata fileMetadata, Dataset dataset, boolean direct) { - DataCitation citation=null; + DataCitation citation=null; if (dataset != null){ - citation = new DataCitation(dataset.getLatestVersion()); + citation = new DataCitation(dataset.getLatestVersion()); } else { citation= new DataCitation(fileMetadata, direct); } @@ -400,9 +400,9 @@ public void downloadDirectDatafileCitationRIS(FileMetadata fileMetadata) { } public void downloadCitationRIS(FileMetadata fileMetadata, Dataset dataset, boolean direct) { - DataCitation citation=null; + DataCitation citation=null; if (dataset != null){ - citation = new DataCitation(dataset.getLatestVersion()); + citation = new DataCitation(dataset.getLatestVersion()); } else { citation= new DataCitation(fileMetadata, direct); } @@ -450,9 +450,9 @@ public void downloadDirectDatafileCitationBibtex(FileMetadata fileMetadata) { } public void downloadCitationBibtex(FileMetadata fileMetadata, Dataset dataset, boolean direct) { - DataCitation citation=null; + DataCitation citation=null; if (dataset != null){ - citation = new DataCitation(dataset.getLatestVersion()); + citation = new DataCitation(dataset.getLatestVersion()); } else { citation= new DataCitation(fileMetadata, direct); } diff --git a/src/main/java/edu/harvard/iq/dataverse/FileMetadata.java b/src/main/java/edu/harvard/iq/dataverse/FileMetadata.java index b32508369f3..ed162574e53 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FileMetadata.java +++ b/src/main/java/edu/harvard/iq/dataverse/FileMetadata.java @@ -237,7 +237,6 @@ public List getCategoriesByName() { return ret; } - public JsonArrayBuilder getCategoryNamesAsJsonArrayBuilder() { JsonArrayBuilder builder = Json.createArrayBuilder(); diff --git a/src/main/java/edu/harvard/iq/dataverse/FilePage.java b/src/main/java/edu/harvard/iq/dataverse/FilePage.java index 8c61cf7da9c..a9e344a1b69 100644 --- a/src/main/java/edu/harvard/iq/dataverse/FilePage.java +++ b/src/main/java/edu/harvard/iq/dataverse/FilePage.java @@ -120,6 +120,8 @@ public class FilePage implements java.io.Serializable { ExternalToolServiceBean externalToolService; @EJB PrivateUrlServiceBean privateUrlService; + @EJB + AuxiliaryFileServiceBean auxiliaryFileService; @Inject DataverseRequestServiceBean dvRequestService; @@ -280,8 +282,15 @@ public void setDatasetVersionId(Long datasetVersionId) { this.datasetVersionId = datasetVersionId; } + // findPreviewTools would be a better name private List sortExternalTools(){ - List retList = externalToolService.findFileToolsByTypeAndContentType(ExternalTool.Type.PREVIEW, file.getContentType()); + List retList = new ArrayList<>(); + List previewTools = externalToolService.findFileToolsByTypeAndContentType(ExternalTool.Type.PREVIEW, file.getContentType()); + for (ExternalTool previewTool : previewTools) { + if (externalToolService.meetsRequirements(previewTool, file)) { + retList.add(previewTool); + } + } Collections.sort(retList, CompareExternalToolName); return retList; } diff --git a/src/main/java/edu/harvard/iq/dataverse/ForeignMetadataFieldMapping.java b/src/main/java/edu/harvard/iq/dataverse/ForeignMetadataFieldMapping.java index db83ab953a1..8e4b9ca3bf7 100644 --- a/src/main/java/edu/harvard/iq/dataverse/ForeignMetadataFieldMapping.java +++ b/src/main/java/edu/harvard/iq/dataverse/ForeignMetadataFieldMapping.java @@ -12,8 +12,8 @@ */ @Table( uniqueConstraints = @UniqueConstraint(columnNames={"foreignMetadataFormatMapping_id","foreignFieldXpath"}) , indexes = {@Index(columnList="foreignmetadataformatmapping_id") - , @Index(columnList="foreignfieldxpath") - , @Index(columnList="parentfieldmapping_id")}) + , @Index(columnList="foreignfieldxpath") + , @Index(columnList="parentfieldmapping_id")}) @NamedQueries({ @NamedQuery( name="ForeignMetadataFieldMapping.findByPath", query="SELECT fmfm FROM ForeignMetadataFieldMapping fmfm WHERE fmfm.foreignMetadataFormatMapping.name=:formatName AND fmfm.foreignFieldXPath=:xPath") diff --git a/src/main/java/edu/harvard/iq/dataverse/GlobalId.java b/src/main/java/edu/harvard/iq/dataverse/GlobalId.java index e135f493fcb..564422a3a2b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/GlobalId.java +++ b/src/main/java/edu/harvard/iq/dataverse/GlobalId.java @@ -207,16 +207,16 @@ private static String formatIdentifierString(String str){ return str.replaceAll("\\s+|'|;",""); /* - < (%3C) -> (%3E) -{ (%7B) -} (%7D) -^ (%5E) -[ (%5B) -] (%5D) -` (%60) -| (%7C) -\ (%5C) + < (%3C) +> (%3E) +{ (%7B) +} (%7D) +^ (%5E) +[ (%5B) +] (%5D) +` (%60) +| (%7C) +\ (%5C) + */ // http://www.doi.org/doi_handbook/2_Numbering.html diff --git a/src/main/java/edu/harvard/iq/dataverse/HarvestingClientsPage.java b/src/main/java/edu/harvard/iq/dataverse/HarvestingClientsPage.java index e9cb2886388..f008db1403f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/HarvestingClientsPage.java +++ b/src/main/java/edu/harvard/iq/dataverse/HarvestingClientsPage.java @@ -77,7 +77,7 @@ public class HarvestingClientsPage implements java.io.Serializable { private Dataverse dataverse; private Long dataverseId = null; private HarvestingClient selectedClient; - private boolean setListTruncated = false; + private boolean setListTruncated = false; //private static final String solrDocIdentifierDataset = "dataset_"; @@ -243,6 +243,7 @@ public void editClient(HarvestingClient harvestingClient) { this.newNickname = harvestingClient.getName(); this.newHarvestingUrl = harvestingClient.getHarvestingUrl(); + this.customHeader = harvestingClient.getCustomHttpHeaders(); this.initialSettingsValidated = false; // TODO: do we want to try and contact the server, again, to make @@ -338,6 +339,7 @@ public void createClient(ActionEvent ae) { getSelectedDestinationDataverse().getHarvestingClientConfigs().add(newHarvestingClient); newHarvestingClient.setHarvestingUrl(newHarvestingUrl); + newHarvestingClient.setCustomHttpHeaders(customHeader); if (!StringUtils.isEmpty(newOaiSet)) { newHarvestingClient.setHarvestingSet(newOaiSet); } @@ -424,6 +426,7 @@ public void saveClient(ActionEvent ae) { // nickname is not editable for existing clients: //harvestingClient.setName(newNickname); harvestingClient.setHarvestingUrl(newHarvestingUrl); + harvestingClient.setCustomHttpHeaders(customHeader); harvestingClient.setHarvestingSet(newOaiSet); harvestingClient.setMetadataPrefix(newMetadataFormat); harvestingClient.setHarvestStyle(newHarvestingStyle); @@ -552,6 +555,9 @@ public boolean validateServerUrlOAI() { if (!StringUtils.isEmpty(getNewHarvestingUrl())) { OaiHandler oaiHandler = new OaiHandler(getNewHarvestingUrl()); + if (getNewCustomHeader() != null) { + oaiHandler.setCustomHeaders(oaiHandler.makeCustomHeaders(getNewCustomHeader())); + } boolean success = true; String message = null; @@ -633,6 +639,23 @@ public boolean validateServerUrlOAI() { return false; } + public boolean validateCustomHeader() { + if (!StringUtils.isEmpty(getNewCustomHeader())) { + // TODO: put this method somewhere else as a static utility + + // check that it's looking like "{header-name}: {header value}" at least + if (!Pattern.matches("^[a-zA-Z0-9\\_\\-]+:.*",getNewCustomHeader())) { + FacesContext.getCurrentInstance().addMessage(getNewClientCustomHeaderInputField().getClientId(), + new FacesMessage(FacesMessage.SEVERITY_ERROR, "", BundleUtil.getStringFromBundle("harvestclients.newClientDialog.customHeader.invalid"))); + + return false; + } + } + + // this setting is optional + return true; + } + public void validateInitialSettings() { if (isHarvestTypeOAI()) { boolean nicknameValidated = true; @@ -642,9 +665,10 @@ public void validateInitialSettings() { destinationDataverseValidated = validateSelectedDataverse(); } boolean urlValidated = validateServerUrlOAI(); + boolean customHeaderValidated = validateCustomHeader(); - if (nicknameValidated && destinationDataverseValidated && urlValidated) { - // In Create mode we want to run all 3 validation tests; this is why + if (nicknameValidated && destinationDataverseValidated && urlValidated && customHeaderValidated) { + // In Create mode we want to run all 4 validation tests; this is why // we are not doing "if ((validateNickname() && validateServerUrlOAI())" // in the line above. -- L.A. 4.4 May 2016. @@ -686,6 +710,7 @@ public void backToStepThree() { UIInput newClientNicknameInputField; UIInput newClientUrlInputField; + UIInput newClientCustomHeaderInputField; UIInput hiddenInputField; /*UISelectOne*/ UIInput metadataFormatMenu; UIInput remoteArchiveStyleMenu; @@ -693,6 +718,7 @@ public void backToStepThree() { private String newNickname = ""; private String newHarvestingUrl = ""; + private String customHeader = null; private boolean initialSettingsValidated = false; private String newOaiSet = ""; private String newMetadataFormat = ""; @@ -716,6 +742,7 @@ public void initNewClient(ActionEvent ae) { //this.selectedClient = new HarvestingClient(); this.newNickname = ""; this.newHarvestingUrl = ""; + this.customHeader = null; this.initialSettingsValidated = false; this.newOaiSet = ""; this.newMetadataFormat = ""; @@ -760,6 +787,14 @@ public void setNewHarvestingUrl(String newHarvestingUrl) { this.newHarvestingUrl = newHarvestingUrl; } + public String getNewCustomHeader() { + return customHeader; + } + + public void setNewCustomHeader(String customHeader) { + this.customHeader = customHeader; + } + public int getHarvestTypeRadio() { return this.harvestTypeRadio; } @@ -869,6 +904,14 @@ public void setNewClientUrlInputField(UIInput newClientInputField) { this.newClientUrlInputField = newClientInputField; } + public UIInput getNewClientCustomHeaderInputField() { + return newClientCustomHeaderInputField; + } + + public void setNewClientCustomHeaderInputField(UIInput newClientInputField) { + this.newClientCustomHeaderInputField = newClientInputField; + } + public UIInput getHiddenInputField() { return hiddenInputField; } diff --git a/src/main/java/edu/harvard/iq/dataverse/HarvestingDataverseConfig.java b/src/main/java/edu/harvard/iq/dataverse/HarvestingDataverseConfig.java index 6709b978c47..dcdd399f03b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/HarvestingDataverseConfig.java +++ b/src/main/java/edu/harvard/iq/dataverse/HarvestingDataverseConfig.java @@ -23,9 +23,9 @@ */ @Entity @Table(indexes = {@Index(columnList="dataverse_id") - , @Index(columnList="harvesttype") - , @Index(columnList="harveststyle") - , @Index(columnList="harvestingurl")}) + , @Index(columnList="harvesttype") + , @Index(columnList="harveststyle") + , @Index(columnList="harvestingurl")}) public class HarvestingDataverseConfig implements Serializable { private static final long serialVersionUID = 1L; @Id diff --git a/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java b/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java index 0fd7c2efbc7..1bcc7bf5bcd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java +++ b/src/main/java/edu/harvard/iq/dataverse/MetadataBlock.java @@ -29,7 +29,7 @@ * @author skraffmiller */ @Table(indexes = {@Index(columnList="name") - , @Index(columnList="owner_id")}) + , @Index(columnList="owner_id")}) @NamedQueries({ @NamedQuery( name="MetadataBlock.listAll", query = "SELECT mdb FROM MetadataBlock mdb"), @NamedQuery( name="MetadataBlock.findByName", query = "SELECT mdb FROM MetadataBlock mdb WHERE mdb.name=:name") diff --git a/src/main/java/edu/harvard/iq/dataverse/RoleAssignment.java b/src/main/java/edu/harvard/iq/dataverse/RoleAssignment.java index df004fe1357..c534bc9d8f6 100644 --- a/src/main/java/edu/harvard/iq/dataverse/RoleAssignment.java +++ b/src/main/java/edu/harvard/iq/dataverse/RoleAssignment.java @@ -25,76 +25,76 @@ */ @Entity @Table( - uniqueConstraints = @UniqueConstraint(columnNames={"assigneeIdentifier","role_id","definitionPoint_id"}) + uniqueConstraints = @UniqueConstraint(columnNames={"assigneeIdentifier","role_id","definitionPoint_id"}) , indexes = {@Index(columnList="assigneeidentifier") - , @Index(columnList="definitionpoint_id") - , @Index(columnList="role_id")} + , @Index(columnList="definitionpoint_id") + , @Index(columnList="role_id")} ) @NamedQueries({ - @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifier_DefinitionPointId", - query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.definitionPoint.id=:definitionPointId" ), - @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifier_DefinitionPointId_RoleId", - query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.definitionPoint.id=:definitionPointId and r.role.id=:roleId" ), - @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifier", - query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier" ), - @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifiers", - query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier in :assigneeIdentifiers AND r.definitionPoint.id in :definitionPointIds" ), - @NamedQuery( name = "RoleAssignment.listByDefinitionPointId", - query = "SELECT r FROM RoleAssignment r WHERE r.definitionPoint.id=:definitionPointId" ), - @NamedQuery( name = "RoleAssignment.listByRoleId", - query = "SELECT r FROM RoleAssignment r WHERE r.role.id=:roleId" ), - @NamedQuery( name = "RoleAssignment.listByPrivateUrlToken", - query = "SELECT r FROM RoleAssignment r WHERE r.privateUrlToken=:privateUrlToken" ), - @NamedQuery( name = "RoleAssignment.deleteByAssigneeIdentifier_RoleIdDefinition_PointId", - query = "DELETE FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.role.id=:roleId AND r.definitionPoint.id=:definitionPointId"), + @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifier_DefinitionPointId", + query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.definitionPoint.id=:definitionPointId" ), + @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifier_DefinitionPointId_RoleId", + query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.definitionPoint.id=:definitionPointId and r.role.id=:roleId" ), + @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifier", + query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier" ), + @NamedQuery( name = "RoleAssignment.listByAssigneeIdentifiers", + query = "SELECT r FROM RoleAssignment r WHERE r.assigneeIdentifier in :assigneeIdentifiers AND r.definitionPoint.id in :definitionPointIds" ), + @NamedQuery( name = "RoleAssignment.listByDefinitionPointId", + query = "SELECT r FROM RoleAssignment r WHERE r.definitionPoint.id=:definitionPointId" ), + @NamedQuery( name = "RoleAssignment.listByRoleId", + query = "SELECT r FROM RoleAssignment r WHERE r.role.id=:roleId" ), + @NamedQuery( name = "RoleAssignment.listByPrivateUrlToken", + query = "SELECT r FROM RoleAssignment r WHERE r.privateUrlToken=:privateUrlToken" ), + @NamedQuery( name = "RoleAssignment.deleteByAssigneeIdentifier_RoleIdDefinition_PointId", + query = "DELETE FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.role.id=:roleId AND r.definitionPoint.id=:definitionPointId"), @NamedQuery( name = "RoleAssignment.deleteAllByAssigneeIdentifier", - query = "DELETE FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier"), + query = "DELETE FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier"), @NamedQuery( name = "RoleAssignment.deleteAllByAssigneeIdentifier_Definition_PointId_RoleType", - query = "DELETE FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.role.id=:roleId and r.definitionPoint.id=:definitionPointId") + query = "DELETE FROM RoleAssignment r WHERE r.assigneeIdentifier=:assigneeIdentifier AND r.role.id=:roleId and r.definitionPoint.id=:definitionPointId") }) public class RoleAssignment implements java.io.Serializable { - @Id - @GeneratedValue(strategy = GenerationType.IDENTITY) - private Long id; - - @Column( nullable=false ) - private String assigneeIdentifier; - - @ManyToOne( cascade = {CascadeType.MERGE} ) - @JoinColumn( nullable=false ) - private DataverseRole role; - - @ManyToOne( cascade = {CascadeType.MERGE} ) - @JoinColumn( nullable=false ) - private DvObject definitionPoint; + @Id + @GeneratedValue(strategy = GenerationType.IDENTITY) + private Long id; + + @Column( nullable=false ) + private String assigneeIdentifier; + + @ManyToOne( cascade = {CascadeType.MERGE} ) + @JoinColumn( nullable=false ) + private DataverseRole role; + + @ManyToOne( cascade = {CascadeType.MERGE} ) + @JoinColumn( nullable=false ) + private DvObject definitionPoint; @Column(nullable = true) private String privateUrlToken; - + @Column(nullable = true) private Boolean privateUrlAnonymizedAccess; - - public RoleAssignment() {} - - public RoleAssignment(DataverseRole aRole, RoleAssignee anAssignee, DvObject aDefinitionPoint, String privateUrlToken) { - this(aRole, anAssignee, aDefinitionPoint, privateUrlToken, false); - } - - public RoleAssignment(DataverseRole aRole, RoleAssignee anAssignee, DvObject aDefinitionPoint, String privateUrlToken, Boolean anonymizedAccess) { + + public RoleAssignment() {} + + public RoleAssignment(DataverseRole aRole, RoleAssignee anAssignee, DvObject aDefinitionPoint, String privateUrlToken) { + this(aRole, anAssignee, aDefinitionPoint, privateUrlToken, false); + } + + public RoleAssignment(DataverseRole aRole, RoleAssignee anAssignee, DvObject aDefinitionPoint, String privateUrlToken, Boolean anonymizedAccess) { role = aRole; assigneeIdentifier = anAssignee.getIdentifier(); definitionPoint = aDefinitionPoint; this.privateUrlToken = privateUrlToken; this.privateUrlAnonymizedAccess=anonymizedAccess; } - - public Long getId() { - return id; - } + + public Long getId() { + return id; + } - public void setId(Long id) { - this.id = id; - } + public void setId(Long id) { + this.id = id; + } public String getAssigneeIdentifier() { return assigneeIdentifier; @@ -104,21 +104,21 @@ public void setAssigneeIdentifier(String assigneeIdentifier) { this.assigneeIdentifier = assigneeIdentifier; } - public DataverseRole getRole() { - return role; - } + public DataverseRole getRole() { + return role; + } - public void setRole(DataverseRole role) { - this.role = role; - } + public void setRole(DataverseRole role) { + this.role = role; + } - public DvObject getDefinitionPoint() { - return definitionPoint; - } + public DvObject getDefinitionPoint() { + return definitionPoint; + } - public void setDefinitionPoint(DvObject definitionPoint) { - this.definitionPoint = definitionPoint; - } + public void setDefinitionPoint(DvObject definitionPoint) { + this.definitionPoint = definitionPoint; + } public String getPrivateUrlToken() { return privateUrlToken; @@ -128,33 +128,33 @@ public boolean isAnonymizedAccess(){ return (privateUrlAnonymizedAccess==null) ? false: privateUrlAnonymizedAccess; } - @Override - public int hashCode() { - int hash = 7; - hash = 97 * hash + Objects.hashCode(role); - hash = 97 * hash + Objects.hashCode(assigneeIdentifier); - return hash; - } + @Override + public int hashCode() { + int hash = 7; + hash = 97 * hash + Objects.hashCode(role); + hash = 97 * hash + Objects.hashCode(assigneeIdentifier); + return hash; + } - @Override - public boolean equals(Object obj) { - if (obj == null) { - return false; - } - if ( ! (obj instanceof RoleAssignment) ) { - return false; - } - final RoleAssignment other = (RoleAssignment) obj; - - return ( Objects.equals(getRole(), other.getRole() ) - && Objects.equals(getAssigneeIdentifier(), other.getAssigneeIdentifier()) - && Objects.equals(getDefinitionPoint(), other.getDefinitionPoint())); - - } + @Override + public boolean equals(Object obj) { + if (obj == null) { + return false; + } + if ( ! (obj instanceof RoleAssignment) ) { + return false; + } + final RoleAssignment other = (RoleAssignment) obj; + + return ( Objects.equals(getRole(), other.getRole() ) + && Objects.equals(getAssigneeIdentifier(), other.getAssigneeIdentifier()) + && Objects.equals(getDefinitionPoint(), other.getDefinitionPoint())); + + } - @Override - public String toString() { - return "RoleAssignment{" + "id=" + id + ", assignee=" + assigneeIdentifier + ", role=" + role + ", definitionPoint=" + definitionPoint + '}'; - } - + @Override + public String toString() { + return "RoleAssignment{" + "id=" + id + ", assignee=" + assigneeIdentifier + ", role=" + role + ", definitionPoint=" + definitionPoint + '}'; + } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/RolePermissionFragment.java b/src/main/java/edu/harvard/iq/dataverse/RolePermissionFragment.java index 67a9e3ee572..c14fbf84e8f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/RolePermissionFragment.java +++ b/src/main/java/edu/harvard/iq/dataverse/RolePermissionFragment.java @@ -179,7 +179,7 @@ public void toggleFileRestrict(ActionEvent evt) { public void grantAccess(ActionEvent evt) { //RoleAssignee assignRoleRoleAssignee = roleAssigneeService.getRoleAssignee(assignRoleUsername); - // Find the built in file downloader role (currently by alias) + // Find the built in file downloader role (currently by alias) assignRole(assignRoleRoleAssignee, roleService.findBuiltinRoleByAlias("fileDownloader")); } public void assignRole(ActionEvent evt) { diff --git a/src/main/java/edu/harvard/iq/dataverse/Shib.java b/src/main/java/edu/harvard/iq/dataverse/Shib.java index bee1182e248..70959d18d7a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Shib.java +++ b/src/main/java/edu/harvard/iq/dataverse/Shib.java @@ -56,8 +56,8 @@ public class Shib implements java.io.Serializable { UserNotificationServiceBean userNotificationService; @EJB SettingsServiceBean settingsService; - @EJB - SystemConfig systemConfig; + @EJB + SystemConfig systemConfig; HttpServletRequest request; @@ -454,9 +454,9 @@ private String getRequiredValueFromAssertion(String key) throws Exception { if (attributeValue.isEmpty()) { throw new Exception(key + " was empty"); } - if(systemConfig.isShibAttributeCharacterSetConversionEnabled()) { - attributeValue= new String( attributeValue.getBytes("ISO-8859-1"), "UTF-8"); - } + if(systemConfig.isShibAttributeCharacterSetConversionEnabled()) { + attributeValue= new String( attributeValue.getBytes("ISO-8859-1"), "UTF-8"); + } String trimmedValue = attributeValue.trim(); logger.fine("The SAML assertion for \"" + key + "\" (required) was \"" + attributeValue + "\" and was trimmed to \"" + trimmedValue + "\"."); return trimmedValue; diff --git a/src/main/java/edu/harvard/iq/dataverse/Template.java b/src/main/java/edu/harvard/iq/dataverse/Template.java index e8e74004750..39705934cfd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/Template.java +++ b/src/main/java/edu/harvard/iq/dataverse/Template.java @@ -9,6 +9,7 @@ import java.util.LinkedList; import java.util.List; import java.util.Map; +import java.util.TreeMap; import java.util.stream.Collectors; import jakarta.json.Json; @@ -139,9 +140,9 @@ public List getDatasetFields() { private Map instructionsMap = null; @Transient - private Map> metadataBlocksForView = new HashMap<>(); + private TreeMap> metadataBlocksForView = new TreeMap<>(); @Transient - private Map> metadataBlocksForEdit = new HashMap<>(); + private TreeMap> metadataBlocksForEdit = new TreeMap<>(); @Transient private boolean isDefaultForDataverse; @@ -166,19 +167,19 @@ public void setDataversesHasAsDefault(List dataversesHasAsDefault) { } - public Map> getMetadataBlocksForView() { + public TreeMap> getMetadataBlocksForView() { return metadataBlocksForView; } - public void setMetadataBlocksForView(Map> metadataBlocksForView) { + public void setMetadataBlocksForView(TreeMap> metadataBlocksForView) { this.metadataBlocksForView = metadataBlocksForView; } - public Map> getMetadataBlocksForEdit() { + public TreeMap> getMetadataBlocksForEdit() { return metadataBlocksForEdit; } - public void setMetadataBlocksForEdit(Map> metadataBlocksForEdit) { + public void setMetadataBlocksForEdit(TreeMap> metadataBlocksForEdit) { this.metadataBlocksForEdit = metadataBlocksForEdit; } diff --git a/src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java b/src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java index 8b4c797ad6c..e60843bf299 100644 --- a/src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java +++ b/src/main/java/edu/harvard/iq/dataverse/ThumbnailServiceWrapper.java @@ -28,10 +28,10 @@ * * @author Leonid Andreev */ -//@ViewScoped +// @ViewScoped @RequestScoped @Named -public class ThumbnailServiceWrapper implements java.io.Serializable { +public class ThumbnailServiceWrapper implements java.io.Serializable { @Inject PermissionsWrapper permissionsWrapper; @EJB @@ -42,7 +42,7 @@ public class ThumbnailServiceWrapper implements java.io.Serializable { DatasetVersionServiceBean datasetVersionService; @EJB DataFileServiceBean dataFileService; - + private Map dvobjectThumbnailsMap = new HashMap<>(); private Map dvobjectViewMap = new HashMap<>(); @@ -58,7 +58,7 @@ private String getAssignedDatasetImage(Dataset dataset, int size) { if (this.dvobjectThumbnailsMap.containsKey(assignedThumbnailFileId)) { // Yes, return previous answer - //logger.info("using cached result for ... "+assignedThumbnailFileId); + // logger.info("using cached result for ... "+assignedThumbnailFileId); if (!"".equals(this.dvobjectThumbnailsMap.get(assignedThumbnailFileId))) { return this.dvobjectThumbnailsMap.get(assignedThumbnailFileId); } @@ -67,19 +67,19 @@ private String getAssignedDatasetImage(Dataset dataset, int size) { String imageSourceBase64 = ImageThumbConverter.getImageThumbnailAsBase64(assignedThumbnailFile, size); - //ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); + // ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); if (imageSourceBase64 != null) { this.dvobjectThumbnailsMap.put(assignedThumbnailFileId, imageSourceBase64); return imageSourceBase64; } - // OK - we can't use this "assigned" image, because of permissions, or because - // the thumbnail failed to generate, etc... in this case we'll + // OK - we can't use this "assigned" image, because of permissions, or because + // the thumbnail failed to generate, etc... in this case we'll // mark this dataset in the lookup map - so that we don't have to // do all these lookups again... this.dvobjectThumbnailsMap.put(assignedThumbnailFileId, ""); - + // TODO: (?) // do we need to cache this datafile object in the view map? // -- L.A., 4.2.2 @@ -92,20 +92,20 @@ private String getAssignedDatasetImage(Dataset dataset, int size) { // it's the responsibility of the user - to make sure the search result // passed to this method is of the Datafile type! public String getFileCardImageAsBase64Url(SolrSearchResult result) { - // Before we do anything else, check if it's a harvested dataset; - // no need to check anything else if so (harvested objects never have + // Before we do anything else, check if it's a harvested dataset; + // no need to check anything else if so (harvested objects never have // thumbnails) - + if (result.isHarvested()) { - return null; + return null; } - + Long imageFileId = result.getEntity().getId(); if (imageFileId != null) { if (this.dvobjectThumbnailsMap.containsKey(imageFileId)) { // Yes, return previous answer - //logger.info("using cached result for ... "+datasetId); + // logger.info("using cached result for ... "+datasetId); if (!"".equals(this.dvobjectThumbnailsMap.get(imageFileId))) { return this.dvobjectThumbnailsMap.get(imageFileId); } @@ -113,7 +113,7 @@ public String getFileCardImageAsBase64Url(SolrSearchResult result) { } String cardImageUrl = null; - + if (result.getTabularDataTags() != null) { for (String tabularTagLabel : result.getTabularDataTags()) { DataFileTag tag = new DataFileTag(); @@ -122,15 +122,15 @@ public String getFileCardImageAsBase64Url(SolrSearchResult result) { tag.setDataFile((DataFile) result.getEntity()); ((DataFile) result.getEntity()).addTag(tag); } catch (IllegalArgumentException iax) { - // ignore + // ignore } } } - if ((!((DataFile)result.getEntity()).isRestricted() - || permissionsWrapper.hasDownloadFilePermission(result.getEntity())) + if ((!((DataFile) result.getEntity()).isRestricted() + || permissionsWrapper.hasDownloadFilePermission(result.getEntity())) && dataFileService.isThumbnailAvailable((DataFile) result.getEntity())) { - + cardImageUrl = ImageThumbConverter.getImageThumbnailAsBase64( (DataFile) result.getEntity(), ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); @@ -138,7 +138,7 @@ public String getFileCardImageAsBase64Url(SolrSearchResult result) { if (cardImageUrl != null) { this.dvobjectThumbnailsMap.put(imageFileId, cardImageUrl); - //logger.info("datafile id " + imageFileId + ", returning " + cardImageUrl); + // logger.info("datafile id " + imageFileId + ", returning " + cardImageUrl); if (!(dvobjectViewMap.containsKey(imageFileId) && dvobjectViewMap.get(imageFileId).isInstanceofDataFile())) { @@ -158,39 +158,40 @@ public String getFileCardImageAsBase64Url(SolrSearchResult result) { // it's the responsibility of the user - to make sure the search result // passed to this method is of the Dataset type! public String getDatasetCardImageAsBase64Url(SolrSearchResult result) { - // Before we do anything else, check if it's a harvested dataset; - // no need to check anything else if so (harvested datasets never have + // Before we do anything else, check if it's a harvested dataset; + // no need to check anything else if so (harvested datasets never have // thumbnails) if (result.isHarvested()) { - return null; + return null; } - - // Check if the search result ("card") contains an entity, before - // attempting to convert it to a Dataset. It occasionally happens that + + // Check if the search result ("card") contains an entity, before + // attempting to convert it to a Dataset. It occasionally happens that // solr has indexed datasets that are no longer in the database. If this // is the case, the entity will be null here; and proceeding any further - // results in a long stack trace in the log file. + // results in a long stack trace in the log file. if (result.getEntity() == null) { return null; } - Dataset dataset = (Dataset)result.getEntity(); - + Dataset dataset = (Dataset) result.getEntity(); + Long versionId = result.getDatasetVersionId(); - return getDatasetCardImageAsBase64Url(dataset, versionId, result.isPublishedState(), ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); + return getDatasetCardImageAsBase64Url(dataset, versionId, result.isPublishedState(), + ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); } - + public String getDatasetCardImageAsBase64Url(Dataset dataset, Long versionId, boolean autoselect, int size) { Long datasetId = dataset.getId(); if (datasetId != null) { if (this.dvobjectThumbnailsMap.containsKey(datasetId)) { // Yes, return previous answer // (at max, there could only be 2 cards for the same dataset - // on the page - the draft, and the published version; but it's + // on the page - the draft, and the published version; but it's // still nice to try and cache the result - especially if it's an - // uploaded logo - we don't want to read it off disk twice). - + // uploaded logo - we don't want to read it off disk twice). + if (!"".equals(this.dvobjectThumbnailsMap.get(datasetId))) { return this.dvobjectThumbnailsMap.get(datasetId); } @@ -200,30 +201,29 @@ public String getDatasetCardImageAsBase64Url(Dataset dataset, Long versionId, bo if (dataset.isUseGenericThumbnail()) { this.dvobjectThumbnailsMap.put(datasetId, ""); - return null; + return null; } - + String cardImageUrl = null; StorageIO dataAccess = null; - - try{ + + try { dataAccess = DataAccess.getStorageIO(dataset); + } catch (IOException ioex) { + // ignore } - catch(IOException ioex){ - // ignore - } - + InputStream in = null; // See if the dataset already has a dedicated thumbnail ("logo") saved as - // an auxilary file on the dataset level: + // an auxilary file on the dataset level: // (don't bother checking if it exists; just try to open the input stream) try { - in = dataAccess.getAuxFileAsInputStream(datasetLogoThumbnail + ".thumb" + size); - //thumb48addedByImageThumbConverter); + in = dataAccess.getAuxFileAsInputStream(datasetLogoThumbnail + ".thumb" + size); + // thumb48addedByImageThumbConverter); } catch (Exception ioex) { - //ignore + // ignore } - + if (in != null) { try { byte[] bytes = IOUtils.toByteArray(in); @@ -233,40 +233,40 @@ public String getDatasetCardImageAsBase64Url(Dataset dataset, Long versionId, bo return cardImageUrl; } catch (IOException ex) { this.dvobjectThumbnailsMap.put(datasetId, ""); - return null; - // (alternatively, we could ignore the exception, and proceed with the - // regular process of selecting the thumbnail from the available + return null; + // (alternatively, we could ignore the exception, and proceed with the + // regular process of selecting the thumbnail from the available // image files - ?) - } finally - { - IOUtils.closeQuietly(in); - } - } + } finally { + IOUtils.closeQuietly(in); + } + } // If not, see if the dataset has one of its image files already assigned // to be the designated thumbnail: cardImageUrl = this.getAssignedDatasetImage(dataset, size); if (cardImageUrl != null) { - //logger.info("dataset id " + result.getEntity().getId() + " has a dedicated image assigned; returning " + cardImageUrl); + // logger.info("dataset id " + result.getEntity().getId() + " has a dedicated + // image assigned; returning " + cardImageUrl); return cardImageUrl; } - + // And finally, try to auto-select the thumbnail (unless instructed not to): - + if (!autoselect) { return null; } - // We attempt to auto-select via the optimized, native query-based method + // We attempt to auto-select via the optimized, native query-based method // from the DatasetVersionService: Long thumbnailImageFileId = datasetVersionService.getThumbnailByVersionId(versionId); if (thumbnailImageFileId != null) { - //cardImageUrl = FILE_CARD_IMAGE_URL + thumbnailImageFileId; + // cardImageUrl = FILE_CARD_IMAGE_URL + thumbnailImageFileId; if (this.dvobjectThumbnailsMap.containsKey(thumbnailImageFileId)) { // Yes, return previous answer - //logger.info("using cached result for ... "+datasetId); + // logger.info("using cached result for ... "+datasetId); if (!"".equals(this.dvobjectThumbnailsMap.get(thumbnailImageFileId))) { return this.dvobjectThumbnailsMap.get(thumbnailImageFileId); } @@ -295,7 +295,7 @@ public String getDatasetCardImageAsBase64Url(Dataset dataset, Long versionId, bo cardImageUrl = ImageThumbConverter.getImageThumbnailAsBase64( thumbnailImageFile, size); - //ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); + // ImageThumbConverter.DEFAULT_CARDIMAGE_SIZE); } if (cardImageUrl != null) { @@ -305,21 +305,21 @@ public String getDatasetCardImageAsBase64Url(Dataset dataset, Long versionId, bo } } - //logger.info("dataset id " + result.getEntityId() + ", returning " + cardImageUrl); + // logger.info("dataset id " + result.getEntityId() + ", returning " + + // cardImageUrl); return cardImageUrl; } - + // it's the responsibility of the user - to make sure the search result // passed to this method is of the Dataverse type! public String getDataverseCardImageAsBase64Url(SolrSearchResult result) { return dataverseService.getDataverseLogoThumbnailAsBase64ById(result.getEntityId()); } - + public void resetObjectMaps() { dvobjectThumbnailsMap = new HashMap<>(); dvobjectViewMap = new HashMap<>(); } - } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java b/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java index a852abe51ac..7669af53011 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/AbstractApiBean.java @@ -30,8 +30,6 @@ import edu.harvard.iq.dataverse.authorization.RoleAssignee; import edu.harvard.iq.dataverse.authorization.groups.GroupServiceBean; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; -import edu.harvard.iq.dataverse.authorization.users.GuestUser; -import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.confirmemail.ConfirmEmailServiceBean; import edu.harvard.iq.dataverse.datacapturemodule.DataCaptureModuleServiceBean; @@ -43,14 +41,11 @@ import edu.harvard.iq.dataverse.externaltools.ExternalToolServiceBean; import edu.harvard.iq.dataverse.license.LicenseServiceBean; import edu.harvard.iq.dataverse.metrics.MetricsServiceBean; -import edu.harvard.iq.dataverse.privateurl.PrivateUrlServiceBean; import edu.harvard.iq.dataverse.locality.StorageSiteServiceBean; import edu.harvard.iq.dataverse.search.savedsearch.SavedSearchServiceBean; -import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.SystemConfig; -import edu.harvard.iq.dataverse.util.UrlSignerUtil; import edu.harvard.iq.dataverse.util.json.JsonParser; import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; import edu.harvard.iq.dataverse.validation.PasswordValidatorServiceBean; @@ -76,6 +71,7 @@ import jakarta.persistence.NoResultException; import jakarta.persistence.PersistenceContext; import jakarta.servlet.http.HttpServletRequest; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.MediaType; import jakarta.ws.rs.core.Response; @@ -93,10 +89,9 @@ public abstract class AbstractApiBean { private static final String DATAVERSE_KEY_HEADER_NAME = "X-Dataverse-key"; private static final String PERSISTENT_ID_KEY=":persistentId"; private static final String ALIAS_KEY=":alias"; - public static final String STATUS_ERROR = "ERROR"; - public static final String STATUS_OK = "OK"; public static final String STATUS_WF_IN_PROGRESS = "WORKFLOW_IN_PROGRESS"; public static final String DATAVERSE_WORKFLOW_INVOCATION_HEADER_NAME = "X-Dataverse-invocationID"; + public static final String RESPONSE_MESSAGE_AUTHENTICATED_USER_REQUIRED = "Only authenticated users can perform the requested operation"; /** * Utility class to convey a proper error response using Java's exceptions. @@ -209,9 +204,6 @@ String getWrappedMessageWhenJson() { @EJB protected SavedSearchServiceBean savedSearchSvc; - @EJB - protected PrivateUrlServiceBean privateUrlSvc; - @EJB protected ConfirmEmailServiceBean confirmEmailSvc; @@ -278,7 +270,7 @@ public JsonParser call() throws Exception { /** * Functional interface for handling HTTP requests in the APIs. * - * @see #response(edu.harvard.iq.dataverse.api.AbstractApiBean.DataverseRequestHandler) + * @see #response(edu.harvard.iq.dataverse.api.AbstractApiBean.DataverseRequestHandler, edu.harvard.iq.dataverse.authorization.users.User) */ protected static interface DataverseRequestHandler { Response handle( DataverseRequest u ) throws WrappedResponse; @@ -320,12 +312,30 @@ protected String getRequestApiKey() { return headerParamApiKey!=null ? headerParamApiKey : queryParamApiKey; } - - protected String getRequestWorkflowInvocationID() { - String headerParamWFKey = httpRequest.getHeader(DATAVERSE_WORKFLOW_INVOCATION_HEADER_NAME); - String queryParamWFKey = httpRequest.getParameter("invocationID"); - - return headerParamWFKey!=null ? headerParamWFKey : queryParamWFKey; + + protected User getRequestUser(ContainerRequestContext crc) { + return (User) crc.getProperty(ApiConstants.CONTAINER_REQUEST_CONTEXT_USER); + } + + /** + * Gets the authenticated user from the ContainerRequestContext user property. If the user from the property + * is not authenticated, throws a wrapped "authenticated user required" user (HTTP UNAUTHORIZED) response. + * @param crc a ContainerRequestContext implementation + * @return The authenticated user + * @throws edu.harvard.iq.dataverse.api.AbstractApiBean.WrappedResponse in case the user is not authenticated. + * + * TODO: + * This method is designed to comply with existing authorization logic, based on the old findAuthenticatedUserOrDie method. + * Ideally, as for authentication, a filter could be implemented for authorization, which would extract and encapsulate the + * authorization logic from the AbstractApiBean. + */ + protected AuthenticatedUser getRequestAuthenticatedUserOrDie(ContainerRequestContext crc) throws WrappedResponse { + User requestUser = (User) crc.getProperty(ApiConstants.CONTAINER_REQUEST_CONTEXT_USER); + if (requestUser.isAuthenticated()) { + return (AuthenticatedUser) requestUser; + } else { + throw new WrappedResponse(authenticatedUserRequired()); + } } /* ========= *\ @@ -346,111 +356,13 @@ protected RoleAssignee findAssignee(String identifier) { } /** - * * @param apiKey the key to find the user with * @return the user, or null - * @see #findUserOrDie(java.lang.String) */ protected AuthenticatedUser findUserByApiToken( String apiKey ) { return authSvc.lookupUser(apiKey); } - /** - * Returns the user of pointed by the API key, or the guest user - * @return a user, may be a guest user. - * @throws edu.harvard.iq.dataverse.api.AbstractApiBean.WrappedResponse iff there is an api key present, but it is invalid. - */ - protected User findUserOrDie() throws WrappedResponse { - final String requestApiKey = getRequestApiKey(); - final String requestWFKey = getRequestWorkflowInvocationID(); - if (requestApiKey == null && requestWFKey == null && getRequestParameter(UrlSignerUtil.SIGNED_URL_TOKEN)==null) { - return GuestUser.get(); - } - PrivateUrlUser privateUrlUser = privateUrlSvc.getPrivateUrlUserFromToken(requestApiKey); - // For privateUrlUsers restricted to anonymized access, all api calls are off-limits except for those used in the UI - // to download the file or image thumbs - if (privateUrlUser != null) { - if (privateUrlUser.hasAnonymizedAccess()) { - String pathInfo = httpRequest.getPathInfo(); - String prefix= "/access/datafile/"; - if (!(pathInfo.startsWith(prefix) && !pathInfo.substring(prefix.length()).contains("/"))) { - logger.info("Anonymized access request for " + pathInfo); - throw new WrappedResponse(error(Status.UNAUTHORIZED, "API Access not allowed with this Key")); - } - } - return privateUrlUser; - } - return findAuthenticatedUserOrDie(requestApiKey, requestWFKey); - } - - /** - * Finds the authenticated user, based on (in order): - *
    - *
  1. The key in the HTTP header {@link #DATAVERSE_KEY_HEADER_NAME}
    2. - *
  2. The key in the query parameter {@code key} - *
- * - * If no user is found, throws a wrapped bad api key (HTTP UNAUTHORIZED) response. - * - * @return The authenticated user which owns the passed api key - * @throws edu.harvard.iq.dataverse.api.AbstractApiBean.WrappedResponse in case said user is not found. - */ - protected AuthenticatedUser findAuthenticatedUserOrDie() throws WrappedResponse { - return findAuthenticatedUserOrDie(getRequestApiKey(), getRequestWorkflowInvocationID()); - } - - - private AuthenticatedUser findAuthenticatedUserOrDie( String key, String wfid ) throws WrappedResponse { - if (key != null) { - // No check for deactivated user because it's done in authSvc.lookupUser. - AuthenticatedUser authUser = authSvc.lookupUser(key); - - if (authUser != null) { - authUser = userSvc.updateLastApiUseTime(authUser); - - return authUser; - } - else { - throw new WrappedResponse(badApiKey(key)); - } - } else if (wfid != null) { - AuthenticatedUser authUser = authSvc.lookupUserForWorkflowInvocationID(wfid); - if (authUser != null) { - return authUser; - } else { - throw new WrappedResponse(badWFKey(wfid)); - } - } else if (getRequestParameter(UrlSignerUtil.SIGNED_URL_TOKEN) != null) { - AuthenticatedUser authUser = getAuthenticatedUserFromSignedUrl(); - if (authUser != null) { - return authUser; - } - } - //Just send info about the apiKey - workflow users will learn about invocationId elsewhere - throw new WrappedResponse(badApiKey(null)); - } - - private AuthenticatedUser getAuthenticatedUserFromSignedUrl() { - AuthenticatedUser authUser = null; - // The signedUrl contains a param telling which user this is supposed to be for. - // We don't trust this. So we lookup that user, and get their API key, and use - // that as a secret in validating the signedURL. If the signature can't be - // validated with their key, the user (or their API key) has been changed and - // we reject the request. - // ToDo - add null checks/ verify that calling methods catch things. - String user = httpRequest.getParameter("user"); - AuthenticatedUser targetUser = authSvc.getAuthenticatedUser(user); - String key = JvmSettings.API_SIGNING_SECRET.lookupOptional().orElse("") - + authSvc.findApiTokenByUser(targetUser).getTokenString(); - String signedUrl = httpRequest.getRequestURL().toString() + "?" + httpRequest.getQueryString(); - String method = httpRequest.getMethod(); - boolean validated = UrlSignerUtil.isValidUrl(signedUrl, user, method, key); - if (validated) { - authUser = targetUser; - } - return authUser; - } - protected Dataverse findDataverseOrDie( String dvIdtf ) throws WrappedResponse { Dataverse dv = findDataverse(dvIdtf); if ( dv == null ) { @@ -504,6 +416,7 @@ protected Dataset findDatasetOrDie(String id) throws WrappedResponse { } protected DataFile findDataFileOrDie(String id) throws WrappedResponse { + DataFile datafile; if (id.equals(PERSISTENT_ID_KEY)) { String persistentId = getRequestParameter(PERSISTENT_ID_KEY.substring(1)); @@ -701,49 +614,39 @@ protected Response response( Callable hdl ) { } catch ( WrappedResponse rr ) { return rr.getResponse(); } catch ( Exception ex ) { - String incidentId = UUID.randomUUID().toString(); - logger.log(Level.SEVERE, "API internal error " + incidentId +": " + ex.getMessage(), ex); - return Response.status(500) - .entity( Json.createObjectBuilder() - .add("status", "ERROR") - .add("code", 500) - .add("message", "Internal server error. More details available at the server logs.") - .add("incidentId", incidentId) - .build()) - .type("application/json").build(); + return handleDataverseRequestHandlerException(ex); } } - /** - * The preferred way of handling a request that requires a user. The system - * looks for the user and, if found, handles it to the handler for doing the - * actual work. - * - * This is a relatively secure way to handle things, since if the user is not - * found, the response is about the bad API key, rather than something else - * (say, 404 NOT FOUND which leaks information about the existence of the - * sought object). + /*** + * The preferred way of handling a request that requires a user. The method + * receives a user and handles it to the handler for doing the actual work. * * @param hdl handling code block. + * @param user the associated request user. * @return HTTP Response appropriate for the way {@code hdl} executed. */ - protected Response response( DataverseRequestHandler hdl ) { + protected Response response(DataverseRequestHandler hdl, User user) { try { - return hdl.handle(createDataverseRequest(findUserOrDie())); + return hdl.handle(createDataverseRequest(user)); } catch ( WrappedResponse rr ) { return rr.getResponse(); } catch ( Exception ex ) { - String incidentId = UUID.randomUUID().toString(); - logger.log(Level.SEVERE, "API internal error " + incidentId +": " + ex.getMessage(), ex); - return Response.status(500) - .entity( Json.createObjectBuilder() - .add("status", "ERROR") - .add("code", 500) - .add("message", "Internal server error. More details available at the server logs.") - .add("incidentId", incidentId) + return handleDataverseRequestHandlerException(ex); + } + } + + private Response handleDataverseRequestHandlerException(Exception ex) { + String incidentId = UUID.randomUUID().toString(); + logger.log(Level.SEVERE, "API internal error " + incidentId +": " + ex.getMessage(), ex); + return Response.status(500) + .entity(Json.createObjectBuilder() + .add("status", "ERROR") + .add("code", 500) + .add("message", "Internal server error. More details available at the server logs.") + .add("incidentId", incidentId) .build()) .type("application/json").build(); - } } /* ====================== *\ @@ -752,21 +655,21 @@ protected Response response( DataverseRequestHandler hdl ) { protected Response ok( JsonArrayBuilder bld ) { return Response.ok(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", bld).build()) .type(MediaType.APPLICATION_JSON).build(); } protected Response ok( JsonArray ja ) { return Response.ok(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", ja).build()) .type(MediaType.APPLICATION_JSON).build(); } protected Response ok( JsonObjectBuilder bld ) { return Response.ok( Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", bld).build() ) .type(MediaType.APPLICATION_JSON) .build(); @@ -774,7 +677,7 @@ protected Response ok( JsonObjectBuilder bld ) { protected Response ok( JsonObject jo ) { return Response.ok( Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", jo).build() ) .type(MediaType.APPLICATION_JSON) .build(); @@ -782,7 +685,7 @@ protected Response ok( JsonObject jo ) { protected Response ok( String msg ) { return Response.ok().entity(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", Json.createObjectBuilder().add("message",msg)).build() ) .type(MediaType.APPLICATION_JSON) .build(); @@ -790,7 +693,7 @@ protected Response ok( String msg ) { protected Response ok( String msg, JsonObjectBuilder bld ) { return Response.ok().entity(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("message", Json.createObjectBuilder().add("message",msg)) .add("data", bld).build()) .type(MediaType.APPLICATION_JSON) @@ -799,7 +702,7 @@ protected Response ok( String msg, JsonObjectBuilder bld ) { protected Response ok( boolean value ) { return Response.ok().entity(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", value).build() ).build(); } @@ -856,16 +759,11 @@ protected Response forbidden( String msg ) { protected Response conflict( String msg ) { return error( Status.CONFLICT, msg ); } - - protected Response badApiKey( String apiKey ) { - return error(Status.UNAUTHORIZED, (apiKey != null ) ? "Bad api key " : "Please provide a key query parameter (?key=XXX) or via the HTTP header " + DATAVERSE_KEY_HEADER_NAME); - } - protected Response badWFKey( String wfId ) { - String message = (wfId != null ) ? "Bad workflow invocationId " : "Please provide an invocationId query parameter (?invocationId=XXX) or via the HTTP header " + DATAVERSE_WORKFLOW_INVOCATION_HEADER_NAME; - return error(Status.UNAUTHORIZED, message ); + protected Response authenticatedUserRequired() { + return error(Status.UNAUTHORIZED, RESPONSE_MESSAGE_AUTHENTICATED_USER_REQUIRED); } - + protected Response permissionError( PermissionException pe ) { return permissionError( pe.getMessage() ); } @@ -881,7 +779,7 @@ protected Response unauthorized( String message ) { protected static Response error( Status sts, String msg ) { return Response.status(sts) .entity( NullSafeJsonBuilder.jsonObjectBuilder() - .add("status", STATUS_ERROR) + .add("status", ApiConstants.STATUS_ERROR) .add( "message", msg ).build() ).type(MediaType.APPLICATION_JSON_TYPE).build(); } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Access.java b/src/main/java/edu/harvard/iq/dataverse/api/Access.java index 95a9a4cc50a..c0e4d5c0dd2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Access.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Access.java @@ -32,6 +32,8 @@ import edu.harvard.iq.dataverse.UserNotificationServiceBean; import static edu.harvard.iq.dataverse.api.Datasets.handleVersion; + +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.authorization.Permission; import edu.harvard.iq.dataverse.authorization.RoleAssignee; @@ -112,6 +114,7 @@ import jakarta.ws.rs.QueryParam; import jakarta.ws.rs.ServiceUnavailableException; import jakarta.ws.rs.WebApplicationException; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.Response; import static jakarta.ws.rs.core.Response.Status.BAD_REQUEST; import jakarta.ws.rs.core.StreamingOutput; @@ -185,36 +188,31 @@ public class Access extends AbstractApiBean { @Inject MakeDataCountLoggingServiceBean mdcLogService; - - private static final String API_KEY_HEADER = "X-Dataverse-key"; - //@EJB // TODO: // versions? -- L.A. 4.0 beta 10 - @Path("datafile/bundle/{fileId}") @GET + @AuthRequired + @Path("datafile/bundle/{fileId}") @Produces({"application/zip"}) - public BundleDownloadInstance datafileBundle(@PathParam("fileId") String fileId, @QueryParam("fileMetadataId") Long fileMetadataId,@QueryParam("gbrecs") boolean gbrecs, @QueryParam("key") String apiToken, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) /*throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { + public BundleDownloadInstance datafileBundle(@Context ContainerRequestContext crc, @PathParam("fileId") String fileId, @QueryParam("fileMetadataId") Long fileMetadataId,@QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) /*throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { GuestbookResponse gbr = null; DataFile df = findDataFileOrDieWrapper(fileId); - if (apiToken == null || apiToken.equals("")) { - apiToken = headers.getHeaderString(API_KEY_HEADER); - } - // This will throw a ForbiddenException if access isn't authorized: - checkAuthorization(df, apiToken); + checkAuthorization(getRequestUser(crc), df); if (gbrecs != true && df.isReleased()){ // Write Guestbook record if not done previously and file is released - User apiTokenUser = findAPITokenUser(apiToken); + //This calls findUserOrDie which will retrieve the key param or api token header, or the workflow token header. + User apiTokenUser = findAPITokenUser(getRequestUser(crc)); gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, apiTokenUser); guestbookResponseService.save(gbr); - MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, df); + MakeDataCountEntry entry = new MakeDataCountEntry(uriInfo, headers, dvRequestService, df); mdcLogService.logEntry(entry); } @@ -273,10 +271,11 @@ private DataFile findDataFileOrDieWrapper(String fileId){ } - @Path("datafile/{fileId:.+}") @GET + @AuthRequired + @Path("datafile/{fileId:.+}") @Produces({"application/xml"}) - public Response datafile(@PathParam("fileId") String fileId, @QueryParam("gbrecs") boolean gbrecs, @QueryParam("key") String apiToken, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) /*throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { + public Response datafile(@Context ContainerRequestContext crc, @PathParam("fileId") String fileId, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) /*throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { // check first if there's a trailing slash, and chop it: while (fileId.lastIndexOf('/') == fileId.length() - 1) { @@ -301,20 +300,16 @@ public Response datafile(@PathParam("fileId") String fileId, @QueryParam("gbrecs throw new NotFoundException(errorMessage); // (nobody should ever be using this API on a harvested DataFile)! } - - if (apiToken == null || apiToken.equals("")) { - apiToken = headers.getHeaderString(API_KEY_HEADER); - } - + + // This will throw a ForbiddenException if access isn't authorized: + checkAuthorization(getRequestUser(crc), df); + if (gbrecs != true && df.isReleased()){ // Write Guestbook record if not done previously and file is released - User apiTokenUser = findAPITokenUser(apiToken); + User apiTokenUser = findAPITokenUser(getRequestUser(crc)); gbr = guestbookResponseService.initAPIGuestbookResponse(df.getOwner(), df, session, apiTokenUser); } - - // This will throw a ForbiddenException if access isn't authorized: - checkAuthorization(df, apiToken); - + DownloadInfo dInfo = new DownloadInfo(df); logger.fine("checking if thumbnails are supported on this file."); @@ -440,11 +435,12 @@ public Response datafile(@PathParam("fileId") String fileId, @QueryParam("gbrecs // Metadata format defaults to DDI: - @Path("datafile/{fileId}/metadata") @GET + @AuthRequired + @Path("datafile/{fileId}/metadata") @Produces({"text/xml"}) - public String tabularDatafileMetadata(@PathParam("fileId") String fileId, @QueryParam("fileMetadataId") Long fileMetadataId, @QueryParam("exclude") String exclude, @QueryParam("include") String include, @Context HttpHeaders header, @Context HttpServletResponse response) throws NotFoundException, ServiceUnavailableException /*, PermissionDeniedException, AuthorizationRequiredException*/ { - return tabularDatafileMetadataDDI(fileId, fileMetadataId, exclude, include, header, response); + public String tabularDatafileMetadata(@Context ContainerRequestContext crc, @PathParam("fileId") String fileId, @QueryParam("fileMetadataId") Long fileMetadataId, @QueryParam("exclude") String exclude, @QueryParam("include") String include, @Context HttpHeaders header, @Context HttpServletResponse response) throws NotFoundException, ServiceUnavailableException /*, PermissionDeniedException, AuthorizationRequiredException*/ { + return tabularDatafileMetadataDDI(crc, fileId, fileMetadataId, exclude, include, header, response); } /* @@ -452,9 +448,10 @@ public String tabularDatafileMetadata(@PathParam("fileId") String fileId, @Query * which we are going to retire. */ @Path("datafile/{fileId}/metadata/ddi") + @AuthRequired @GET @Produces({"text/xml"}) - public String tabularDatafileMetadataDDI(@PathParam("fileId") String fileId, @QueryParam("fileMetadataId") Long fileMetadataId, @QueryParam("exclude") String exclude, @QueryParam("include") String include, @Context HttpHeaders header, @Context HttpServletResponse response) throws NotFoundException, ServiceUnavailableException /*, PermissionDeniedException, AuthorizationRequiredException*/ { + public String tabularDatafileMetadataDDI(@Context ContainerRequestContext crc, @PathParam("fileId") String fileId, @QueryParam("fileMetadataId") Long fileMetadataId, @QueryParam("exclude") String exclude, @QueryParam("include") String include, @Context HttpHeaders header, @Context HttpServletResponse response) throws NotFoundException, ServiceUnavailableException /*, PermissionDeniedException, AuthorizationRequiredException*/ { String retValue = ""; DataFile dataFile = null; @@ -469,11 +466,7 @@ public String tabularDatafileMetadataDDI(@PathParam("fileId") String fileId, @Q if (dataFile.isRestricted() || FileUtil.isActivelyEmbargoed(dataFile)) { boolean hasPermissionToDownloadFile = false; DataverseRequest dataverseRequest; - try { - dataverseRequest = createDataverseRequest(findUserOrDie()); - } catch (WrappedResponse ex) { - throw new BadRequestException("cannot find user"); - } + dataverseRequest = createDataverseRequest(getRequestUser(crc)); if (dataverseRequest != null && dataverseRequest.getUser() instanceof GuestUser) { // We must be in the UI. Try to get a non-GuestUser from the session. dataverseRequest = dvRequestService.getDataverseRequest(); @@ -527,44 +520,42 @@ public String tabularDatafileMetadataDDI(@PathParam("fileId") String fileId, @Q * a tabular datafile. */ - @Path("datafile/{fileId}/auxiliary") @GET - public Response listDatafileMetadataAux(@PathParam("fileId") String fileId, - @QueryParam("key") String apiToken, - @Context UriInfo uriInfo, - @Context HttpHeaders headers, - @Context HttpServletResponse response) throws ServiceUnavailableException { - return listAuxiliaryFiles(fileId, null, apiToken, uriInfo, headers, response); + @AuthRequired + @Path("datafile/{fileId}/auxiliary") + public Response listDatafileMetadataAux(@Context ContainerRequestContext crc, + @PathParam("fileId") String fileId, + @Context UriInfo uriInfo, + @Context HttpHeaders headers, + @Context HttpServletResponse response) throws ServiceUnavailableException { + return listAuxiliaryFiles(getRequestUser(crc), fileId, null, uriInfo, headers, response); } /* * GET method for retrieving a list auxiliary files associated with * a tabular datafile and having the specified origin. */ - - @Path("datafile/{fileId}/auxiliary/{origin}") + @GET - public Response listDatafileMetadataAuxByOrigin(@PathParam("fileId") String fileId, - @PathParam("origin") String origin, - @QueryParam("key") String apiToken, - @Context UriInfo uriInfo, - @Context HttpHeaders headers, - @Context HttpServletResponse response) throws ServiceUnavailableException { - return listAuxiliaryFiles(fileId, origin, apiToken, uriInfo, headers, response); + @AuthRequired + @Path("datafile/{fileId}/auxiliary/{origin}") + public Response listDatafileMetadataAuxByOrigin(@Context ContainerRequestContext crc, + @PathParam("fileId") String fileId, + @PathParam("origin") String origin, + @Context UriInfo uriInfo, + @Context HttpHeaders headers, + @Context HttpServletResponse response) throws ServiceUnavailableException { + return listAuxiliaryFiles(getRequestUser(crc), fileId, origin, uriInfo, headers, response); } - private Response listAuxiliaryFiles(String fileId, String origin, String apiToken, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response) { + private Response listAuxiliaryFiles(User user, String fileId, String origin, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response) { DataFile df = findDataFileOrDieWrapper(fileId); - if (apiToken == null || apiToken.equals("")) { - apiToken = headers.getHeaderString(API_KEY_HEADER); - } - List auxFileList = auxiliaryFileService.findAuxiliaryFiles(df, origin); if (auxFileList == null || auxFileList.isEmpty()) { throw new NotFoundException("No Auxiliary files exist for datafile " + fileId + (origin==null ? "": " and the specified origin")); } - boolean isAccessAllowed = isAccessAuthorized(df, apiToken); + boolean isAccessAllowed = isAccessAuthorized(user, df); JsonArrayBuilder jab = Json.createArrayBuilder(); auxFileList.forEach(auxFile -> { if (isAccessAllowed || auxFile.getIsPublic()) { @@ -587,22 +578,19 @@ private Response listAuxiliaryFiles(String fileId, String origin, String apiToke * */ + @GET + @AuthRequired @Path("datafile/{fileId}/auxiliary/{formatTag}/{formatVersion}") - @GET - public DownloadInstance downloadAuxiliaryFile(@PathParam("fileId") String fileId, - @PathParam("formatTag") String formatTag, - @PathParam("formatVersion") String formatVersion, - @QueryParam("key") String apiToken, - @Context UriInfo uriInfo, - @Context HttpHeaders headers, - @Context HttpServletResponse response) throws ServiceUnavailableException { + public DownloadInstance downloadAuxiliaryFile(@Context ContainerRequestContext crc, + @PathParam("fileId") String fileId, + @PathParam("formatTag") String formatTag, + @PathParam("formatVersion") String formatVersion, + @Context UriInfo uriInfo, + @Context HttpHeaders headers, + @Context HttpServletResponse response) throws ServiceUnavailableException { DataFile df = findDataFileOrDieWrapper(fileId); - if (apiToken == null || apiToken.equals("")) { - apiToken = headers.getHeaderString(API_KEY_HEADER); - } - DownloadInfo dInfo = new DownloadInfo(df); boolean publiclyAvailable = false; @@ -652,7 +640,7 @@ public DownloadInstance downloadAuxiliaryFile(@PathParam("fileId") String fileId // as defined for the DataFile itself), and will throw a ForbiddenException // if access is denied: if (!publiclyAvailable) { - checkAuthorization(df, apiToken); + checkAuthorization(getRequestUser(crc), df); } return downloadInstance; @@ -664,22 +652,24 @@ public DownloadInstance downloadAuxiliaryFile(@PathParam("fileId") String fileId // TODO: Rather than only supporting looking up files by their database IDs, // consider supporting persistent identifiers. - @Path("datafiles") @POST + @AuthRequired + @Path("datafiles") @Consumes("text/plain") @Produces({ "application/zip" }) - public Response postDownloadDatafiles(String fileIds, @QueryParam("gbrecs") boolean gbrecs, @QueryParam("key") String apiTokenParam, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { + public Response postDownloadDatafiles(@Context ContainerRequestContext crc, String fileIds, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { - return downloadDatafiles(fileIds, gbrecs, apiTokenParam, uriInfo, headers, response); + return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response); } - @Path("dataset/{id}") @GET + @AuthRequired + @Path("dataset/{id}") @Produces({"application/zip"}) - public Response downloadAllFromLatest(@PathParam("id") String datasetIdOrPersistentId, @QueryParam("gbrecs") boolean gbrecs, @QueryParam("key") String apiTokenParam, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { + public Response downloadAllFromLatest(@Context ContainerRequestContext crc, @PathParam("id") String datasetIdOrPersistentId, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { try { - User user = findUserOrDie(); + User user = getRequestUser(crc); DataverseRequest req = createDataverseRequest(user); final Dataset retrieved = findDatasetOrDie(datasetIdOrPersistentId); if (!(user instanceof GuestUser)) { @@ -691,7 +681,7 @@ public Response downloadAllFromLatest(@PathParam("id") String datasetIdOrPersist // We don't want downloads from Draft versions to be counted, // so we are setting the gbrecs (aka "do not write guestbook response") // variable accordingly: - return downloadDatafiles(fileIds, true, apiTokenParam, uriInfo, headers, response); + return downloadDatafiles(getRequestUser(crc), fileIds, true, uriInfo, headers, response); } } @@ -712,18 +702,19 @@ public Response downloadAllFromLatest(@PathParam("id") String datasetIdOrPersist } String fileIds = getFileIdsAsCommaSeparated(latest.getFileMetadatas()); - return downloadDatafiles(fileIds, gbrecs, apiTokenParam, uriInfo, headers, response); + return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response); } catch (WrappedResponse wr) { return wr.getResponse(); } } - @Path("dataset/{id}/versions/{versionId}") @GET + @AuthRequired + @Path("dataset/{id}/versions/{versionId}") @Produces({"application/zip"}) - public Response downloadAllFromVersion(@PathParam("id") String datasetIdOrPersistentId, @PathParam("versionId") String versionId, @QueryParam("gbrecs") boolean gbrecs, @QueryParam("key") String apiTokenParam, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { + public Response downloadAllFromVersion(@Context ContainerRequestContext crc, @PathParam("id") String datasetIdOrPersistentId, @PathParam("versionId") String versionId, @QueryParam("gbrecs") boolean gbrecs, @QueryParam("key") String apiTokenParam, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { try { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); final Dataset ds = execCommand(new GetDatasetCommand(req, findDatasetOrDie(datasetIdOrPersistentId))); DatasetVersion dsv = execCommand(handleVersion(versionId, new Datasets.DsVersionHandler>() { @@ -761,7 +752,7 @@ public Command handleLatestPublished() { if (dsv.isDraft()) { gbrecs = true; } - return downloadDatafiles(fileIds, gbrecs, apiTokenParam, uriInfo, headers, response); + return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -779,14 +770,15 @@ private static String getFileIdsAsCommaSeparated(List fileMetadata /* * API method for downloading zipped bundles of multiple files: */ - @Path("datafiles/{fileIds}") @GET + @AuthRequired + @Path("datafiles/{fileIds}") @Produces({"application/zip"}) - public Response datafiles(@PathParam("fileIds") String fileIds, @QueryParam("gbrecs") boolean gbrecs, @QueryParam("key") String apiTokenParam, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { - return downloadDatafiles(fileIds, gbrecs, apiTokenParam, uriInfo, headers, response); + public Response datafiles(@Context ContainerRequestContext crc, @PathParam("fileIds") String fileIds, @QueryParam("gbrecs") boolean gbrecs, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WebApplicationException { + return downloadDatafiles(getRequestUser(crc), fileIds, gbrecs, uriInfo, headers, response); } - private Response downloadDatafiles(String rawFileIds, boolean donotwriteGBResponse, String apiTokenParam, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response) throws WebApplicationException /* throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { + private Response downloadDatafiles(User user, String rawFileIds, boolean donotwriteGBResponse, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response) throws WebApplicationException /* throws NotFoundException, ServiceUnavailableException, PermissionDeniedException, AuthorizationRequiredException*/ { final long zipDownloadSizeLimit = systemConfig.getZipDownloadLimit(); logger.fine("setting zip download size limit to " + zipDownloadSizeLimit + " bytes."); @@ -808,11 +800,7 @@ private Response downloadDatafiles(String rawFileIds, boolean donotwriteGBRespon String customZipServiceUrl = settingsService.getValueForKey(SettingsServiceBean.Key.CustomZipDownloadServiceUrl); boolean useCustomZipService = customZipServiceUrl != null; - String apiToken = (apiTokenParam == null || apiTokenParam.equals("")) - ? headers.getHeaderString(API_KEY_HEADER) - : apiTokenParam; - - User apiTokenUser = findAPITokenUser(apiToken); //for use in adding gb records if necessary + User apiTokenUser = findAPITokenUser(user); //for use in adding gb records if necessary Boolean getOrig = false; for (String key : uriInfo.getQueryParameters().keySet()) { @@ -825,7 +813,7 @@ private Response downloadDatafiles(String rawFileIds, boolean donotwriteGBRespon if (useCustomZipService) { URI redirect_uri = null; try { - redirect_uri = handleCustomZipDownload(customZipServiceUrl, fileIds, apiToken, apiTokenUser, uriInfo, headers, donotwriteGBResponse, true); + redirect_uri = handleCustomZipDownload(user, customZipServiceUrl, fileIds, apiTokenUser, uriInfo, headers, donotwriteGBResponse, true); } catch (WebApplicationException wae) { throw wae; } @@ -857,7 +845,7 @@ public void write(OutputStream os) throws IOException, logger.fine("token: " + fileIdParams[i]); Long fileId = null; try { - fileId = new Long(fileIdParams[i]); + fileId = Long.parseLong(fileIdParams[i]); } catch (NumberFormatException nfe) { fileId = null; } @@ -865,8 +853,8 @@ public void write(OutputStream os) throws IOException, logger.fine("attempting to look up file id " + fileId); DataFile file = dataFileService.find(fileId); if (file != null) { - if (isAccessAuthorized(file, apiToken)) { - + if (isAccessAuthorized(user, file)) { + logger.fine("adding datafile (id=" + file.getId() + ") to the download list of the ZippedDownloadInstance."); //downloadInstance.addDataFile(file); if (donotwriteGBResponse != true && file.isReleased()){ @@ -1258,23 +1246,22 @@ private String getWebappImageResource(String imageName) { * @return * */ - @Path("datafile/{fileId}/auxiliary/{formatTag}/{formatVersion}") @POST + @AuthRequired + @Path("datafile/{fileId}/auxiliary/{formatTag}/{formatVersion}") @Consumes(MediaType.MULTIPART_FORM_DATA) - - public Response saveAuxiliaryFileWithVersion(@PathParam("fileId") Long fileId, - @PathParam("formatTag") String formatTag, - @PathParam("formatVersion") String formatVersion, - @FormDataParam("origin") String origin, - @FormDataParam("isPublic") boolean isPublic, - @FormDataParam("type") String type, - @FormDataParam("file") final FormDataBodyPart formDataBodyPart, - @FormDataParam("file") InputStream fileInputStream - - ) { + public Response saveAuxiliaryFileWithVersion(@Context ContainerRequestContext crc, + @PathParam("fileId") Long fileId, + @PathParam("formatTag") String formatTag, + @PathParam("formatVersion") String formatVersion, + @FormDataParam("origin") String origin, + @FormDataParam("isPublic") boolean isPublic, + @FormDataParam("type") String type, + @FormDataParam("file") final FormDataBodyPart formDataBodyPart, + @FormDataParam("file") InputStream fileInputStream) { AuthenticatedUser authenticatedUser; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(FORBIDDEN, "Authorized users only."); } @@ -1316,14 +1303,16 @@ public Response saveAuxiliaryFileWithVersion(@PathParam("fileId") Long fileId, * @param formDataBodyPart * @return */ - @Path("datafile/{fileId}/auxiliary/{formatTag}/{formatVersion}") @DELETE - public Response deleteAuxiliaryFileWithVersion(@PathParam("fileId") Long fileId, - @PathParam("formatTag") String formatTag, - @PathParam("formatVersion") String formatVersion) { + @AuthRequired + @Path("datafile/{fileId}/auxiliary/{formatTag}/{formatVersion}") + public Response deleteAuxiliaryFileWithVersion(@Context ContainerRequestContext crc, + @PathParam("fileId") Long fileId, + @PathParam("formatTag") String formatTag, + @PathParam("formatVersion") String formatVersion) { AuthenticatedUser authenticatedUser; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(FORBIDDEN, "Authorized users only."); } @@ -1360,8 +1349,9 @@ public Response deleteAuxiliaryFileWithVersion(@PathParam("fileId") Long fileId, * @return */ @PUT + @AuthRequired @Path("{id}/allowAccessRequest") - public Response allowAccessRequest(@PathParam("id") String datasetToAllowAccessId, String requestStr) { + public Response allowAccessRequest(@Context ContainerRequestContext crc, @PathParam("id") String datasetToAllowAccessId, String requestStr) { DataverseRequest dataverseRequest = null; Dataset dataset; @@ -1375,12 +1365,7 @@ public Response allowAccessRequest(@PathParam("id") String datasetToAllowAccessI boolean allowRequest = Boolean.valueOf(requestStr); - try { - dataverseRequest = createDataverseRequest(findUserOrDie()); - } catch (WrappedResponse wr) { - List args = Arrays.asList(wr.getLocalizedMessage()); - return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.fileAccess.failure.noUser", args)); - } + dataverseRequest = createDataverseRequest(getRequestUser(crc)); dataset.getOrCreateEditVersion().getTermsOfUseAndAccess().setFileAccessRequest(allowRequest); @@ -1402,14 +1387,15 @@ public Response allowAccessRequest(@PathParam("id") String datasetToAllowAccessI * * @author sekmiller * + * @param crc * @param fileToRequestAccessId - * @param apiToken * @param headers * @return */ @PUT + @AuthRequired @Path("/datafile/{id}/requestAccess") - public Response requestFileAccess(@PathParam("id") String fileToRequestAccessId, @Context HttpHeaders headers) { + public Response requestFileAccess(@Context ContainerRequestContext crc, @PathParam("id") String fileToRequestAccessId, @Context HttpHeaders headers) { DataverseRequest dataverseRequest; DataFile dataFile; @@ -1428,14 +1414,14 @@ public Response requestFileAccess(@PathParam("id") String fileToRequestAccessId, AuthenticatedUser requestor; try { - requestor = findAuthenticatedUserOrDie(); + requestor = getRequestAuthenticatedUserOrDie(crc); dataverseRequest = createDataverseRequest(requestor); } catch (WrappedResponse wr) { List args = Arrays.asList(wr.getLocalizedMessage()); return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.fileAccess.failure.noUser", args)); } - - if (isAccessAuthorized(dataFile, getRequestApiKey())) { + //Already have access + if (isAccessAuthorized(getRequestUser(crc), dataFile)) { return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.requestAccess.failure.invalidRequest")); } @@ -1466,8 +1452,9 @@ public Response requestFileAccess(@PathParam("id") String fileToRequestAccessId, * @return */ @GET + @AuthRequired @Path("/datafile/{id}/listRequests") - public Response listFileAccessRequests(@PathParam("id") String fileToRequestAccessId, @Context HttpHeaders headers) { + public Response listFileAccessRequests(@Context ContainerRequestContext crc, @PathParam("id") String fileToRequestAccessId, @Context HttpHeaders headers) { DataverseRequest dataverseRequest; @@ -1480,7 +1467,7 @@ public Response listFileAccessRequests(@PathParam("id") String fileToRequestAcce } try { - dataverseRequest = createDataverseRequest(findAuthenticatedUserOrDie()); + dataverseRequest = createDataverseRequest(getRequestAuthenticatedUserOrDie(crc)); } catch (WrappedResponse wr) { List args = Arrays.asList(wr.getLocalizedMessage()); return error(UNAUTHORIZED, BundleUtil.getStringFromBundle("access.api.fileAccess.failure.noUser", args)); @@ -1511,15 +1498,16 @@ public Response listFileAccessRequests(@PathParam("id") String fileToRequestAcce * * @author sekmiller * + * @param crc * @param fileToRequestAccessId * @param identifier - * @param apiToken * @param headers * @return */ @PUT + @AuthRequired @Path("/datafile/{id}/grantAccess/{identifier}") - public Response grantFileAccess(@PathParam("id") String fileToRequestAccessId, @PathParam("identifier") String identifier, @Context HttpHeaders headers) { + public Response grantFileAccess(@Context ContainerRequestContext crc, @PathParam("id") String fileToRequestAccessId, @PathParam("identifier") String identifier, @Context HttpHeaders headers) { DataverseRequest dataverseRequest; DataFile dataFile; @@ -1538,12 +1526,7 @@ public Response grantFileAccess(@PathParam("id") String fileToRequestAccessId, @ return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.grantAccess.noAssigneeFound", args)); } - try { - dataverseRequest = createDataverseRequest(findUserOrDie()); - } catch (WrappedResponse wr) { - List args = Arrays.asList(identifier); - return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.fileAccess.failure.noUser", args)); - } + dataverseRequest = createDataverseRequest(getRequestUser(crc)); DataverseRole fileDownloaderRole = roleService.findBuiltinRoleByAlias(DataverseRole.FILE_DOWNLOADER); @@ -1575,15 +1558,16 @@ public Response grantFileAccess(@PathParam("id") String fileToRequestAccessId, @ * * @author sekmiller * + * @param crc * @param fileToRequestAccessId * @param identifier - * @param apiToken * @param headers * @return */ @DELETE + @AuthRequired @Path("/datafile/{id}/revokeAccess/{identifier}") - public Response revokeFileAccess(@PathParam("id") String fileToRequestAccessId, @PathParam("identifier") String identifier, @Context HttpHeaders headers) { + public Response revokeFileAccess(@Context ContainerRequestContext crc, @PathParam("id") String fileToRequestAccessId, @PathParam("identifier") String identifier, @Context HttpHeaders headers) { DataverseRequest dataverseRequest; DataFile dataFile; @@ -1595,12 +1579,7 @@ public Response revokeFileAccess(@PathParam("id") String fileToRequestAccessId, return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.requestAccess.fileNotFound", args)); } - try { - dataverseRequest = createDataverseRequest(findUserOrDie()); - } catch (WrappedResponse wr) { - List args = Arrays.asList(wr.getLocalizedMessage()); - return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.fileAccess.failure.noUser", args)); - } + dataverseRequest = createDataverseRequest(getRequestUser(crc)); if (identifier == null || identifier.equals("")) { return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.requestAccess.noKey")); @@ -1645,15 +1624,16 @@ public Response revokeFileAccess(@PathParam("id") String fileToRequestAccessId, * * @author sekmiller * + * @param crc * @param fileToRequestAccessId * @param identifier - * @param apiToken * @param headers * @return */ @PUT + @AuthRequired @Path("/datafile/{id}/rejectAccess/{identifier}") - public Response rejectFileAccess(@PathParam("id") String fileToRequestAccessId, @PathParam("identifier") String identifier, @Context HttpHeaders headers) { + public Response rejectFileAccess(@Context ContainerRequestContext crc, @PathParam("id") String fileToRequestAccessId, @PathParam("identifier") String identifier, @Context HttpHeaders headers) { DataverseRequest dataverseRequest; DataFile dataFile; @@ -1672,13 +1652,8 @@ public Response rejectFileAccess(@PathParam("id") String fileToRequestAccessId, return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.grantAccess.noAssigneeFound", args)); } - try { - dataverseRequest = createDataverseRequest(findUserOrDie()); - } catch (WrappedResponse wr) { - List args = Arrays.asList(identifier); - return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.fileAccess.failure.noUser", args)); - } - + dataverseRequest = createDataverseRequest(getRequestUser(crc)); + if (!(dataverseRequest.getAuthenticatedUser().isSuperuser() || permissionService.requestOn(dataverseRequest, dataFile).has(Permission.ManageFilePermissions))) { return error(BAD_REQUEST, BundleUtil.getStringFromBundle("access.api.rejectAccess.failure.noPermissions")); } @@ -1706,15 +1681,15 @@ public Response rejectFileAccess(@PathParam("id") String fileToRequestAccessId, // checkAuthorization is a convenience method; it calls the boolean method // isAccessAuthorized(), the actual workhorse, tand throws a 403 exception if not. - private void checkAuthorization(DataFile df, String apiToken) throws WebApplicationException { + private void checkAuthorization(User user, DataFile df) throws WebApplicationException { - if (!isAccessAuthorized(df, apiToken)) { + if (!isAccessAuthorized(user, df)) { throw new ForbiddenException(); } } - private boolean isAccessAuthorized(DataFile df, String apiToken) { + private boolean isAccessAuthorized(User requestUser, DataFile df) { // First, check if the file belongs to a released Dataset version: boolean published = false; @@ -1785,37 +1760,42 @@ private boolean isAccessAuthorized(DataFile df, String apiToken) { } } - if (!restricted && !embargoed) { - // And if they are not published, they can still be downloaded, if the user + + + //The one case where we don't need to check permissions + if (!restricted && !embargoed && published) { + // If they are not published, they can still be downloaded, if the user // has the permission to view unpublished versions! (this case will // be handled below) - if (published) { - return true; - } + return true; } - User user = null; + //For permissions check decide if we have a session user, or an API user + User sessionUser = null; /** * Authentication/authorization: + */ + + User apiUser = requestUser; + + /* + * If API user is not authenticated, and a session user exists, we use that. + * If the API user indicates a GuestUser, we will use that if there's no session. * - * note that the fragment below - that retrieves the session object - * and tries to find the user associated with the session - is really - * for logging/debugging purposes only; for practical purposes, it - * would be enough to just call "permissionService.on(df).has(Permission.DownloadFile)" - * and the method does just that, tries to authorize for the user in - * the current session (or guest user, if no session user is available): + * This is currently the only API call that supports sessions. If the rest of + * the API is opened up, the custom logic here wouldn't be needed. */ - - if (session != null) { + + if ((apiUser instanceof GuestUser) && session != null) { if (session.getUser() != null) { - if (session.getUser().isAuthenticated()) { - user = session.getUser(); - } else { + sessionUser = session.getUser(); + apiUser = null; + //Fine logging + if (!session.getUser().isAuthenticated()) { logger.fine("User associated with the session is not an authenticated user."); if (session.getUser() instanceof PrivateUrlUser) { logger.fine("User associated with the session is a PrivateUrlUser user."); - user = session.getUser(); } if (session.getUser() instanceof GuestUser) { logger.fine("User associated with the session is indeed a guest user."); @@ -1827,192 +1807,74 @@ private boolean isAccessAuthorized(DataFile df, String apiToken) { } else { logger.fine("Session is null."); } - - User apiTokenUser = null; - - if ((apiToken != null)&&(apiToken.length()!=64)) { - // We'll also try to obtain the user information from the API token, - // if supplied: - - try { - logger.fine("calling apiTokenUser = findUserOrDie()..."); - apiTokenUser = findUserOrDie(); - } catch (WrappedResponse wr) { - logger.log(Level.FINE, "Message from findUserOrDie(): {0}", wr.getMessage()); - } - - if (apiTokenUser == null) { - logger.warning("API token-based auth: Unable to find a user with the API token provided."); - } + //If we don't have a user, nothing more to do. (Note session could have returned GuestUser) + if (sessionUser == null && apiUser == null) { + logger.warning("Unable to find a user via session or with a token."); + return false; } - - // OK, let's revisit the case of non-restricted files, this time in - // an unpublished version: - // (if (published) was already addressed above) - - if (!restricted && !embargoed) { - // If the file is not published, they can still download the file, if the user - // has the permission to view unpublished versions: - - if ( user != null ) { - // used in JSF context - if (permissionService.requestOn(dvRequestService.getDataverseRequest(), df.getOwner()).has(Permission.ViewUnpublishedDataset)) { - // it's not unthinkable, that a null user (i.e., guest user) could be given - // the ViewUnpublished permission! - logger.log(Level.FINE, "Session-based auth: user {0} has access rights on the non-restricted, unpublished datafile.", user.getIdentifier()); - return true; - } - } - if (apiTokenUser != null) { - // used in an API context - if (permissionService.requestOn( createDataverseRequest(apiTokenUser), df.getOwner()).has(Permission.ViewUnpublishedDataset)) { - logger.log(Level.FINE, "Token-based auth: user {0} has access rights on the non-restricted, unpublished datafile.", apiTokenUser.getIdentifier()); - return true; - } - } + /* + * Since published and not restricted/embargoed is handled above, the main split + * now is whether it is published or not. If it's published, the only case left + * is with restricted/embargoed. With unpublished, both the restricted/embargoed + * and not restricted/embargoed both get handled the same way. + */ - // last option - guest user in either contexts - // Guset user is impled by the code above. - if ( permissionService.requestOn(dvRequestService.getDataverseRequest(), df.getOwner()).has(Permission.ViewUnpublishedDataset) ) { - return true; - } - + DataverseRequest dvr = null; + if (apiUser != null) { + dvr = createDataverseRequest(apiUser); } else { - - // OK, this is a restricted and/or embargoed file. - - boolean hasAccessToRestrictedBySession = false; - boolean hasAccessToRestrictedByToken = false; - - if (permissionService.on(df).has(Permission.DownloadFile)) { - // Note: PermissionServiceBean.on(Datafile df) will obtain the - // User from the Session object, just like in the code fragment - // above. That's why it's not passed along as an argument. - hasAccessToRestrictedBySession = true; - } else if (apiTokenUser != null && permissionService.requestOn(createDataverseRequest(apiTokenUser), df).has(Permission.DownloadFile)) { - hasAccessToRestrictedByToken = true; - } - - if (hasAccessToRestrictedBySession || hasAccessToRestrictedByToken) { - if (published) { - if (hasAccessToRestrictedBySession) { - if (user != null) { - logger.log(Level.FINE, "Session-based auth: user {0} is granted access to the restricted, published datafile.", user.getIdentifier()); - } else { - logger.fine("Session-based auth: guest user is granted access to the restricted, published datafile."); - } - } else { - logger.log(Level.FINE, "Token-based auth: user {0} is granted access to the restricted, published datafile.", apiTokenUser.getIdentifier()); - } - return true; - } else { - // if the file is NOT published, we will let them download the - // file ONLY if they also have the permission to view - // unpublished versions: - // Note that the code below does not allow a case where it is the - // session user that has the permission on the file, and the API token - // user with the ViewUnpublished permission, or vice versa! - if (hasAccessToRestrictedBySession) { - if (permissionService.on(df.getOwner()).has(Permission.ViewUnpublishedDataset)) { - if (user != null) { - logger.log(Level.FINE, "Session-based auth: user {0} is granted access to the restricted, unpublished datafile.", user.getIdentifier()); - } else { - logger.fine("Session-based auth: guest user is granted access to the restricted, unpublished datafile."); - } - return true; - } - } else { - if (apiTokenUser != null && permissionService.requestOn(createDataverseRequest(apiTokenUser), df.getOwner()).has(Permission.ViewUnpublishedDataset)) { - logger.log(Level.FINE, "Token-based auth: user {0} is granted access to the restricted, unpublished datafile.", apiTokenUser.getIdentifier()); - return true; - } - } - } - } - } + // used in JSF context, user may be Guest + dvr = dvRequestService.getDataverseRequest(); + } + if (!published) { // and restricted or embargoed (implied by earlier processing) + // If the file is not published, they can still download the file, if the user + // has the permission to view unpublished versions: - - if ((apiToken != null)) { - // Will try to obtain the user information from the API token, - // if supplied: - - try { - logger.fine("calling user = findUserOrDie()..."); - user = findUserOrDie(); - } catch (WrappedResponse wr) { - logger.log(Level.FINE, "Message from findUserOrDie(): {0}", wr.getMessage()); + // This line handles all three authenticated session user, token user, and guest cases. + if (permissionService.requestOn(dvr, df.getOwner()).has(Permission.ViewUnpublishedDataset)) { + // it's not unthinkable, that a GuestUser could be given + // the ViewUnpublished permission! + logger.log(Level.FINE, + "Session-based auth: user {0} has access rights on the non-restricted, unpublished datafile.", + dvr.getUser().getIdentifier()); + return true; } - - if (user == null) { - logger.warning("API token-based auth: Unable to find a user with the API token provided."); - return false; - } - - - //Doesn't this ~duplicate logic above - if so, if there's a way to get here, I think it still works for embargoed files (you only get access if you have download permissions, and, if not published, also view unpublished) - if (permissionService.requestOn(createDataverseRequest(user), df).has(Permission.DownloadFile)) { - if (published) { - logger.log(Level.FINE, "API token-based auth: User {0} has rights to access the datafile.", user.getIdentifier()); - //Same case as line 1809 (and part of 1708 though when published you don't need the DownloadFile permission) - return true; - } else { - // if the file is NOT published, we will let them download the - // file ONLY if they also have the permission to view - // unpublished versions: - if (permissionService.requestOn(createDataverseRequest(user), df.getOwner()).has(Permission.ViewUnpublishedDataset)) { - logger.log(Level.FINE, "API token-based auth: User {0} has rights to access the (unpublished) datafile.", user.getIdentifier()); - //Same case as line 1843? - return true; - } else { - logger.log(Level.FINE, "API token-based auth: User {0} is not authorized to access the (unpublished) datafile.", user.getIdentifier()); - } - } - } else { - logger.log(Level.FINE, "API token-based auth: User {0} is not authorized to access the datafile.", user.getIdentifier()); + } else { // published and restricted and/or embargoed + // This line also handles all three authenticated session user, token user, and guest cases. + if (permissionService.requestOn(dvr, df).has(Permission.DownloadFile)) { + return true; } - - return false; - } - - if (user != null) { - logger.log(Level.FINE, "Session-based auth: user {0} has NO access rights on the requested datafile.", user.getIdentifier()); + } + if (sessionUser != null) { + logger.log(Level.FINE, "Session-based auth: user {0} has NO access rights on the requested datafile.", sessionUser.getIdentifier()); } - if (apiTokenUser != null) { - logger.log(Level.FINE, "Token-based auth: user {0} has NO access rights on the requested datafile.", apiTokenUser.getIdentifier()); + if (apiUser != null) { + logger.log(Level.FINE, "Token-based auth: user {0} has NO access rights on the requested datafile.", apiUser.getIdentifier()); } - - if (user == null && apiTokenUser == null) { - logger.fine("Unauthenticated access: No guest access to the datafile."); - } - return false; } - private User findAPITokenUser(String apiToken) { - User apiTokenUser = null; - - if ((apiToken != null) && (apiToken.length() != 64)) { - // We'll also try to obtain the user information from the API token, - // if supplied: - - try { - logger.fine("calling apiTokenUser = findUserOrDie()..."); - apiTokenUser = findUserOrDie(); - return apiTokenUser; - } catch (WrappedResponse wr) { - logger.log(Level.FINE, "Message from findUserOrDie(): {0}", wr.getMessage()); - return null; + private User findAPITokenUser(User requestUser) { + User apiTokenUser = requestUser; + /* + * The idea here is to not let a guest user coming from the request (which + * happens when there is no key/token, and which we want if there's no session) + * from overriding an authenticated session user. + */ + if(apiTokenUser instanceof GuestUser) { + if(session!=null && session.getUser()!=null) { + //The apiTokenUser, if set, will override the sessionUser in permissions calcs, so set it to null if we have a session user + apiTokenUser=null; } - } return apiTokenUser; } - private URI handleCustomZipDownload(String customZipServiceUrl, String fileIds, String apiToken, User apiTokenUser, UriInfo uriInfo, HttpHeaders headers, boolean donotwriteGBResponse, boolean orig) throws WebApplicationException { + private URI handleCustomZipDownload(User user, String customZipServiceUrl, String fileIds, User apiTokenUser, UriInfo uriInfo, HttpHeaders headers, boolean donotwriteGBResponse, boolean orig) throws WebApplicationException { String zipServiceKey = null; Timestamp timestamp = null; @@ -2029,7 +1891,7 @@ private URI handleCustomZipDownload(String customZipServiceUrl, String fileIds, for (int i = 0; i < fileIdParams.length; i++) { Long fileId = null; try { - fileId = new Long(fileIdParams[i]); + fileId = Long.parseLong(fileIdParams[i]); validIdCount++; } catch (NumberFormatException nfe) { fileId = null; @@ -2038,7 +1900,7 @@ private URI handleCustomZipDownload(String customZipServiceUrl, String fileIds, DataFile file = dataFileService.find(fileId); if (file != null) { validFileCount++; - if (isAccessAuthorized(file, apiToken)) { + if (isAccessAuthorized(user, file)) { logger.fine("adding datafile (id=" + file.getId() + ") to the download list of the ZippedDownloadInstance."); if (donotwriteGBResponse != true && file.isReleased()) { GuestbookResponse gbr = guestbookResponseService.initAPIGuestbookResponse(file.getOwner(), file, session, apiTokenUser); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java index 6f57d7332b3..c3bab27fc08 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Admin.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Admin.java @@ -14,6 +14,7 @@ import edu.harvard.iq.dataverse.DataverseServiceBean; import edu.harvard.iq.dataverse.DataverseSession; import edu.harvard.iq.dataverse.DvObject; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.validation.EMailValidator; import edu.harvard.iq.dataverse.EjbDataverseEngine; @@ -56,6 +57,7 @@ import jakarta.ws.rs.PUT; import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; @@ -118,6 +120,7 @@ import jakarta.persistence.Query; import jakarta.ws.rs.QueryParam; import jakarta.ws.rs.WebApplicationException; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.StreamingOutput; /** @@ -516,10 +519,11 @@ public Response publishDataverseAsCreator(@PathParam("id") long id) { @Deprecated @GET + @AuthRequired @Path("authenticatedUsers") - public Response listAuthenticatedUsers() { + public Response listAuthenticatedUsers(@Context ContainerRequestContext crc) { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -534,22 +538,18 @@ public Response listAuthenticatedUsers() { } @GET + @AuthRequired @Path(listUsersPartialAPIPath) @Produces({ "application/json" }) public Response filterAuthenticatedUsers( + @Context ContainerRequestContext crc, @QueryParam("searchTerm") String searchTerm, @QueryParam("selectedPage") Integer selectedPage, @QueryParam("itemsPerPage") Integer itemsPerPage, @QueryParam("sortKey") String sortKey ) { - User authUser; - try { - authUser = this.findUserOrDie(); - } catch (AbstractApiBean.WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, - BundleUtil.getStringFromBundle("dashboard.list_users.api.auth.invalid_apikey")); - } + User authUser = getRequestUser(crc); if (!authUser.isSuperuser()) { return error(Response.Status.FORBIDDEN, @@ -602,11 +602,12 @@ public Response createAuthenicatedUser(JsonObject jsonObject) { * Shib-specfic one. */ @PUT + @AuthRequired @Path("authenticatedUsers/id/{id}/convertShibToBuiltIn") @Deprecated - public Response convertShibUserToBuiltin(@PathParam("id") Long id, String newEmailAddress) { - try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + public Response convertShibUserToBuiltin(@Context ContainerRequestContext crc, @PathParam("id") Long id, String newEmailAddress) { + try { + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -639,10 +640,11 @@ public Response convertShibUserToBuiltin(@PathParam("id") Long id, String newEma } @PUT + @AuthRequired @Path("authenticatedUsers/id/{id}/convertRemoteToBuiltIn") - public Response convertOAuthUserToBuiltin(@PathParam("id") Long id, String newEmailAddress) { - try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + public Response convertOAuthUserToBuiltin(@Context ContainerRequestContext crc, @PathParam("id") Long id, String newEmailAddress) { + try { + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -679,12 +681,13 @@ public Response convertOAuthUserToBuiltin(@PathParam("id") Long id, String newEm * This is used in testing via AdminIT.java but we don't expect sysadmins to use * this. */ - @Path("authenticatedUsers/convert/builtin2shib") @PUT - public Response builtin2shib(String content) { + @AuthRequired + @Path("authenticatedUsers/convert/builtin2shib") + public Response builtin2shib(@Context ContainerRequestContext crc, String content) { logger.info("entering builtin2shib..."); try { - AuthenticatedUser userToRunThisMethod = findAuthenticatedUserOrDie(); + AuthenticatedUser userToRunThisMethod = getRequestAuthenticatedUserOrDie(crc); if (!userToRunThisMethod.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -829,12 +832,13 @@ public Response builtin2shib(String content) { * This is used in testing via AdminIT.java but we don't expect sysadmins to use * this. */ - @Path("authenticatedUsers/convert/builtin2oauth") @PUT - public Response builtin2oauth(String content) { + @AuthRequired + @Path("authenticatedUsers/convert/builtin2oauth") + public Response builtin2oauth(@Context ContainerRequestContext crc, String content) { logger.info("entering builtin2oauth..."); try { - AuthenticatedUser userToRunThisMethod = findAuthenticatedUserOrDie(); + AuthenticatedUser userToRunThisMethod = getRequestAuthenticatedUserOrDie(crc); if (!userToRunThisMethod.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -1009,14 +1013,15 @@ public Response listBuiltinRoles() { } @DELETE + @AuthRequired @Path("roles/{id}") - public Response deleteRole(@PathParam("id") String id) { + public Response deleteRole(@Context ContainerRequestContext crc, @PathParam("id") String id) { return response(req -> { DataverseRole doomed = findRoleOrDie(id); execCommand(new DeleteRoleCommand(req, doomed)); return ok("role " + doomed.getName() + " deleted."); - }); + }, getRequestUser(crc)); } @Path("superuser/{identifier}") @@ -1323,23 +1328,20 @@ public Response convertUserFromBcryptToSha1(String json) { } @Path("permissions/{dvo}") + @AuthRequired @GET - public Response findPermissonsOn(@PathParam("dvo") String dvo) { + public Response findPermissonsOn(@Context ContainerRequestContext crc, @PathParam("dvo") String dvo) { try { DvObject dvObj = findDvo(dvo); if (dvObj == null) { return notFound("DvObject " + dvo + " not found"); } - try { - User aUser = findUserOrDie(); - JsonObjectBuilder bld = Json.createObjectBuilder(); - bld.add("user", aUser.getIdentifier()); - bld.add("permissions", json(permissionSvc.permissionsFor(createDataverseRequest(aUser), dvObj))); - return ok(bld); - - } catch (WrappedResponse wr) { - return wr.getResponse(); - } + User aUser = getRequestUser(crc); + JsonObjectBuilder bld = Json.createObjectBuilder(); + bld.add("user", aUser.getIdentifier()); + bld.add("permissions", json(permissionSvc.permissionsFor(createDataverseRequest(aUser), dvObj))); + return ok(bld); + } catch (Exception e) { logger.log(Level.SEVERE, "Error while testing permissions", e); return error(Response.Status.INTERNAL_SERVER_ERROR, e.getMessage()); @@ -1465,8 +1467,9 @@ public Response isOrcidEnabled() { } @POST + @AuthRequired @Path("{id}/reregisterHDLToPID") - public Response reregisterHdlToPID(@PathParam("id") String id) { + public Response reregisterHdlToPID(@Context ContainerRequestContext crc, @PathParam("id") String id) { logger.info("Starting to reregister " + id + " Dataset Id. (from hdl to doi)" + new Date()); try { if (settingsSvc.get(SettingsServiceBean.Key.Protocol.toString()).equals(GlobalId.HDL_PROTOCOL)) { @@ -1474,7 +1477,7 @@ public Response reregisterHdlToPID(@PathParam("id") String id) { return error(Status.BAD_REQUEST, BundleUtil.getStringFromBundle("admin.api.migrateHDL.failure.must.be.set.for.doi")); } - User u = findUserOrDie(); + User u = getRequestUser(crc); if (!u.isSuperuser()) { logger.info("Bad Request Unauthor " ); return error(Status.UNAUTHORIZED, BundleUtil.getStringFromBundle("admin.api.auth.mustBeSuperUser")); @@ -1501,12 +1504,13 @@ public Response reregisterHdlToPID(@PathParam("id") String id) { } @GET + @AuthRequired @Path("{id}/registerDataFile") - public Response registerDataFile(@PathParam("id") String id) { + public Response registerDataFile(@Context ContainerRequestContext crc, @PathParam("id") String id) { logger.info("Starting to register " + id + " file id. " + new Date()); try { - User u = findUserOrDie(); + User u = getRequestUser(crc); DataverseRequest r = createDataverseRequest(u); DataFile df = findDataFileOrDie(id); if (df.getIdentifier() == null || df.getIdentifier().isEmpty()) { @@ -1524,8 +1528,9 @@ public Response registerDataFile(@PathParam("id") String id) { } @GET + @AuthRequired @Path("/registerDataFileAll") - public Response registerDataFileAll() { + public Response registerDataFileAll(@Context ContainerRequestContext crc) { Integer count = fileService.findAll().size(); Integer successes = 0; Integer alreadyRegistered = 0; @@ -1538,7 +1543,7 @@ public Response registerDataFileAll() { if ((df.getIdentifier() == null || df.getIdentifier().isEmpty())) { if (df.isReleased()) { released++; - User u = findAuthenticatedUserOrDie(); + User u = getRequestAuthenticatedUserOrDie(crc); DataverseRequest r = createDataverseRequest(u); execCommand(new RegisterDvObjectCommand(r, df)); successes++; @@ -1573,8 +1578,9 @@ public Response registerDataFileAll() { } @GET + @AuthRequired @Path("/updateHashValues/{alg}") - public Response updateHashValues(@PathParam("alg") String alg, @QueryParam("num") int num) { + public Response updateHashValues(@Context ContainerRequestContext crc, @PathParam("alg") String alg, @QueryParam("num") int num) { Integer count = fileService.findAll().size(); Integer successes = 0; Integer alreadyUpdated = 0; @@ -1593,7 +1599,7 @@ public Response updateHashValues(@PathParam("alg") String alg, @QueryParam("num" logger.info("Hashes not created with " + alg + " will be verified, and, if valid, replaced with a hash using " + alg); try { - User u = findAuthenticatedUserOrDie(); + User u = getRequestAuthenticatedUserOrDie(crc); if (!u.isSuperuser()) return error(Status.UNAUTHORIZED, "must be superuser"); } catch (WrappedResponse e1) { @@ -1681,11 +1687,12 @@ public Response updateHashValues(@PathParam("alg") String alg, @QueryParam("num" } @POST + @AuthRequired @Path("/computeDataFileHashValue/{fileId}/algorithm/{alg}") - public Response computeDataFileHashValue(@PathParam("fileId") String fileId, @PathParam("alg") String alg) { + public Response computeDataFileHashValue(@Context ContainerRequestContext crc, @PathParam("fileId") String fileId, @PathParam("alg") String alg) { try { - User u = findAuthenticatedUserOrDie(); + User u = getRequestAuthenticatedUserOrDie(crc); if (!u.isSuperuser()) { return error(Status.UNAUTHORIZED, "must be superuser"); } @@ -1742,11 +1749,12 @@ public Response computeDataFileHashValue(@PathParam("fileId") String fileId, @Pa } @POST + @AuthRequired @Path("/validateDataFileHashValue/{fileId}") - public Response validateDataFileHashValue(@PathParam("fileId") String fileId) { + public Response validateDataFileHashValue(@Context ContainerRequestContext crc, @PathParam("fileId") String fileId) { try { - User u = findAuthenticatedUserOrDie(); + User u = getRequestAuthenticatedUserOrDie(crc); if (!u.isSuperuser()) { return error(Status.UNAUTHORIZED, "must be superuser"); } @@ -1808,12 +1816,13 @@ public Response validateDataFileHashValue(@PathParam("fileId") String fileId) { } @POST + @AuthRequired @Path("/submitDatasetVersionToArchive/{id}/{version}") - public Response submitDatasetVersionToArchive(@PathParam("id") String dsid, + public Response submitDatasetVersionToArchive(@Context ContainerRequestContext crc, @PathParam("id") String dsid, @PathParam("version") String versionNumber) { try { - AuthenticatedUser au = findAuthenticatedUserOrDie(); + AuthenticatedUser au = getRequestAuthenticatedUserOrDie(crc); Dataset ds = findDatasetOrDie(dsid); @@ -1880,11 +1889,12 @@ public void run() { * @return */ @POST + @AuthRequired @Path("/archiveAllUnarchivedDatasetVersions") - public Response archiveAllUnarchivedDatasetVersions(@QueryParam("listonly") boolean listonly, @QueryParam("limit") Integer limit, @QueryParam("latestonly") boolean latestonly) { + public Response archiveAllUnarchivedDatasetVersions(@Context ContainerRequestContext crc, @QueryParam("listonly") boolean listonly, @QueryParam("limit") Integer limit, @QueryParam("latestonly") boolean latestonly) { try { - AuthenticatedUser au = findAuthenticatedUserOrDie(); + AuthenticatedUser au = getRequestAuthenticatedUserOrDie(crc); List dsl = datasetversionService.getUnarchivedDatasetVersions(); if (dsl != null) { @@ -1978,14 +1988,15 @@ public Response clearMetricsCacheByName(@PathParam("name") String name) { } @GET + @AuthRequired @Path("/dataverse/{alias}/addRoleAssignmentsToChildren") - public Response addRoleAssignementsToChildren(@PathParam("alias") String alias) throws WrappedResponse { + public Response addRoleAssignementsToChildren(@Context ContainerRequestContext crc, @PathParam("alias") String alias) throws WrappedResponse { Dataverse owner = dataverseSvc.findByAlias(alias); if (owner == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); } try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2008,14 +2019,15 @@ public Response addRoleAssignementsToChildren(@PathParam("alias") String alias) } @GET + @AuthRequired @Path("/dataverse/{alias}/storageDriver") - public Response getStorageDriver(@PathParam("alias") String alias) throws WrappedResponse { + public Response getStorageDriver(@Context ContainerRequestContext crc, @PathParam("alias") String alias) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); } try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2027,14 +2039,15 @@ public Response getStorageDriver(@PathParam("alias") String alias) throws Wrappe } @PUT + @AuthRequired @Path("/dataverse/{alias}/storageDriver") - public Response setStorageDriver(@PathParam("alias") String alias, String label) throws WrappedResponse { + public Response setStorageDriver(@Context ContainerRequestContext crc, @PathParam("alias") String alias, String label) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); } try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2052,14 +2065,15 @@ public Response setStorageDriver(@PathParam("alias") String alias, String label) } @DELETE + @AuthRequired @Path("/dataverse/{alias}/storageDriver") - public Response resetStorageDriver(@PathParam("alias") String alias) throws WrappedResponse { + public Response resetStorageDriver(@Context ContainerRequestContext crc, @PathParam("alias") String alias) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); } try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2071,10 +2085,11 @@ public Response resetStorageDriver(@PathParam("alias") String alias) throws Wrap } @GET + @AuthRequired @Path("/dataverse/storageDrivers") - public Response listStorageDrivers() throws WrappedResponse { + public Response listStorageDrivers(@Context ContainerRequestContext crc) throws WrappedResponse { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2087,14 +2102,15 @@ public Response listStorageDrivers() throws WrappedResponse { } @GET + @AuthRequired @Path("/dataverse/{alias}/curationLabelSet") - public Response getCurationLabelSet(@PathParam("alias") String alias) throws WrappedResponse { + public Response getCurationLabelSet(@Context ContainerRequestContext crc, @PathParam("alias") String alias) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); } try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2108,14 +2124,15 @@ public Response getCurationLabelSet(@PathParam("alias") String alias) throws Wra } @PUT + @AuthRequired @Path("/dataverse/{alias}/curationLabelSet") - public Response setCurationLabelSet(@PathParam("alias") String alias, @QueryParam("name") String name) throws WrappedResponse { + public Response setCurationLabelSet(@Context ContainerRequestContext crc, @PathParam("alias") String alias, @QueryParam("name") String name) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); } try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2138,14 +2155,15 @@ public Response setCurationLabelSet(@PathParam("alias") String alias, @QueryPara } @DELETE + @AuthRequired @Path("/dataverse/{alias}/curationLabelSet") - public Response resetCurationLabelSet(@PathParam("alias") String alias) throws WrappedResponse { + public Response resetCurationLabelSet(@Context ContainerRequestContext crc, @PathParam("alias") String alias) throws WrappedResponse { Dataverse dataverse = dataverseSvc.findByAlias(alias); if (dataverse == null) { return error(Response.Status.NOT_FOUND, "Could not find dataverse based on alias supplied: " + alias + "."); } try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2157,10 +2175,11 @@ public Response resetCurationLabelSet(@PathParam("alias") String alias) throws W } @GET + @AuthRequired @Path("/dataverse/curationLabelSets") - public Response listCurationLabelSets() throws WrappedResponse { + public Response listCurationLabelSets(@Context ContainerRequestContext crc) throws WrappedResponse { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -2251,12 +2270,13 @@ public Response getBannerMessages(@PathParam("id") Long id) throws WrappedRespon } @POST + @AuthRequired @Consumes("application/json") @Path("/requestSignedUrl") - public Response getSignedUrl(JsonObject urlInfo) { + public Response getSignedUrl(@Context ContainerRequestContext crc, JsonObject urlInfo) { AuthenticatedUser superuser = null; try { - superuser = findAuthenticatedUserOrDie(); + superuser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse wr) { return wr.getResponse(); } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/ApiConfiguration.java b/src/main/java/edu/harvard/iq/dataverse/api/ApiConfiguration.java index 85a9a818378..65f0c3e744e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/ApiConfiguration.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/ApiConfiguration.java @@ -4,6 +4,8 @@ import org.glassfish.jersey.media.multipart.MultiPartFeature; import org.glassfish.jersey.server.ResourceConfig; +import edu.harvard.iq.dataverse.api.auth.AuthFilter; + @ApplicationPath("api/v1") public class ApiConfiguration extends ResourceConfig { @@ -11,9 +13,6 @@ public ApiConfiguration() { packages("edu.harvard.iq.dataverse.api"); packages("edu.harvard.iq.dataverse.mydata"); register(MultiPartFeature.class); + register(AuthFilter.class); } } -/* -public class ApiConfiguration extends ResourceConfi { -} -*/ \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/api/ApiConstants.java b/src/main/java/edu/harvard/iq/dataverse/api/ApiConstants.java new file mode 100644 index 00000000000..296869762da --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/ApiConstants.java @@ -0,0 +1,15 @@ +package edu.harvard.iq.dataverse.api; + +public final class ApiConstants { + + private ApiConstants() { + // Restricting instantiation + } + + // Statuses + public static final String STATUS_OK = "OK"; + public static final String STATUS_ERROR = "ERROR"; + + // Authentication + public static final String CONTAINER_REQUEST_CONTEXT_USER = "user"; +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java b/src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java index c1d54284406..a2d06bff93e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/BatchImport.java @@ -1,5 +1,6 @@ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.api.imports.ImportServiceBean; import edu.harvard.iq.dataverse.DatasetFieldServiceBean; import edu.harvard.iq.dataverse.DatasetServiceBean; @@ -9,6 +10,7 @@ import edu.harvard.iq.dataverse.api.imports.ImportException; import edu.harvard.iq.dataverse.api.imports.ImportUtil.ImportType; +import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import java.io.IOException; @@ -20,6 +22,8 @@ import jakarta.ws.rs.POST; import jakarta.ws.rs.Path; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; @Stateless @@ -42,10 +46,14 @@ public class BatchImport extends AbstractApiBean { BatchServiceBean batchService; @GET + @AuthRequired @Path("harvest") - public Response harvest(@QueryParam("path") String fileDir, @QueryParam("dv") String parentIdtf, @QueryParam("createDV") Boolean createDV, @QueryParam("key") String apiKey) throws IOException { - return startBatchJob(fileDir, parentIdtf, apiKey, ImportType.HARVEST, createDV); - + public Response harvest(@Context ContainerRequestContext crc, @QueryParam("path") String fileDir, @QueryParam("dv") String parentIdtf, @QueryParam("createDV") Boolean createDV, @QueryParam("key") String apiKey) throws IOException { + try { + return startBatchJob(getRequestAuthenticatedUserOrDie(crc), fileDir, parentIdtf, apiKey, ImportType.HARVEST, createDV); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } } /** @@ -57,12 +65,13 @@ public Response harvest(@QueryParam("path") String fileDir, @QueryParam("dv") St * @return import status (including id of the dataset created) */ @POST + @AuthRequired @Path("import") - public Response postImport(String body, @QueryParam("dv") String parentIdtf, @QueryParam("key") String apiKey) { + public Response postImport(@Context ContainerRequestContext crc, String body, @QueryParam("dv") String parentIdtf, @QueryParam("key") String apiKey) { DataverseRequest dataverseRequest; try { - dataverseRequest = createDataverseRequest(findAuthenticatedUserOrDie()); + dataverseRequest = createDataverseRequest(getRequestAuthenticatedUserOrDie(crc)); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -94,24 +103,23 @@ public Response postImport(String body, @QueryParam("dv") String parentIdtf, @Qu * @return import status (including id's of the datasets created) */ @GET + @AuthRequired @Path("import") - public Response getImport(@QueryParam("path") String fileDir, @QueryParam("dv") String parentIdtf, @QueryParam("createDV") Boolean createDV, @QueryParam("key") String apiKey) { - - return startBatchJob(fileDir, parentIdtf, apiKey, ImportType.NEW, createDV); - + public Response getImport(@Context ContainerRequestContext crc, @QueryParam("path") String fileDir, @QueryParam("dv") String parentIdtf, @QueryParam("createDV") Boolean createDV, @QueryParam("key") String apiKey) { + try { + return startBatchJob(getRequestAuthenticatedUserOrDie(crc), fileDir, parentIdtf, apiKey, ImportType.NEW, createDV); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } } - private Response startBatchJob(String fileDir, String parentIdtf, String apiKey, ImportType importType, Boolean createDV) { + private Response startBatchJob(User user, String fileDir, String parentIdtf, String apiKey, ImportType importType, Boolean createDV) { if (createDV == null) { createDV = Boolean.FALSE; } try { DataverseRequest dataverseRequest; - try { - dataverseRequest = createDataverseRequest(findAuthenticatedUserOrDie()); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } + dataverseRequest = createDataverseRequest(user); if (parentIdtf == null) { parentIdtf = "root"; } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/BuiltinUsers.java b/src/main/java/edu/harvard/iq/dataverse/api/BuiltinUsers.java index 723681210d7..50862bc0d35 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/BuiltinUsers.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/BuiltinUsers.java @@ -3,6 +3,7 @@ import edu.harvard.iq.dataverse.Dataverse; import edu.harvard.iq.dataverse.UserNotification; import edu.harvard.iq.dataverse.actionlogging.ActionLogRecord; +import edu.harvard.iq.dataverse.api.auth.ApiKeyAuthMechanism; import edu.harvard.iq.dataverse.authorization.UserRecordIdentifier; import edu.harvard.iq.dataverse.authorization.providers.builtin.BuiltinAuthenticationProvider; import edu.harvard.iq.dataverse.authorization.providers.builtin.BuiltinUser; @@ -209,17 +210,14 @@ private Response internalSave(BuiltinUser user, String password, String key, Boo } } + /*** + * This method was moved here from AbstractApiBean during the filter-based auth + * refactoring, in order to preserve the existing BuiltinUsers endpoints behavior. + * + * @param apiKey from request + * @return error Response + */ + private Response badApiKey(String apiKey) { + return error(Status.UNAUTHORIZED, (apiKey != null) ? "Bad api key " : "Please provide a key query parameter (?key=XXX) or via the HTTP header " + ApiKeyAuthMechanism.DATAVERSE_API_KEY_REQUEST_HEADER_NAME); + } } - - - - - - - - - - - - - diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java index c654f97d887..e05d02e12b3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Datasets.java @@ -3,6 +3,7 @@ import edu.harvard.iq.dataverse.*; import edu.harvard.iq.dataverse.DatasetLock.Reason; import edu.harvard.iq.dataverse.actionlogging.ActionLogRecord; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.authorization.Permission; @@ -67,6 +68,7 @@ import edu.harvard.iq.dataverse.dataaccess.DataAccess; import edu.harvard.iq.dataverse.dataaccess.ImageThumbConverter; import edu.harvard.iq.dataverse.dataaccess.S3AccessIO; +import edu.harvard.iq.dataverse.dataaccess.StorageIO; import edu.harvard.iq.dataverse.engine.command.exception.CommandException; import edu.harvard.iq.dataverse.engine.command.exception.UnforcedCommandException; import edu.harvard.iq.dataverse.engine.command.impl.GetDatasetStorageSizeCommand; @@ -93,6 +95,7 @@ import edu.harvard.iq.dataverse.util.json.JsonParseException; import edu.harvard.iq.dataverse.util.json.JsonUtil; import edu.harvard.iq.dataverse.search.IndexServiceBean; + import static edu.harvard.iq.dataverse.util.json.JsonPrinter.*; import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; @@ -114,11 +117,13 @@ import java.time.LocalDateTime; import java.util.*; import java.util.concurrent.*; +import java.util.function.Predicate; import java.time.ZoneId; import java.time.format.DateTimeFormatter; import java.util.Map.Entry; import java.util.logging.Level; import java.util.logging.Logger; +import java.util.regex.Pattern; import java.util.stream.Collectors; import jakarta.ejb.EJB; @@ -147,6 +152,7 @@ import jakarta.ws.rs.PathParam; import jakarta.ws.rs.Produces; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.HttpHeaders; import jakarta.ws.rs.core.MediaType; @@ -166,6 +172,7 @@ public class Datasets extends AbstractApiBean { private static final Logger logger = Logger.getLogger(Datasets.class.getCanonicalName()); + private static final Pattern dataFilePattern = Pattern.compile("^[0-9a-f]{11}-[0-9a-f]{12}\\.?.*"); @Inject DataverseSession session; @@ -248,8 +255,9 @@ public interface DsVersionHandler { } @GET + @AuthRequired @Path("{id}") - public Response getDataset(@PathParam("id") String id, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) { + public Response getDataset(@Context ContainerRequestContext crc, @PathParam("id") String id, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) { return response( req -> { final Dataset retrieved = execCommand(new GetDatasetCommand(req, findDatasetOrDie(id))); final DatasetVersion latest = execCommand(new GetLatestAccessibleDatasetVersionCommand(req, retrieved)); @@ -260,7 +268,7 @@ public Response getDataset(@PathParam("id") String id, @Context UriInfo uriInfo, mdcLogService.logEntry(entry); } return ok(jsonbuilder.add("latestVersion", (latest != null) ? json(latest) : null)); - }); + }, getRequestUser(crc)); } // TODO: @@ -300,8 +308,9 @@ public Response exportDataset(@QueryParam("persistentId") String persistentId, @ } @DELETE + @AuthRequired @Path("{id}") - public Response deleteDataset( @PathParam("id") String id) { + public Response deleteDataset(@Context ContainerRequestContext crc, @PathParam("id") String id) { // Internally, "DeleteDatasetCommand" simply redirects to "DeleteDatasetVersionCommand" // (and there's a comment that says "TODO: remove this command") // do we need an exposed API call for it? @@ -311,11 +320,11 @@ public Response deleteDataset( @PathParam("id") String id) { // "destroyDataset" API calls. // (The logic below follows the current implementation of the underlying // commands!) - + + User u = getRequestUser(crc); return response( req -> { Dataset doomed = findDatasetOrDie(id); DatasetVersion doomedVersion = doomed.getLatestVersion(); - User u = findUserOrDie(); boolean destroy = false; if (doomed.getVersions().size() == 1) { @@ -345,17 +354,18 @@ public Response deleteDataset( @PathParam("id") String id) { } return ok("Dataset " + id + " deleted"); - }); + }, u); } @DELETE + @AuthRequired @Path("{id}/destroy") - public Response destroyDataset(@PathParam("id") String id) { + public Response destroyDataset(@Context ContainerRequestContext crc, @PathParam("id") String id) { + User u = getRequestUser(crc); return response(req -> { // first check if dataset is released, and if so, if user is a superuser Dataset doomed = findDatasetOrDie(id); - User u = findUserOrDie(); if (doomed.isReleased() && (!(u instanceof AuthenticatedUser) || !u.isSuperuser())) { throw new WrappedResponse(error(Response.Status.UNAUTHORIZED, "Destroy can only be called by superusers.")); @@ -377,12 +387,13 @@ public Response destroyDataset(@PathParam("id") String id) { } return ok("Dataset " + id + " destroyed"); - }); + }, u); } @DELETE + @AuthRequired @Path("{id}/versions/{versionId}") - public Response deleteDraftVersion( @PathParam("id") String id, @PathParam("versionId") String versionId ){ + public Response deleteDraftVersion(@Context ContainerRequestContext crc, @PathParam("id") String id, @PathParam("versionId") String versionId ){ if ( ! ":draft".equals(versionId) ) { return badRequest("Only the :draft version can be deleted"); } @@ -414,22 +425,24 @@ public Response deleteDraftVersion( @PathParam("id") String id, @PathParam("ver } return ok("Draft version of dataset " + id + " deleted"); - }); + }, getRequestUser(crc)); } @DELETE + @AuthRequired @Path("{datasetId}/deleteLink/{linkedDataverseId}") - public Response deleteDatasetLinkingDataverse( @PathParam("datasetId") String datasetId, @PathParam("linkedDataverseId") String linkedDataverseId) { + public Response deleteDatasetLinkingDataverse(@Context ContainerRequestContext crc, @PathParam("datasetId") String datasetId, @PathParam("linkedDataverseId") String linkedDataverseId) { boolean index = true; return response(req -> { execCommand(new DeleteDatasetLinkingDataverseCommand(req, findDatasetOrDie(datasetId), findDatasetLinkingDataverseOrDie(datasetId, linkedDataverseId), index)); return ok("Link from Dataset " + datasetId + " to linked Dataverse " + linkedDataverseId + " deleted"); - }); + }, getRequestUser(crc)); } @PUT + @AuthRequired @Path("{id}/citationdate") - public Response setCitationDate( @PathParam("id") String id, String dsfTypeName) { + public Response setCitationDate(@Context ContainerRequestContext crc, @PathParam("id") String id, String dsfTypeName) { return response( req -> { if ( dsfTypeName.trim().isEmpty() ){ return badRequest("Please provide a dataset field type in the requst body."); @@ -444,56 +457,61 @@ public Response setCitationDate( @PathParam("id") String id, String dsfTypeName) execCommand(new SetDatasetCitationDateCommand(req, findDatasetOrDie(id), dsfType)); return ok("Citation Date for dataset " + id + " set to: " + (dsfType != null ? dsfType.getDisplayName() : "default")); - }); + }, getRequestUser(crc)); } @DELETE + @AuthRequired @Path("{id}/citationdate") - public Response useDefaultCitationDate( @PathParam("id") String id) { + public Response useDefaultCitationDate(@Context ContainerRequestContext crc, @PathParam("id") String id) { return response( req -> { execCommand(new SetDatasetCitationDateCommand(req, findDatasetOrDie(id), null)); return ok("Citation Date for dataset " + id + " set to default"); - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/versions") - public Response listVersions( @PathParam("id") String id ) { + public Response listVersions(@Context ContainerRequestContext crc, @PathParam("id") String id ) { return response( req -> ok( execCommand( new ListVersionsCommand(req, findDatasetOrDie(id)) ) .stream() .map( d -> json(d) ) - .collect(toJsonArray()))); + .collect(toJsonArray())), getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/versions/{versionId}") - public Response getVersion( @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { + public Response getVersion(@Context ContainerRequestContext crc, @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { return response( req -> { DatasetVersion dsv = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); return (dsv == null || dsv.getId() == null) ? notFound("Dataset version not found") : ok(json(dsv)); - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/versions/{versionId}/files") - public Response getVersionFiles( @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { + public Response getVersionFiles(@Context ContainerRequestContext crc, @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { return response( req -> ok( jsonFileMetadatas( - getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers).getFileMetadatas()))); + getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers).getFileMetadatas())), getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/dirindex") @Produces("text/html") - public Response getFileAccessFolderView(@PathParam("id") String datasetId, @QueryParam("version") String versionId, @QueryParam("folder") String folderName, @QueryParam("original") Boolean originals, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) { + public Response getFileAccessFolderView(@Context ContainerRequestContext crc, @PathParam("id") String datasetId, @QueryParam("version") String versionId, @QueryParam("folder") String folderName, @QueryParam("original") Boolean originals, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) { folderName = folderName == null ? "" : folderName; versionId = versionId == null ? ":latest-published" : versionId; DatasetVersion version; try { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); version = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -520,21 +538,24 @@ public Response getFileAccessFolderView(@PathParam("id") String datasetId, @Quer } @GET + @AuthRequired @Path("{id}/versions/{versionId}/metadata") - public Response getVersionMetadata( @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { + public Response getVersionMetadata(@Context ContainerRequestContext crc, @PathParam("id") String datasetId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { return response( req -> ok( jsonByBlocks( getDatasetVersionOrDie(req, versionId, findDatasetOrDie(datasetId), uriInfo, headers ) - .getDatasetFields()))); + .getDatasetFields())), getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/versions/{versionNumber}/metadata/{block}") - public Response getVersionMetadataBlock( @PathParam("id") String datasetId, - @PathParam("versionNumber") String versionNumber, - @PathParam("block") String blockName, - @Context UriInfo uriInfo, - @Context HttpHeaders headers ) { + public Response getVersionMetadataBlock(@Context ContainerRequestContext crc, + @PathParam("id") String datasetId, + @PathParam("versionNumber") String versionNumber, + @PathParam("block") String blockName, + @Context UriInfo uriInfo, + @Context HttpHeaders headers) { return response( req -> { DatasetVersion dsv = getDatasetVersionOrDie(req, versionNumber, findDatasetOrDie(datasetId), uriInfo, headers ); @@ -546,21 +567,23 @@ public Response getVersionMetadataBlock( @PathParam("id") String datasetId, } } return notFound("metadata block named " + blockName + " not found"); - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/modifyRegistration") - public Response updateDatasetTargetURL(@PathParam("id") String id ) { + public Response updateDatasetTargetURL(@Context ContainerRequestContext crc, @PathParam("id") String id ) { return response( req -> { execCommand(new UpdateDatasetTargetURLCommand(findDatasetOrDie(id), req)); return ok("Dataset " + id + " target url updated"); - }); + }, getRequestUser(crc)); } @POST + @AuthRequired @Path("/modifyRegistrationAll") - public Response updateDatasetTargetURLAll() { + public Response updateDatasetTargetURLAll(@Context ContainerRequestContext crc) { return response( req -> { datasetService.findAll().forEach( ds -> { try { @@ -570,12 +593,13 @@ public Response updateDatasetTargetURLAll() { } }); return ok("Update All Dataset target url completed"); - }); + }, getRequestUser(crc)); } @POST + @AuthRequired @Path("{id}/modifyRegistrationMetadata") - public Response updateDatasetPIDMetadata(@PathParam("id") String id) { + public Response updateDatasetPIDMetadata(@Context ContainerRequestContext crc, @PathParam("id") String id) { try { Dataset dataset = findDatasetOrDie(id); @@ -590,12 +614,13 @@ public Response updateDatasetPIDMetadata(@PathParam("id") String id) { execCommand(new UpdateDvObjectPIDMetadataCommand(findDatasetOrDie(id), req)); List args = Arrays.asList(id); return ok(BundleUtil.getStringFromBundle("datasets.api.updatePIDMetadata.success.for.single.dataset", args)); - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("/modifyRegistrationPIDMetadataAll") - public Response updateDatasetPIDMetadataAll() { + public Response updateDatasetPIDMetadataAll(@Context ContainerRequestContext crc) { return response( req -> { datasetService.findAll().forEach( ds -> { try { @@ -605,20 +630,21 @@ public Response updateDatasetPIDMetadataAll() { } }); return ok(BundleUtil.getStringFromBundle("datasets.api.updatePIDMetadata.success.for.update.all")); - }); + }, getRequestUser(crc)); } @PUT + @AuthRequired @Path("{id}/versions/{versionId}") @Consumes(MediaType.APPLICATION_JSON) - public Response updateDraftVersion( String jsonBody, @PathParam("id") String id, @PathParam("versionId") String versionId ){ + public Response updateDraftVersion(@Context ContainerRequestContext crc, String jsonBody, @PathParam("id") String id, @PathParam("versionId") String versionId){ if ( ! ":draft".equals(versionId) ) { return error( Response.Status.BAD_REQUEST, "Only the :draft version can be updated"); } try ( StringReader rdr = new StringReader(jsonBody) ) { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); Dataset ds = findDatasetOrDie(id); JsonObject json = Json.createReader(rdr).readObject(); DatasetVersion incomingVersion = jsonParser().parseDatasetVersion(json); @@ -674,11 +700,12 @@ public Response updateDraftVersion( String jsonBody, @PathParam("id") String id, } @GET + @AuthRequired @Path("{id}/versions/{versionId}/metadata") @Produces("application/ld+json, application/json-ld") - public Response getVersionJsonLDMetadata(@PathParam("id") String id, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { + public Response getVersionJsonLDMetadata(@Context ContainerRequestContext crc, @PathParam("id") String id, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers) { try { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); DatasetVersion dsv = getDatasetVersionOrDie(req, versionId, findDatasetOrDie(id), uriInfo, headers); OREMap ore = new OREMap(dsv, settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false)); @@ -695,20 +722,22 @@ public Response getVersionJsonLDMetadata(@PathParam("id") String id, @PathParam( } @GET + @AuthRequired @Path("{id}/metadata") @Produces("application/ld+json, application/json-ld") - public Response getVersionJsonLDMetadata(@PathParam("id") String id, @Context UriInfo uriInfo, @Context HttpHeaders headers) { - return getVersionJsonLDMetadata(id, ":draft", uriInfo, headers); + public Response getVersionJsonLDMetadata(@Context ContainerRequestContext crc, @PathParam("id") String id, @Context UriInfo uriInfo, @Context HttpHeaders headers) { + return getVersionJsonLDMetadata(crc, id, ":draft", uriInfo, headers); } @PUT + @AuthRequired @Path("{id}/metadata") @Consumes("application/ld+json, application/json-ld") - public Response updateVersionMetadata(String jsonLDBody, @PathParam("id") String id, @DefaultValue("false") @QueryParam("replace") boolean replaceTerms) { + public Response updateVersionMetadata(@Context ContainerRequestContext crc, String jsonLDBody, @PathParam("id") String id, @DefaultValue("false") @QueryParam("replace") boolean replaceTerms) { try { Dataset ds = findDatasetOrDie(id); - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); DatasetVersion dsv = ds.getOrCreateEditVersion(); boolean updateDraft = ds.getLatestVersion().isDraft(); dsv = JSONLDUtil.updateDatasetVersionMDFromJsonLD(dsv, jsonLDBody, metadataBlockService, datasetFieldSvc, !replaceTerms, false, licenseSvc); @@ -736,12 +765,13 @@ public Response updateVersionMetadata(String jsonLDBody, @PathParam("id") String } @PUT + @AuthRequired @Path("{id}/metadata/delete") @Consumes("application/ld+json, application/json-ld") - public Response deleteMetadata(String jsonLDBody, @PathParam("id") String id) { + public Response deleteMetadata(@Context ContainerRequestContext crc, String jsonLDBody, @PathParam("id") String id) { try { Dataset ds = findDatasetOrDie(id); - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); DatasetVersion dsv = ds.getOrCreateEditVersion(); boolean updateDraft = ds.getLatestVersion().isDraft(); dsv = JSONLDUtil.deleteDatasetVersionMDFromJsonLD(dsv, jsonLDBody, metadataBlockService, licenseSvc); @@ -767,10 +797,11 @@ public Response deleteMetadata(String jsonLDBody, @PathParam("id") String id) { } @PUT + @AuthRequired @Path("{id}/deleteMetadata") - public Response deleteVersionMetadata(String jsonBody, @PathParam("id") String id) throws WrappedResponse { + public Response deleteVersionMetadata(@Context ContainerRequestContext crc, String jsonBody, @PathParam("id") String id) throws WrappedResponse { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); return processDatasetFieldDataDelete(jsonBody, id, req); } @@ -922,17 +953,13 @@ private String getCompoundDisplayValue (DatasetFieldCompoundValue dscv){ } @PUT + @AuthRequired @Path("{id}/editMetadata") - public Response editVersionMetadata(String jsonBody, @PathParam("id") String id, @QueryParam("replace") Boolean replace) { + public Response editVersionMetadata(@Context ContainerRequestContext crc, String jsonBody, @PathParam("id") String id, @QueryParam("replace") Boolean replace) { Boolean replaceData = replace != null; DataverseRequest req = null; - try { - req = createDataverseRequest(findUserOrDie()); - } catch (WrappedResponse ex) { - logger.log(Level.SEVERE, "Edit metdata error: " + ex.getMessage(), ex); - return ex.getResponse(); - } + req = createDataverseRequest(getRequestUser(crc)); return processDatasetUpdate(jsonBody, id, req, replaceData); } @@ -1089,22 +1116,24 @@ private String validateDatasetFieldValues(List fields) { * @deprecated This was shipped as a GET but should have been a POST, see https://github.com/IQSS/dataverse/issues/2431 */ @GET + @AuthRequired @Path("{id}/actions/:publish") @Deprecated - public Response publishDataseUsingGetDeprecated( @PathParam("id") String id, @QueryParam("type") String type ) { + public Response publishDataseUsingGetDeprecated(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("type") String type ) { logger.info("publishDataseUsingGetDeprecated called on id " + id + ". Encourage use of POST rather than GET, which is deprecated."); - return publishDataset(id, type, false); + return publishDataset(crc, id, type, false); } @POST + @AuthRequired @Path("{id}/actions/:publish") - public Response publishDataset(@PathParam("id") String id, @QueryParam("type") String type, @QueryParam("assureIsIndexed") boolean mustBeIndexed) { + public Response publishDataset(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("type") String type, @QueryParam("assureIsIndexed") boolean mustBeIndexed) { try { if (type == null) { return error(Response.Status.BAD_REQUEST, "Missing 'type' parameter (either 'major','minor', or 'updatecurrent')."); } boolean updateCurrent=false; - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); type = type.toLowerCase(); boolean isMinor=false; switch (type) { @@ -1205,7 +1234,7 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S return error(Response.Status.INTERNAL_SERVER_ERROR, errorMsg); } else { return Response.ok(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("status_details", successMsg) .add("data", json(ds)).build()) .type(MediaType.APPLICATION_JSON) @@ -1223,11 +1252,12 @@ public Response publishDataset(@PathParam("id") String id, @QueryParam("type") S } @POST + @AuthRequired @Path("{id}/actions/:releasemigrated") @Consumes("application/ld+json, application/json-ld") - public Response publishMigratedDataset(String jsonldBody, @PathParam("id") String id, @DefaultValue("false") @QueryParam ("updatepidatprovider") boolean contactPIDProvider) { + public Response publishMigratedDataset(@Context ContainerRequestContext crc, String jsonldBody, @PathParam("id") String id, @DefaultValue("false") @QueryParam ("updatepidatprovider") boolean contactPIDProvider) { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Only superusers can release migrated datasets"); } @@ -1313,10 +1343,11 @@ public Response publishMigratedDataset(String jsonldBody, @PathParam("id") Strin } @POST + @AuthRequired @Path("{id}/move/{targetDataverseAlias}") - public Response moveDataset(@PathParam("id") String id, @PathParam("targetDataverseAlias") String targetDataverseAlias, @QueryParam("forceMove") Boolean force) { + public Response moveDataset(@Context ContainerRequestContext crc, @PathParam("id") String id, @PathParam("targetDataverseAlias") String targetDataverseAlias, @QueryParam("forceMove") Boolean force) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); Dataset ds = findDatasetOrDie(id); Dataverse target = dataverseService.findByAlias(targetDataverseAlias); if (target == null) { @@ -1337,13 +1368,14 @@ public Response moveDataset(@PathParam("id") String id, @PathParam("targetDatave } @POST + @AuthRequired @Path("{id}/files/actions/:set-embargo") - public Response createFileEmbargo(@PathParam("id") String id, String jsonBody){ + public Response createFileEmbargo(@Context ContainerRequestContext crc, @PathParam("id") String id, String jsonBody){ // user is authenticated AuthenticatedUser authenticatedUser = null; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Status.UNAUTHORIZED, "Authentication is required."); } @@ -1447,7 +1479,7 @@ public Response createFileEmbargo(@PathParam("id") String id, String jsonBody){ } if (badFiles) { return Response.status(Status.FORBIDDEN) - .entity(NullSafeJsonBuilder.jsonObjectBuilder().add("status", STATUS_ERROR) + .entity(NullSafeJsonBuilder.jsonObjectBuilder().add("status", ApiConstants.STATUS_ERROR) .add("message", "You do not have permission to embargo the following files") .add("files", restrictedFiles).build()) .type(MediaType.APPLICATION_JSON_TYPE).build(); @@ -1493,13 +1525,14 @@ public Response createFileEmbargo(@PathParam("id") String id, String jsonBody){ } @POST + @AuthRequired @Path("{id}/files/actions/:unset-embargo") - public Response removeFileEmbargo(@PathParam("id") String id, String jsonBody){ + public Response removeFileEmbargo(@Context ContainerRequestContext crc, @PathParam("id") String id, String jsonBody){ // user is authenticated AuthenticatedUser authenticatedUser = null; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Status.UNAUTHORIZED, "Authentication is required."); } @@ -1566,7 +1599,7 @@ public Response removeFileEmbargo(@PathParam("id") String id, String jsonBody){ } if (badFiles) { return Response.status(Status.FORBIDDEN) - .entity(NullSafeJsonBuilder.jsonObjectBuilder().add("status", STATUS_ERROR) + .entity(NullSafeJsonBuilder.jsonObjectBuilder().add("status", ApiConstants.STATUS_ERROR) .add("message", "The following files do not have embargoes or you do not have permission to remove their embargoes") .add("files", restrictedFiles).build()) .type(MediaType.APPLICATION_JSON_TYPE).build(); @@ -1604,10 +1637,11 @@ public Response removeFileEmbargo(@PathParam("id") String id, String jsonBody){ @PUT + @AuthRequired @Path("{linkedDatasetId}/link/{linkingDataverseAlias}") - public Response linkDataset(@PathParam("linkedDatasetId") String linkedDatasetId, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { + public Response linkDataset(@Context ContainerRequestContext crc, @PathParam("linkedDatasetId") String linkedDatasetId, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); Dataset linked = findDatasetOrDie(linkedDatasetId); Dataverse linking = findDataverseOrDie(linkingDataverseAlias); if (linked == null){ @@ -1648,10 +1682,11 @@ public Response getCustomTermsTab(@PathParam("id") String id, @PathParam("versio @GET + @AuthRequired @Path("{id}/links") - public Response getLinks(@PathParam("id") String idSupplied ) { + public Response getLinks(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied ) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); if (!u.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Not a superuser"); } @@ -1678,8 +1713,9 @@ public Response getLinks(@PathParam("id") String idSupplied ) { * @param apiKey */ @POST + @AuthRequired @Path("{identifier}/assignments") - public Response createAssignment(RoleAssignmentDTO ra, @PathParam("identifier") String id, @QueryParam("key") String apiKey) { + public Response createAssignment(@Context ContainerRequestContext crc, RoleAssignmentDTO ra, @PathParam("identifier") String id, @QueryParam("key") String apiKey) { try { Dataset dataset = findDatasetOrDie(id); @@ -1707,7 +1743,7 @@ public Response createAssignment(RoleAssignmentDTO ra, @PathParam("identifier") String privateUrlToken = null; return ok( - json(execCommand(new AssignRoleCommand(assignee, theRole, dataset, createDataverseRequest(findUserOrDie()), privateUrlToken)))); + json(execCommand(new AssignRoleCommand(assignee, theRole, dataset, createDataverseRequest(getRequestUser(crc)), privateUrlToken)))); } catch (WrappedResponse ex) { List args = Arrays.asList(ex.getMessage()); logger.log(Level.WARNING, BundleUtil.getStringFromBundle("datasets.api.grant.role.cant.create.assignment.error", args)); @@ -1717,13 +1753,14 @@ public Response createAssignment(RoleAssignmentDTO ra, @PathParam("identifier") } @DELETE + @AuthRequired @Path("{identifier}/assignments/{id}") - public Response deleteAssignment(@PathParam("id") long assignmentId, @PathParam("identifier") String dsId) { + public Response deleteAssignment(@Context ContainerRequestContext crc, @PathParam("id") long assignmentId, @PathParam("identifier") String dsId) { RoleAssignment ra = em.find(RoleAssignment.class, assignmentId); if (ra != null) { try { findDatasetOrDie(dsId); - execCommand(new RevokeRoleCommand(ra, createDataverseRequest(findUserOrDie()))); + execCommand(new RevokeRoleCommand(ra, createDataverseRequest(getRequestUser(crc)))); List args = Arrays.asList(ra.getRole().getName(), ra.getAssigneeIdentifier(), ra.getDefinitionPoint().accept(DvObject.NamePrinter)); return ok(BundleUtil.getStringFromBundle("datasets.api.revoke.role.success", args)); } catch (WrappedResponse ex) { @@ -1736,38 +1773,42 @@ public Response deleteAssignment(@PathParam("id") long assignmentId, @PathParam( } @GET + @AuthRequired @Path("{identifier}/assignments") - public Response getAssignments(@PathParam("identifier") String id) { + public Response getAssignments(@Context ContainerRequestContext crc, @PathParam("identifier") String id) { return response(req -> ok(execCommand( new ListRoleAssignments(req, findDatasetOrDie(id))) - .stream().map(ra -> json(ra)).collect(toJsonArray()))); + .stream().map(ra -> json(ra)).collect(toJsonArray())), getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/privateUrl") - public Response getPrivateUrlData(@PathParam("id") String idSupplied) { + public Response getPrivateUrlData(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { return response( req -> { PrivateUrl privateUrl = execCommand(new GetPrivateUrlCommand(req, findDatasetOrDie(idSupplied))); return (privateUrl != null) ? ok(json(privateUrl)) : error(Response.Status.NOT_FOUND, "Private URL not found."); - }); + }, getRequestUser(crc)); } @POST + @AuthRequired @Path("{id}/privateUrl") - public Response createPrivateUrl(@PathParam("id") String idSupplied,@DefaultValue("false") @QueryParam ("anonymizedAccess") boolean anonymizedAccess) { + public Response createPrivateUrl(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied,@DefaultValue("false") @QueryParam ("anonymizedAccess") boolean anonymizedAccess) { if(anonymizedAccess && settingsSvc.getValueForKey(SettingsServiceBean.Key.AnonymizedFieldTypeNames)==null) { throw new NotAcceptableException("Anonymized Access not enabled"); } return response(req -> ok(json(execCommand( - new CreatePrivateUrlCommand(req, findDatasetOrDie(idSupplied), anonymizedAccess))))); + new CreatePrivateUrlCommand(req, findDatasetOrDie(idSupplied), anonymizedAccess)))), getRequestUser(crc)); } @DELETE + @AuthRequired @Path("{id}/privateUrl") - public Response deletePrivateUrl(@PathParam("id") String idSupplied) { + public Response deletePrivateUrl(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { return response( req -> { Dataset dataset = findDatasetOrDie(idSupplied); PrivateUrl privateUrl = execCommand(new GetPrivateUrlCommand(req, dataset)); @@ -1777,20 +1818,17 @@ public Response deletePrivateUrl(@PathParam("id") String idSupplied) { } else { return notFound("No Private URL to delete."); } - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("{id}/thumbnail/candidates") - public Response getDatasetThumbnailCandidates(@PathParam("id") String idSupplied) { + public Response getDatasetThumbnailCandidates(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { try { Dataset dataset = findDatasetOrDie(idSupplied); boolean canUpdateThumbnail = false; - try { - canUpdateThumbnail = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset).canIssue(UpdateDatasetThumbnailCommand.class); - } catch (WrappedResponse ex) { - logger.info("Exception thrown while trying to figure out permissions while getting thumbnail for dataset id " + dataset.getId() + ": " + ex.getLocalizedMessage()); - } + canUpdateThumbnail = permissionSvc.requestOn(createDataverseRequest(getRequestUser(crc)), dataset).canIssue(UpdateDatasetThumbnailCommand.class); if (!canUpdateThumbnail) { return error(Response.Status.FORBIDDEN, "You are not permitted to list dataset thumbnail candidates."); } @@ -1833,10 +1871,11 @@ public Response getDatasetThumbnail(@PathParam("id") String idSupplied) { // TODO: Rather than only supporting looking up files by their database IDs (dataFileIdSupplied), consider supporting persistent identifiers. @POST + @AuthRequired @Path("{id}/thumbnail/{dataFileId}") - public Response setDataFileAsThumbnail(@PathParam("id") String idSupplied, @PathParam("dataFileId") long dataFileIdSupplied) { + public Response setDataFileAsThumbnail(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied, @PathParam("dataFileId") long dataFileIdSupplied) { try { - DatasetThumbnail datasetThumbnail = execCommand(new UpdateDatasetThumbnailCommand(createDataverseRequest(findUserOrDie()), findDatasetOrDie(idSupplied), UpdateDatasetThumbnailCommand.UserIntent.setDatasetFileAsThumbnail, dataFileIdSupplied, null)); + DatasetThumbnail datasetThumbnail = execCommand(new UpdateDatasetThumbnailCommand(createDataverseRequest(getRequestUser(crc)), findDatasetOrDie(idSupplied), UpdateDatasetThumbnailCommand.UserIntent.setDatasetFileAsThumbnail, dataFileIdSupplied, null)); return ok("Thumbnail set to " + datasetThumbnail.getBase64image()); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -1844,12 +1883,12 @@ public Response setDataFileAsThumbnail(@PathParam("id") String idSupplied, @Path } @POST + @AuthRequired @Path("{id}/thumbnail") @Consumes(MediaType.MULTIPART_FORM_DATA) - public Response uploadDatasetLogo(@PathParam("id") String idSupplied, @FormDataParam("file") InputStream inputStream - ) { + public Response uploadDatasetLogo(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied, @FormDataParam("file") InputStream inputStream) { try { - DatasetThumbnail datasetThumbnail = execCommand(new UpdateDatasetThumbnailCommand(createDataverseRequest(findUserOrDie()), findDatasetOrDie(idSupplied), UpdateDatasetThumbnailCommand.UserIntent.setNonDatasetFileAsThumbnail, null, inputStream)); + DatasetThumbnail datasetThumbnail = execCommand(new UpdateDatasetThumbnailCommand(createDataverseRequest(getRequestUser(crc)), findDatasetOrDie(idSupplied), UpdateDatasetThumbnailCommand.UserIntent.setNonDatasetFileAsThumbnail, null, inputStream)); return ok("Thumbnail is now " + datasetThumbnail.getBase64image()); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -1857,10 +1896,11 @@ public Response uploadDatasetLogo(@PathParam("id") String idSupplied, @FormDataP } @DELETE + @AuthRequired @Path("{id}/thumbnail") - public Response removeDatasetLogo(@PathParam("id") String idSupplied) { + public Response removeDatasetLogo(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { try { - DatasetThumbnail datasetThumbnail = execCommand(new UpdateDatasetThumbnailCommand(createDataverseRequest(findUserOrDie()), findDatasetOrDie(idSupplied), UpdateDatasetThumbnailCommand.UserIntent.removeThumbnail, null, null)); + execCommand(new UpdateDatasetThumbnailCommand(createDataverseRequest(getRequestUser(crc)), findDatasetOrDie(idSupplied), UpdateDatasetThumbnailCommand.UserIntent.removeThumbnail, null, null)); return ok("Dataset thumbnail removed."); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -1868,8 +1908,9 @@ public Response removeDatasetLogo(@PathParam("id") String idSupplied) { } @GET + @AuthRequired @Path("{identifier}/dataCaptureModule/rsync") - public Response getRsync(@PathParam("identifier") String id) { + public Response getRsync(@Context ContainerRequestContext crc, @PathParam("identifier") String id) { //TODO - does it make sense to switch this to dataset identifier for consistency with the rest of the DCM APIs? if (!DataCaptureModuleUtil.rsyncSupportEnabled(settingsSvc.getValueForKey(SettingsServiceBean.Key.UploadMethods))) { return error(Response.Status.METHOD_NOT_ALLOWED, SettingsServiceBean.Key.UploadMethods + " does not contain " + SystemConfig.FileUploadMethods.RSYNC + "."); @@ -1877,7 +1918,7 @@ public Response getRsync(@PathParam("identifier") String id) { Dataset dataset = null; try { dataset = findDatasetOrDie(id); - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); ScriptRequestResponse scriptRequestResponse = execCommand(new RequestRsyncScriptCommand(createDataverseRequest(user), dataset)); DatasetLock lock = datasetService.addDatasetLock(dataset.getId(), DatasetLock.Reason.DcmUpload, user.getId(), "script downloaded"); @@ -1908,12 +1949,13 @@ public Response getRsync(@PathParam("identifier") String id) { * -MAD 4.9.1 */ @POST + @AuthRequired @Path("{identifier}/dataCaptureModule/checksumValidation") - public Response receiveChecksumValidationResults(@PathParam("identifier") String id, JsonObject jsonFromDcm) { + public Response receiveChecksumValidationResults(@Context ContainerRequestContext crc, @PathParam("identifier") String id, JsonObject jsonFromDcm) { logger.log(Level.FINE, "jsonFromDcm: {0}", jsonFromDcm); AuthenticatedUser authenticatedUser = null; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Response.Status.BAD_REQUEST, "Authentication is required."); } @@ -1936,7 +1978,7 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String ImportMode importMode = ImportMode.MERGE; try { - JsonObject jsonFromImportJobKickoff = execCommand(new ImportFromFileSystemCommand(createDataverseRequest(findUserOrDie()), dataset, uploadFolder, new Long(totalSize), importMode)); + JsonObject jsonFromImportJobKickoff = execCommand(new ImportFromFileSystemCommand(createDataverseRequest(getRequestUser(crc)), dataset, uploadFolder, new Long(totalSize), importMode)); long jobId = jsonFromImportJobKickoff.getInt("executionId"); String message = jsonFromImportJobKickoff.getString("message"); JsonObjectBuilder job = Json.createObjectBuilder(); @@ -2015,10 +2057,11 @@ public Response receiveChecksumValidationResults(@PathParam("identifier") String @POST + @AuthRequired @Path("{id}/submitForReview") - public Response submitForReview(@PathParam("id") String idSupplied) { + public Response submitForReview(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { try { - Dataset updatedDataset = execCommand(new SubmitDatasetForReviewCommand(createDataverseRequest(findUserOrDie()), findDatasetOrDie(idSupplied))); + Dataset updatedDataset = execCommand(new SubmitDatasetForReviewCommand(createDataverseRequest(getRequestUser(crc)), findDatasetOrDie(idSupplied))); JsonObjectBuilder result = Json.createObjectBuilder(); boolean inReview = updatedDataset.isLockedFor(DatasetLock.Reason.InReview); @@ -2032,9 +2075,10 @@ public Response submitForReview(@PathParam("id") String idSupplied) { } @POST + @AuthRequired @Path("{id}/returnToAuthor") - public Response returnToAuthor(@PathParam("id") String idSupplied, String jsonBody) { - + public Response returnToAuthor(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied, String jsonBody) { + if (jsonBody == null || jsonBody.isEmpty()) { return error(Response.Status.BAD_REQUEST, "You must supply JSON to this API endpoint and it must contain a reason for returning the dataset (field: reasonForReturn)."); } @@ -2048,7 +2092,7 @@ public Response returnToAuthor(@PathParam("id") String idSupplied, String jsonBo if (reasonForReturn == null || reasonForReturn.isEmpty()) { return error(Response.Status.BAD_REQUEST, "You must enter a reason for returning a dataset to the author(s)."); } - AuthenticatedUser authenticatedUser = findAuthenticatedUserOrDie(); + AuthenticatedUser authenticatedUser = getRequestAuthenticatedUserOrDie(crc); Dataset updatedDataset = execCommand(new ReturnDatasetToAuthorCommand(createDataverseRequest(authenticatedUser), dataset, reasonForReturn )); JsonObjectBuilder result = Json.createObjectBuilder(); @@ -2061,13 +2105,15 @@ public Response returnToAuthor(@PathParam("id") String idSupplied, String jsonBo } @GET + @AuthRequired @Path("{id}/curationStatus") - public Response getCurationStatus(@PathParam("id") String idSupplied) { + public Response getCurationStatus(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { try { Dataset ds = findDatasetOrDie(idSupplied); DatasetVersion dsv = ds.getLatestVersion(); - if (dsv.isDraft() && permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), ds).has(Permission.PublishDataset)) { - return response(req -> ok(dsv.getExternalStatusLabel()==null ? "":dsv.getExternalStatusLabel())); + User user = getRequestUser(crc); + if (dsv.isDraft() && permissionSvc.requestOn(createDataverseRequest(user), ds).has(Permission.PublishDataset)) { + return response(req -> ok(dsv.getExternalStatusLabel()==null ? "":dsv.getExternalStatusLabel()), user); } else { return error(Response.Status.FORBIDDEN, "You are not permitted to view the curation status of this dataset."); } @@ -2077,13 +2123,14 @@ public Response getCurationStatus(@PathParam("id") String idSupplied) { } @PUT + @AuthRequired @Path("{id}/curationStatus") - public Response setCurationStatus(@PathParam("id") String idSupplied, @QueryParam("label") String label) { + public Response setCurationStatus(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied, @QueryParam("label") String label) { Dataset ds = null; User u = null; try { ds = findDatasetOrDie(idSupplied); - u = findUserOrDie(); + u = getRequestUser(crc); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -2097,13 +2144,14 @@ public Response setCurationStatus(@PathParam("id") String idSupplied, @QueryPara } @DELETE + @AuthRequired @Path("{id}/curationStatus") - public Response deleteCurationStatus(@PathParam("id") String idSupplied) { + public Response deleteCurationStatus(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { Dataset ds = null; User u = null; try { ds = findDatasetOrDie(idSupplied); - u = findUserOrDie(); + u = getRequestUser(crc); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -2117,19 +2165,15 @@ public Response deleteCurationStatus(@PathParam("id") String idSupplied) { } @GET + @AuthRequired @Path("{id}/uploadsid") @Deprecated - public Response getUploadUrl(@PathParam("id") String idSupplied) { + public Response getUploadUrl(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { try { Dataset dataset = findDatasetOrDie(idSupplied); boolean canUpdateDataset = false; - try { - canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset).canIssue(UpdateDatasetVersionCommand.class); - } catch (WrappedResponse ex) { - logger.info("Exception thrown while trying to figure out permissions while getting upload URL for dataset id " + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } + canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(getRequestUser(crc)), dataset).canIssue(UpdateDatasetVersionCommand.class); if (!canUpdateDataset) { return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); } @@ -2157,21 +2201,15 @@ public Response getUploadUrl(@PathParam("id") String idSupplied) { } @GET + @AuthRequired @Path("{id}/uploadurls") - public Response getMPUploadUrls(@PathParam("id") String idSupplied, @QueryParam("size") long fileSize) { + public Response getMPUploadUrls(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied, @QueryParam("size") long fileSize) { try { Dataset dataset = findDatasetOrDie(idSupplied); boolean canUpdateDataset = false; - try { - canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(findUserOrDie()), dataset) - .canIssue(UpdateDatasetVersionCommand.class); - } catch (WrappedResponse ex) { - logger.info( - "Exception thrown while trying to figure out permissions while getting upload URLs for dataset id " - + dataset.getId() + ": " + ex.getLocalizedMessage()); - throw ex; - } + canUpdateDataset = permissionSvc.requestOn(createDataverseRequest(getRequestUser(crc)), dataset) + .canIssue(UpdateDatasetVersionCommand.class); if (!canUpdateDataset) { return error(Response.Status.FORBIDDEN, "You are not permitted to upload files to this dataset."); } @@ -2200,15 +2238,16 @@ public Response getMPUploadUrls(@PathParam("id") String idSupplied, @QueryParam( } @DELETE + @AuthRequired @Path("mpupload") - public Response abortMPUpload(@QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { + public Response abortMPUpload(@Context ContainerRequestContext crc, @QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { try { Dataset dataset = datasetSvc.findByGlobalId(idSupplied); //Allow the API to be used within a session (e.g. for direct upload in the UI) User user = session.getUser(); if (!user.isAuthenticated()) { try { - user = findAuthenticatedUserOrDie(); + user = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { logger.info( "Exception thrown while trying to figure out permissions while getting aborting upload for dataset id " @@ -2254,15 +2293,16 @@ public Response abortMPUpload(@QueryParam("globalid") String idSupplied, @QueryP } @PUT + @AuthRequired @Path("mpupload") - public Response completeMPUpload(String partETagBody, @QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { + public Response completeMPUpload(@Context ContainerRequestContext crc, String partETagBody, @QueryParam("globalid") String idSupplied, @QueryParam("storageidentifier") String storageidentifier, @QueryParam("uploadid") String uploadId) { try { Dataset dataset = datasetSvc.findByGlobalId(idSupplied); //Allow the API to be used within a session (e.g. for direct upload in the UI) User user = session.getUser(); if (!user.isAuthenticated()) { try { - user = findAuthenticatedUserOrDie(); + user = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { logger.info( "Exception thrown while trying to figure out permissions to complete mpupload for dataset id " @@ -2339,9 +2379,11 @@ public Response completeMPUpload(String partETagBody, @QueryParam("globalid") St * @return */ @POST + @AuthRequired @Path("{id}/add") @Consumes(MediaType.MULTIPART_FORM_DATA) - public Response addFileToDataset(@PathParam("id") String idSupplied, + public Response addFileToDataset(@Context ContainerRequestContext crc, + @PathParam("id") String idSupplied, @FormDataParam("jsonData") String jsonData, @FormDataParam("file") InputStream fileInputStream, @FormDataParam("file") FormDataContentDisposition contentDispositionHeader, @@ -2353,18 +2395,11 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } // ------------------------------------- - // (1) Get the user from the API key + // (1) Get the user from the ContainerRequestContext // ------------------------------------- User authUser; - try { - authUser = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, - BundleUtil.getStringFromBundle("file.addreplace.error.auth") - ); - } - - + authUser = getRequestUser(crc); + // ------------------------------------- // (2) Get the Dataset Id // @@ -2501,7 +2536,7 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } else { return ok(addFileHelper.getSuccessResultAsJsonObjectBuilder()); } - + //"Look at that! You added a file! (hey hey, it may have worked)"); } catch (NoFilesException ex) { Logger.getLogger(Files.class.getName()).log(Level.SEVERE, null, ex); @@ -2513,6 +2548,70 @@ public Response addFileToDataset(@PathParam("id") String idSupplied, } // end: addFileToDataset + /** + * Clean storage of a Dataset + * + * @param idSupplied + * @return + */ + @GET + @AuthRequired + @Path("{id}/cleanStorage") + public Response cleanStorage(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied, @QueryParam("dryrun") Boolean dryrun) { + // get user and dataset + User authUser = getRequestUser(crc); + + Dataset dataset; + try { + dataset = findDatasetOrDie(idSupplied); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + + // check permissions + if (!permissionSvc.permissionsFor(createDataverseRequest(authUser), dataset).contains(Permission.EditDataset)) { + return error(Response.Status.INTERNAL_SERVER_ERROR, "Access denied!"); + } + + boolean doDryRun = dryrun != null && dryrun.booleanValue(); + + // check if no legacy files are present + Set datasetFilenames = getDatasetFilenames(dataset); + if (datasetFilenames.stream().anyMatch(x -> !dataFilePattern.matcher(x).matches())) { + logger.log(Level.WARNING, "Dataset contains legacy files not matching the naming pattern!"); + } + + Predicate filter = getToDeleteFilesFilter(datasetFilenames); + List deleted; + try { + StorageIO datasetIO = DataAccess.getStorageIO(dataset); + deleted = datasetIO.cleanUp(filter, doDryRun); + } catch (IOException ex) { + logger.log(Level.SEVERE, null, ex); + return error(Response.Status.INTERNAL_SERVER_ERROR, "IOException! Serious Error! See administrator!"); + } + + return ok("Found: " + datasetFilenames.stream().collect(Collectors.joining(", ")) + "\n" + "Deleted: " + deleted.stream().collect(Collectors.joining(", "))); + + } + + private static Set getDatasetFilenames(Dataset dataset) { + Set files = new HashSet<>(); + for (DataFile dataFile: dataset.getFiles()) { + String storageIdentifier = dataFile.getStorageIdentifier(); + String location = storageIdentifier.substring(storageIdentifier.indexOf("://") + 3); + String[] locationParts = location.split(":");//separate bucket, swift container, etc. from fileName + files.add(locationParts[locationParts.length-1]); + } + return files; + } + + public static Predicate getToDeleteFilesFilter(Set datasetFilenames) { + return f -> { + return dataFilePattern.matcher(f).matches() && datasetFilenames.stream().noneMatch(x -> f.startsWith(x)); + }; + } + private void msg(String m) { //System.out.println(m); logger.fine(m); @@ -2616,12 +2715,13 @@ public Response getLocksForDataset(@PathParam("identifier") String id, @QueryPar } @DELETE + @AuthRequired @Path("{identifier}/locks") - public Response deleteLocks(@PathParam("identifier") String id, @QueryParam("type") DatasetLock.Reason lockType) { + public Response deleteLocks(@Context ContainerRequestContext crc, @PathParam("identifier") String id, @QueryParam("type") DatasetLock.Reason lockType) { return response(req -> { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "This API end point can be used by superusers only."); } @@ -2675,16 +2775,17 @@ public Response deleteLocks(@PathParam("identifier") String id, @QueryParam("typ return wr.getResponse(); } - }); + }, getRequestUser(crc)); } @POST + @AuthRequired @Path("{identifier}/lock/{type}") - public Response lockDataset(@PathParam("identifier") String id, @PathParam("type") DatasetLock.Reason lockType) { + public Response lockDataset(@Context ContainerRequestContext crc, @PathParam("identifier") String id, @PathParam("type") DatasetLock.Reason lockType) { return response(req -> { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "This API end point can be used by superusers only."); } @@ -2712,12 +2813,13 @@ public Response lockDataset(@PathParam("identifier") String id, @PathParam("type return wr.getResponse(); } - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("locks") - public Response listLocks(@QueryParam("type") String lockType, @QueryParam("userIdentifier") String userIdentifier) { //DatasetLock.Reason lockType) { + public Response listLocks(@Context ContainerRequestContext crc, @QueryParam("type") String lockType, @QueryParam("userIdentifier") String userIdentifier) { //DatasetLock.Reason lockType) { // This API is here, under /datasets, and not under /admin, because we // likely want it to be accessible to admin users who may not necessarily // have localhost access, that would be required to get to /api/admin in @@ -2725,7 +2827,7 @@ public Response listLocks(@QueryParam("type") String lockType, @QueryParam("user // this api to admin users only. AuthenticatedUser apiUser; try { - apiUser = findAuthenticatedUserOrDie(); + apiUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Response.Status.UNAUTHORIZED, "Authentication is required."); } @@ -2808,21 +2910,23 @@ public Response getMakeDataCountMetricCurrentMonth(@PathParam("id") String idSup } @GET + @AuthRequired @Path("{identifier}/storagesize") - public Response getStorageSize(@PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached, + public Response getStorageSize(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { return response(req -> ok(MessageFormat.format(BundleUtil.getStringFromBundle("datasets.api.datasize.storage"), - execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), includeCached, GetDatasetStorageSizeCommand.Mode.STORAGE, null))))); + execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), includeCached, GetDatasetStorageSizeCommand.Mode.STORAGE, null)))), getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/versions/{versionId}/downloadsize") - public Response getDownloadSize(@PathParam("identifier") String dvIdtf, @PathParam("versionId") String version, + public Response getDownloadSize(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @PathParam("versionId") String version, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { return response(req -> ok(MessageFormat.format(BundleUtil.getStringFromBundle("datasets.api.datasize.download"), - execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), false, GetDatasetStorageSizeCommand.Mode.DOWNLOAD, getDatasetVersionOrDie(req, version, findDatasetOrDie(dvIdtf), uriInfo, headers)))))); + execCommand(new GetDatasetStorageSizeCommand(req, findDatasetOrDie(dvIdtf), false, GetDatasetStorageSizeCommand.Mode.DOWNLOAD, getDatasetVersionOrDie(req, version, findDatasetOrDie(dvIdtf), uriInfo, headers))))), getRequestUser(crc)); } @GET @@ -2935,8 +3039,9 @@ public Response getMakeDataCountMetric(@PathParam("id") String idSupplied, @Path } @GET + @AuthRequired @Path("{identifier}/storageDriver") - public Response getFileStore(@PathParam("identifier") String dvIdtf, + public Response getFileStore(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { Dataset dataset; @@ -2947,19 +3052,20 @@ public Response getFileStore(@PathParam("identifier") String dvIdtf, return error(Response.Status.NOT_FOUND, "No such dataset"); } - return response(req -> ok(dataset.getEffectiveStorageDriverId())); + return response(req -> ok(dataset.getEffectiveStorageDriverId()), getRequestUser(crc)); } @PUT + @AuthRequired @Path("{identifier}/storageDriver") - public Response setFileStore(@PathParam("identifier") String dvIdtf, + public Response setFileStore(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, String storageDriverLabel, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { // Superuser-only: AuthenticatedUser user; try { - user = findAuthenticatedUserOrDie(); + user = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Response.Status.BAD_REQUEST, "Authentication is required."); } @@ -2988,14 +3094,15 @@ public Response setFileStore(@PathParam("identifier") String dvIdtf, } @DELETE + @AuthRequired @Path("{identifier}/storageDriver") - public Response resetFileStore(@PathParam("identifier") String dvIdtf, + public Response resetFileStore(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { // Superuser-only: AuthenticatedUser user; try { - user = findAuthenticatedUserOrDie(); + user = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Response.Status.BAD_REQUEST, "Authentication is required."); } @@ -3017,12 +3124,13 @@ public Response resetFileStore(@PathParam("identifier") String dvIdtf, } @GET + @AuthRequired @Path("{identifier}/curationLabelSet") - public Response getCurationLabelSet(@PathParam("identifier") String dvIdtf, + public Response getCurationLabelSet(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -3038,19 +3146,22 @@ public Response getCurationLabelSet(@PathParam("identifier") String dvIdtf, return ex.getResponse(); } - return response(req -> ok(dataset.getEffectiveCurationLabelSetName())); + return response(req -> ok(dataset.getEffectiveCurationLabelSetName()), getRequestUser(crc)); } @PUT + @AuthRequired @Path("{identifier}/curationLabelSet") - public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, - @QueryParam("name") String curationLabelSet, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + public Response setCurationLabelSet(@Context ContainerRequestContext crc, + @PathParam("identifier") String dvIdtf, + @QueryParam("name") String curationLabelSet, + @Context UriInfo uriInfo, + @Context HttpHeaders headers) throws WrappedResponse { // Superuser-only: AuthenticatedUser user; try { - user = findAuthenticatedUserOrDie(); + user = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Response.Status.UNAUTHORIZED, "Authentication is required."); } @@ -3083,14 +3194,15 @@ public Response setCurationLabelSet(@PathParam("identifier") String dvIdtf, } @DELETE + @AuthRequired @Path("{identifier}/curationLabelSet") - public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, + public Response resetCurationLabelSet(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { // Superuser-only: AuthenticatedUser user; try { - user = findAuthenticatedUserOrDie(); + user = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Response.Status.BAD_REQUEST, "Authentication is required."); } @@ -3112,12 +3224,15 @@ public Response resetCurationLabelSet(@PathParam("identifier") String dvIdtf, } @GET + @AuthRequired @Path("{identifier}/allowedCurationLabels") - public Response getAllowedCurationLabels(@PathParam("identifier") String dvIdtf, - @Context UriInfo uriInfo, @Context HttpHeaders headers) throws WrappedResponse { + public Response getAllowedCurationLabels(@Context ContainerRequestContext crc, + @PathParam("identifier") String dvIdtf, + @Context UriInfo uriInfo, + @Context HttpHeaders headers) throws WrappedResponse { AuthenticatedUser user = null; try { - user = findAuthenticatedUserOrDie(); + user = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -3131,22 +3246,23 @@ public Response getAllowedCurationLabels(@PathParam("identifier") String dvIdtf, } if (permissionSvc.requestOn(createDataverseRequest(user), dataset).has(Permission.PublishDataset)) { String[] labelArray = systemConfig.getCurationLabels().get(dataset.getEffectiveCurationLabelSetName()); - return response(req -> ok(String.join(",", labelArray))); + return response(req -> ok(String.join(",", labelArray)), getRequestUser(crc)); } else { return error(Response.Status.FORBIDDEN, "You are not permitted to view the allowed curation labels for this dataset."); } } @GET + @AuthRequired @Path("{identifier}/timestamps") @Produces(MediaType.APPLICATION_JSON) - public Response getTimestamps(@PathParam("identifier") String id) { + public Response getTimestamps(@Context ContainerRequestContext crc, @PathParam("identifier") String id) { Dataset dataset = null; DateTimeFormatter formatter = DateTimeFormatter.ISO_LOCAL_DATE_TIME; try { dataset = findDatasetOrDie(id); - User u = findUserOrDie(); + User u = getRequestUser(crc); Set perms = new HashSet(); perms.add(Permission.ViewUnpublishedDataset); boolean canSeeDraft = permissionSvc.hasPermissionsFor(u, dataset, perms); @@ -3212,9 +3328,11 @@ public Response getTimestamps(@PathParam("identifier") String id) { @POST + @AuthRequired @Path("{id}/addglobusFiles") @Consumes(MediaType.MULTIPART_FORM_DATA) - public Response addGlobusFilesToDataset(@PathParam("id") String datasetId, + public Response addGlobusFilesToDataset(@Context ContainerRequestContext crc, + @PathParam("id") String datasetId, @FormDataParam("jsonData") String jsonData, @Context UriInfo uriInfo, @Context HttpHeaders headers @@ -3231,7 +3349,7 @@ public Response addGlobusFilesToDataset(@PathParam("id") String datasetId, // ------------------------------------- AuthenticatedUser authUser; try { - authUser = findAuthenticatedUserOrDie(); + authUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") ); @@ -3291,9 +3409,10 @@ public Response addGlobusFilesToDataset(@PathParam("id") String datasetId, } @POST + @AuthRequired @Path("{id}/deleteglobusRule") @Consumes(MediaType.MULTIPART_FORM_DATA) - public Response deleteglobusRule(@PathParam("id") String datasetId,@FormDataParam("jsonData") String jsonData + public Response deleteglobusRule(@Context ContainerRequestContext crc, @PathParam("id") String datasetId,@FormDataParam("jsonData") String jsonData ) throws IOException, ExecutionException, InterruptedException { @@ -3305,15 +3424,10 @@ public Response deleteglobusRule(@PathParam("id") String datasetId,@FormDataPara } // ------------------------------------- - // (1) Get the user from the API key + // (1) Get the user from the ContainerRequestContext // ------------------------------------- User authUser; - try { - authUser = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") - ); - } + authUser = getRequestUser(crc); // ------------------------------------- // (2) Get the Dataset Id @@ -3342,9 +3456,11 @@ public Response deleteglobusRule(@PathParam("id") String datasetId,@FormDataPara * @return */ @POST + @AuthRequired @Path("{id}/addFiles") @Consumes(MediaType.MULTIPART_FORM_DATA) - public Response addFilesToDataset(@PathParam("id") String idSupplied, + public Response addFilesToDataset(@Context ContainerRequestContext crc, + @PathParam("id") String idSupplied, @FormDataParam("jsonData") String jsonData) { if (!systemConfig.isHTTPUpload()) { @@ -3352,15 +3468,10 @@ public Response addFilesToDataset(@PathParam("id") String idSupplied, } // ------------------------------------- - // (1) Get the user from the API key + // (1) Get the user from the ContainerRequestContext // ------------------------------------- User authUser; - try { - authUser = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") - ); - } + authUser = getRequestUser(crc); // ------------------------------------- // (2) Get the Dataset Id @@ -3413,25 +3524,22 @@ public Response addFilesToDataset(@PathParam("id") String idSupplied, * @return */ @POST + @AuthRequired @Path("{id}/replaceFiles") @Consumes(MediaType.MULTIPART_FORM_DATA) - public Response replaceFilesInDataset(@PathParam("id") String idSupplied, - @FormDataParam("jsonData") String jsonData) { + public Response replaceFilesInDataset(@Context ContainerRequestContext crc, + @PathParam("id") String idSupplied, + @FormDataParam("jsonData") String jsonData) { if (!systemConfig.isHTTPUpload()) { return error(Response.Status.SERVICE_UNAVAILABLE, BundleUtil.getStringFromBundle("file.api.httpDisabled")); } // ------------------------------------- - // (1) Get the user from the API key + // (1) Get the user from the ContainerRequestContext // ------------------------------------- User authUser; - try { - authUser = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("file.addreplace.error.auth") - ); - } + authUser = getRequestUser(crc); // ------------------------------------- // (2) Get the Dataset Id @@ -3483,12 +3591,13 @@ public Response replaceFilesInDataset(@PathParam("id") String idSupplied, * @throws WrappedResponse */ @GET + @AuthRequired @Path("/listCurationStates") @Produces("text/csv") - public Response getCurationStates() throws WrappedResponse { + public Response getCurationStates(@Context ContainerRequestContext crc) throws WrappedResponse { try { - AuthenticatedUser user = findAuthenticatedUserOrDie(); + AuthenticatedUser user = getRequestAuthenticatedUserOrDie(crc); if (!user.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -3540,13 +3649,17 @@ public Response getCurationStates() throws WrappedResponse { // APIs to manage archival status @GET + @AuthRequired @Produces(MediaType.APPLICATION_JSON) @Path("/{id}/{version}/archivalStatus") - public Response getDatasetVersionArchivalStatus(@PathParam("id") String datasetId, - @PathParam("version") String versionNumber, @Context UriInfo uriInfo, @Context HttpHeaders headers) { + public Response getDatasetVersionArchivalStatus(@Context ContainerRequestContext crc, + @PathParam("id") String datasetId, + @PathParam("version") String versionNumber, + @Context UriInfo uriInfo, + @Context HttpHeaders headers) { try { - AuthenticatedUser au = findAuthenticatedUserOrDie(); + AuthenticatedUser au = getRequestAuthenticatedUserOrDie(crc); if (!au.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -3566,15 +3679,19 @@ public Response getDatasetVersionArchivalStatus(@PathParam("id") String datasetI } @PUT + @AuthRequired @Consumes(MediaType.APPLICATION_JSON) @Path("/{id}/{version}/archivalStatus") - public Response setDatasetVersionArchivalStatus(@PathParam("id") String datasetId, - @PathParam("version") String versionNumber, String newStatus, @Context UriInfo uriInfo, - @Context HttpHeaders headers) { + public Response setDatasetVersionArchivalStatus(@Context ContainerRequestContext crc, + @PathParam("id") String datasetId, + @PathParam("version") String versionNumber, + String newStatus, + @Context UriInfo uriInfo, + @Context HttpHeaders headers) { logger.fine(newStatus); try { - AuthenticatedUser au = findAuthenticatedUserOrDie(); + AuthenticatedUser au = getRequestAuthenticatedUserOrDie(crc); if (!au.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); @@ -3620,13 +3737,17 @@ public Response setDatasetVersionArchivalStatus(@PathParam("id") String datasetI } @DELETE + @AuthRequired @Produces(MediaType.APPLICATION_JSON) @Path("/{id}/{version}/archivalStatus") - public Response deleteDatasetVersionArchivalStatus(@PathParam("id") String datasetId, - @PathParam("version") String versionNumber, @Context UriInfo uriInfo, @Context HttpHeaders headers) { + public Response deleteDatasetVersionArchivalStatus(@Context ContainerRequestContext crc, + @PathParam("id") String datasetId, + @PathParam("version") String versionNumber, + @Context UriInfo uriInfo, + @Context HttpHeaders headers) { try { - AuthenticatedUser au = findAuthenticatedUserOrDie(); + AuthenticatedUser au = getRequestAuthenticatedUserOrDie(crc); if (!au.isSuperuser()) { return error(Response.Status.FORBIDDEN, "Superusers only."); } @@ -3670,11 +3791,15 @@ private boolean isSingleVersionArchiving() { // This supports the cases where a tool is accessing a restricted resource (e.g. // for a draft dataset), or public case. @GET + @AuthRequired @Path("{id}/versions/{version}/toolparams/{tid}") - public Response getExternalToolDVParams(@PathParam("tid") long externalToolId, - @PathParam("id") String datasetId, @PathParam("version") String version, @QueryParam(value = "locale") String locale) { + public Response getExternalToolDVParams(@Context ContainerRequestContext crc, + @PathParam("tid") long externalToolId, + @PathParam("id") String datasetId, + @PathParam("version") String version, + @QueryParam(value = "locale") String locale) { try { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); DatasetVersion target = getDatasetVersionOrDie(req, version, findDatasetOrDie(datasetId), null, null); if (target == null) { return error(BAD_REQUEST, "DatasetVersion not found."); @@ -3688,7 +3813,7 @@ public Response getExternalToolDVParams(@PathParam("tid") long externalToolId, return error(BAD_REQUEST, "External tool does not have dataset scope."); } ApiToken apiToken = null; - User u = findUserOrDie(); + User u = getRequestUser(crc); if (u instanceof AuthenticatedUser) { apiToken = authSvc.findApiTokenByUser((AuthenticatedUser) u); } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java index a3e088e172a..0d13289483d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Dataverses.java @@ -9,6 +9,7 @@ import edu.harvard.iq.dataverse.DataverseContact; import edu.harvard.iq.dataverse.DataverseMetadataBlockFacet; import edu.harvard.iq.dataverse.DataverseServiceBean; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.api.datadeposit.SwordServiceBean; import edu.harvard.iq.dataverse.api.dto.DataverseMetadataBlockFacetDTO; import edu.harvard.iq.dataverse.authorization.DataverseRole; @@ -120,6 +121,7 @@ import java.util.stream.Collectors; import jakarta.servlet.http.HttpServletResponse; import jakarta.ws.rs.WebApplicationException; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.StreamingOutput; import javax.xml.stream.XMLStreamException; @@ -158,14 +160,16 @@ public class Dataverses extends AbstractApiBean { SwordServiceBean swordService; @POST - public Response addRoot(String body) { + @AuthRequired + public Response addRoot(@Context ContainerRequestContext crc, String body) { logger.info("Creating root dataverse"); - return addDataverse(body, ""); + return addDataverse(crc, body, ""); } @POST + @AuthRequired @Path("{identifier}") - public Response addDataverse(String body, @PathParam("identifier") String parentIdtf) { + public Response addDataverse(@Context ContainerRequestContext crc, String body, @PathParam("identifier") String parentIdtf) { Dataverse d; JsonObject dvJson; @@ -192,7 +196,7 @@ public Response addDataverse(String body, @PathParam("identifier") String parent dc.setDataverse(d); } - AuthenticatedUser u = findAuthenticatedUserOrDie(); + AuthenticatedUser u = getRequestAuthenticatedUserOrDie(crc); d = execCommand(new CreateDataverseCommand(d, createDataverseRequest(u), null, null)); return created("/dataverses/" + d.getAlias(), json(d)); } catch (WrappedResponse ww) { @@ -224,12 +228,13 @@ public Response addDataverse(String body, @PathParam("identifier") String parent } @POST + @AuthRequired @Path("{identifier}/datasets") @Consumes("application/json") - public Response createDataset(String jsonBody, @PathParam("identifier") String parentIdtf) { + public Response createDataset(@Context ContainerRequestContext crc, String jsonBody, @PathParam("identifier") String parentIdtf) { try { logger.fine("Json is: " + jsonBody); - User u = findUserOrDie(); + User u = getRequestUser(crc); Dataverse owner = findDataverseOrDie(parentIdtf); Dataset ds = parseDataset(jsonBody); ds.setOwner(owner); @@ -292,11 +297,12 @@ public Response createDataset(String jsonBody, @PathParam("identifier") String p } @POST + @AuthRequired @Path("{identifier}/datasets") @Consumes("application/ld+json, application/json-ld") - public Response createDatasetFromJsonLd(String jsonLDBody, @PathParam("identifier") String parentIdtf) { + public Response createDatasetFromJsonLd(@Context ContainerRequestContext crc, String jsonLDBody, @PathParam("identifier") String parentIdtf) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); Dataverse owner = findDataverseOrDie(parentIdtf); Dataset ds = new Dataset(); @@ -334,10 +340,11 @@ public Response createDatasetFromJsonLd(String jsonLDBody, @PathParam("identifie } @POST + @AuthRequired @Path("{identifier}/datasets/:import") - public Response importDataset(String jsonBody, @PathParam("identifier") String parentIdtf, @QueryParam("pid") String pidParam, @QueryParam("release") String releaseParam) { + public Response importDataset(@Context ContainerRequestContext crc, String jsonBody, @PathParam("identifier") String parentIdtf, @QueryParam("pid") String pidParam, @QueryParam("release") String releaseParam) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); if (!u.isSuperuser()) { return error(Status.FORBIDDEN, "Not a superuser"); } @@ -408,10 +415,11 @@ public Response importDataset(String jsonBody, @PathParam("identifier") String p // TODO decide if I merge importddi with import just below (xml and json on same api, instead of 2 api) @POST + @AuthRequired @Path("{identifier}/datasets/:importddi") - public Response importDatasetDdi(String xml, @PathParam("identifier") String parentIdtf, @QueryParam("pid") String pidParam, @QueryParam("release") String releaseParam) { + public Response importDatasetDdi(@Context ContainerRequestContext crc, String xml, @PathParam("identifier") String parentIdtf, @QueryParam("pid") String pidParam, @QueryParam("release") String releaseParam) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); if (!u.isSuperuser()) { return error(Status.FORBIDDEN, "Not a superuser"); } @@ -482,11 +490,12 @@ public Response importDatasetDdi(String xml, @PathParam("identifier") String par } @POST + @AuthRequired @Path("{identifier}/datasets/:startmigration") @Consumes("application/ld+json, application/json-ld") - public Response recreateDataset(String jsonLDBody, @PathParam("identifier") String parentIdtf) { + public Response recreateDataset(@Context ContainerRequestContext crc, String jsonLDBody, @PathParam("identifier") String parentIdtf) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); if (!u.isSuperuser()) { return error(Status.FORBIDDEN, "Not a superuser"); } @@ -546,39 +555,43 @@ private Dataset parseDataset(String datasetJson) throws WrappedResponse { } @GET + @AuthRequired @Path("{identifier}") - public Response viewDataverse(@PathParam("identifier") String idtf) { + public Response viewDataverse(@Context ContainerRequestContext crc, @PathParam("identifier") String idtf) { return response(req -> ok( json(execCommand(new GetDataverseCommand(req, findDataverseOrDie(idtf))), settingsService.isTrueForKey(SettingsServiceBean.Key.ExcludeEmailFromExport, false) - ))); + )), getRequestUser(crc)); } @DELETE + @AuthRequired @Path("{identifier}") - public Response deleteDataverse(@PathParam("identifier") String idtf) { + public Response deleteDataverse(@Context ContainerRequestContext crc, @PathParam("identifier") String idtf) { return response(req -> { execCommand(new DeleteDataverseCommand(req, findDataverseOrDie(idtf))); return ok("Dataverse " + idtf + " deleted"); - }); + }, getRequestUser(crc)); } @DELETE + @AuthRequired @Path("{linkingDataverseId}/deleteLink/{linkedDataverseId}") - public Response deleteDataverseLinkingDataverse(@PathParam("linkingDataverseId") String linkingDataverseId, @PathParam("linkedDataverseId") String linkedDataverseId) { + public Response deleteDataverseLinkingDataverse(@Context ContainerRequestContext crc, @PathParam("linkingDataverseId") String linkingDataverseId, @PathParam("linkedDataverseId") String linkedDataverseId) { boolean index = true; return response(req -> { execCommand(new DeleteDataverseLinkingDataverseCommand(req, findDataverseOrDie(linkingDataverseId), findDataverseLinkingDataverseOrDie(linkingDataverseId, linkedDataverseId), index)); return ok("Link from Dataverse " + linkingDataverseId + " to linked Dataverse " + linkedDataverseId + " deleted"); - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/metadatablocks") - public Response listMetadataBlocks(@PathParam("identifier") String dvIdtf) { + public Response listMetadataBlocks(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { try { JsonArrayBuilder arr = Json.createArrayBuilder(); - final List blocks = execCommand(new ListMetadataBlocksCommand(createDataverseRequest(findUserOrDie()), findDataverseOrDie(dvIdtf))); + final List blocks = execCommand(new ListMetadataBlocksCommand(createDataverseRequest(getRequestUser(crc)), findDataverseOrDie(dvIdtf))); for (MetadataBlock mdb : blocks) { arr.add(brief.json(mdb)); } @@ -589,9 +602,10 @@ public Response listMetadataBlocks(@PathParam("identifier") String dvIdtf) { } @POST + @AuthRequired @Path("{identifier}/metadatablocks") @Produces(MediaType.APPLICATION_JSON) - public Response setMetadataBlocks(@PathParam("identifier") String dvIdtf, String blockIds) { + public Response setMetadataBlocks(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, String blockIds) { List blocks = new LinkedList<>(); try { @@ -609,7 +623,7 @@ public Response setMetadataBlocks(@PathParam("identifier") String dvIdtf, String } try { - execCommand(new UpdateDataverseMetadataBlocksCommand.SetBlocks(createDataverseRequest(findUserOrDie()), findDataverseOrDie(dvIdtf), blocks)); + execCommand(new UpdateDataverseMetadataBlocksCommand.SetBlocks(createDataverseRequest(getRequestUser(crc)), findDataverseOrDie(dvIdtf), blocks)); return ok("Metadata blocks of dataverse " + dvIdtf + " updated."); } catch (WrappedResponse ex) { @@ -618,15 +632,17 @@ public Response setMetadataBlocks(@PathParam("identifier") String dvIdtf, String } @GET + @AuthRequired @Path("{identifier}/metadatablocks/:isRoot") - public Response getMetadataRoot_legacy(@PathParam("identifier") String dvIdtf) { - return getMetadataRoot(dvIdtf); + public Response getMetadataRoot_legacy(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { + return getMetadataRoot(crc, dvIdtf); } @GET + @AuthRequired @Path("{identifier}/metadatablocks/isRoot") @Produces(MediaType.APPLICATION_JSON) - public Response getMetadataRoot(@PathParam("identifier") String dvIdtf) { + public Response getMetadataRoot(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { return response(req -> { final Dataverse dataverse = findDataverseOrDie(dvIdtf); if (permissionSvc.request(req) @@ -636,38 +652,41 @@ public Response getMetadataRoot(@PathParam("identifier") String dvIdtf) { } else { return error(Status.FORBIDDEN, "Not authorized"); } - }); + }, getRequestUser(crc)); } @POST + @AuthRequired @Path("{identifier}/metadatablocks/:isRoot") @Produces(MediaType.APPLICATION_JSON) @Consumes(MediaType.WILDCARD) - public Response setMetadataRoot_legacy(@PathParam("identifier") String dvIdtf, String body) { - return setMetadataRoot(dvIdtf, body); + public Response setMetadataRoot_legacy(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, String body) { + return setMetadataRoot(crc, dvIdtf, body); } @PUT + @AuthRequired @Path("{identifier}/metadatablocks/isRoot") @Produces(MediaType.APPLICATION_JSON) @Consumes(MediaType.WILDCARD) - public Response setMetadataRoot(@PathParam("identifier") String dvIdtf, String body) { + public Response setMetadataRoot(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, String body) { return response(req -> { final boolean root = parseBooleanOrDie(body); final Dataverse dataverse = findDataverseOrDie(dvIdtf); execCommand(new UpdateDataverseMetadataBlocksCommand.SetRoot(req, dataverse, root)); return ok("Dataverse " + dataverse.getName() + " is now a metadata " + (root ? "" : "non-") + "root"); - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/facets/") /** * return list of facets for the dataverse with alias `dvIdtf` */ - public Response listFacets(@PathParam("identifier") String dvIdtf) { + public Response listFacets(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); DataverseRequest r = createDataverseRequest(u); Dataverse dataverse = findDataverseOrDie(dvIdtf); JsonArrayBuilder fs = Json.createArrayBuilder(); @@ -681,6 +700,7 @@ public Response listFacets(@PathParam("identifier") String dvIdtf) { } @POST + @AuthRequired @Path("{identifier}/facets") @Produces(MediaType.APPLICATION_JSON) /** @@ -690,7 +710,7 @@ public Response listFacets(@PathParam("identifier") String dvIdtf) { * where foo.json contains a list of datasetField names, works as expected * (judging by the UI). This triggers a 500 when '-d @foo.json' is used. */ - public Response setFacets(@PathParam("identifier") String dvIdtf, String facetIds) { + public Response setFacets(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, String facetIds) { List facets = new LinkedList<>(); for (JsonString facetId : Util.asJsonArray(facetIds).getValuesAs(JsonString.class)) { @@ -706,7 +726,7 @@ public Response setFacets(@PathParam("identifier") String dvIdtf, String facetId try { Dataverse dataverse = findDataverseOrDie(dvIdtf); // by passing null for Featured Dataverses and DataverseFieldTypeInputLevel, those are not changed - execCommand(new UpdateDataverseCommand(dataverse, facets, null, createDataverseRequest(findUserOrDie()), null)); + execCommand(new UpdateDataverseCommand(dataverse, facets, null, createDataverseRequest(getRequestUser(crc)), null)); return ok("Facets of dataverse " + dvIdtf + " updated."); } catch (WrappedResponse ex) { @@ -715,11 +735,12 @@ public Response setFacets(@PathParam("identifier") String dvIdtf, String facetId } @GET + @AuthRequired @Path("{identifier}/metadatablockfacets") @Produces(MediaType.APPLICATION_JSON) - public Response listMetadataBlockFacets(@PathParam("identifier") String dvIdtf) { + public Response listMetadataBlockFacets(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); DataverseRequest request = createDataverseRequest(u); Dataverse dataverse = findDataverseOrDie(dvIdtf); List metadataBlockFacets = Optional.ofNullable(execCommand(new ListMetadataBlockFacetsCommand(request, dataverse))).orElse(Collections.emptyList()); @@ -734,10 +755,11 @@ public Response listMetadataBlockFacets(@PathParam("identifier") String dvIdtf) } @POST + @AuthRequired @Path("{identifier}/metadatablockfacets") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) - public Response setMetadataBlockFacets(@PathParam("identifier") String dvIdtf, List metadataBlockNames) { + public Response setMetadataBlockFacets(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, List metadataBlockNames) { try { Dataverse dataverse = findDataverseOrDie(dvIdtf); @@ -758,7 +780,7 @@ public Response setMetadataBlockFacets(@PathParam("identifier") String dvIdtf, L metadataBlockFacets.add(metadataBlockFacet); } - execCommand(new UpdateMetadataBlockFacetsCommand(createDataverseRequest(findUserOrDie()), dataverse, metadataBlockFacets)); + execCommand(new UpdateMetadataBlockFacetsCommand(createDataverseRequest(getRequestUser(crc)), dataverse, metadataBlockFacets)); return ok(String.format("Metadata block facets updated. DataverseId: %s blocks: %s", dvIdtf, metadataBlockNames)); } catch (WrappedResponse ex) { @@ -767,10 +789,11 @@ public Response setMetadataBlockFacets(@PathParam("identifier") String dvIdtf, L } @POST + @AuthRequired @Path("{identifier}/metadatablockfacets/isRoot") @Consumes(MediaType.APPLICATION_JSON) @Produces(MediaType.APPLICATION_JSON) - public Response updateMetadataBlockFacetsRoot(@PathParam("identifier") String dvIdtf, String body) { + public Response updateMetadataBlockFacetsRoot(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, String body) { try { final boolean blockFacetsRoot = parseBooleanOrDie(body); Dataverse dataverse = findDataverseOrDie(dvIdtf); @@ -778,7 +801,7 @@ public Response updateMetadataBlockFacetsRoot(@PathParam("identifier") String dv return ok(String.format("No update needed, dataverse already consistent with new value. DataverseId: %s blockFacetsRoot: %s", dvIdtf, blockFacetsRoot)); } - execCommand(new UpdateMetadataBlockFacetRootCommand(createDataverseRequest(findUserOrDie()), dataverse, blockFacetsRoot)); + execCommand(new UpdateMetadataBlockFacetRootCommand(createDataverseRequest(getRequestUser(crc)), dataverse, blockFacetsRoot)); return ok(String.format("Metadata block facets root updated. DataverseId: %s blockFacetsRoot: %s", dvIdtf, blockFacetsRoot)); } catch (WrappedResponse ex) { @@ -791,8 +814,9 @@ public Response updateMetadataBlockFacetsRoot(@PathParam("identifier") String dv // (2438-4295-dois-for-files branch) such that a contributor API token no longer allows this method // to be called without a PermissionException being thrown. @GET + @AuthRequired @Path("{identifier}/contents") - public Response listContent(@PathParam("identifier") String dvIdtf) throws WrappedResponse { + public Response listContent(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) throws WrappedResponse { DvObject.Visitor ser = new DvObject.Visitor() { @Override @@ -818,43 +842,47 @@ public JsonObjectBuilder visit(DataFile df) { .stream() .map(dvo -> (JsonObjectBuilder) dvo.accept(ser)) .collect(toJsonArray()) - )); + ), getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/storagesize") - public Response getStorageSize(@PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached) throws WrappedResponse { + public Response getStorageSize(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @QueryParam("includeCached") boolean includeCached) throws WrappedResponse { return response(req -> ok(MessageFormat.format(BundleUtil.getStringFromBundle("dataverse.datasize"), - execCommand(new GetDataverseStorageSizeCommand(req, findDataverseOrDie(dvIdtf), includeCached))))); + execCommand(new GetDataverseStorageSizeCommand(req, findDataverseOrDie(dvIdtf), includeCached)))), getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/roles") - public Response listRoles(@PathParam("identifier") String dvIdtf) { + public Response listRoles(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { return response(req -> ok( execCommand(new ListRolesCommand(req, findDataverseOrDie(dvIdtf))) .stream().map(r -> json(r)) .collect(toJsonArray()) - )); + ), getRequestUser(crc)); } @POST + @AuthRequired @Path("{identifier}/roles") - public Response createRole(RoleDTO roleDto, @PathParam("identifier") String dvIdtf) { - return response(req -> ok(json(execCommand(new CreateRoleCommand(roleDto.asRole(), req, findDataverseOrDie(dvIdtf)))))); + public Response createRole(@Context ContainerRequestContext crc, RoleDTO roleDto, @PathParam("identifier") String dvIdtf) { + return response(req -> ok(json(execCommand(new CreateRoleCommand(roleDto.asRole(), req, findDataverseOrDie(dvIdtf))))), getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/assignments") - public Response listAssignments(@PathParam("identifier") String dvIdtf) { + public Response listAssignments(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { return response(req -> ok( execCommand(new ListRoleAssignments(req, findDataverseOrDie(dvIdtf))) .stream() .map(a -> json(a)) .collect(toJsonArray()) - )); + ), getRequestUser(crc)); } /** @@ -947,11 +975,12 @@ public Response listAssignments(@PathParam("identifier") String dvIdtf) { // } // } @POST + @AuthRequired @Path("{identifier}/assignments") - public Response createAssignment(RoleAssignmentDTO ra, @PathParam("identifier") String dvIdtf, @QueryParam("key") String apiKey) { + public Response createAssignment(@Context ContainerRequestContext crc, RoleAssignmentDTO ra, @PathParam("identifier") String dvIdtf, @QueryParam("key") String apiKey) { try { - final DataverseRequest req = createDataverseRequest(findUserOrDie()); + final DataverseRequest req = createDataverseRequest(getRequestUser(crc)); final Dataverse dataverse = findDataverseOrDie(dvIdtf); RoleAssignee assignee = findAssignee(ra.getAssignee()); @@ -985,13 +1014,14 @@ public Response createAssignment(RoleAssignmentDTO ra, @PathParam("identifier") } @DELETE + @AuthRequired @Path("{identifier}/assignments/{id}") - public Response deleteAssignment(@PathParam("id") long assignmentId, @PathParam("identifier") String dvIdtf) { + public Response deleteAssignment(@Context ContainerRequestContext crc, @PathParam("id") long assignmentId, @PathParam("identifier") String dvIdtf) { RoleAssignment ra = em.find(RoleAssignment.class, assignmentId); if (ra != null) { try { findDataverseOrDie(dvIdtf); - execCommand(new RevokeRoleCommand(ra, createDataverseRequest(findUserOrDie()))); + execCommand(new RevokeRoleCommand(ra, createDataverseRequest(getRequestUser(crc)))); return ok("Role " + ra.getRole().getName() + " revoked for assignee " + ra.getAssigneeIdentifier() + " in " + ra.getDefinitionPoint().accept(DvObject.NamePrinter)); @@ -1004,11 +1034,12 @@ public Response deleteAssignment(@PathParam("id") long assignmentId, @PathParam( } @POST + @AuthRequired @Path("{identifier}/actions/:publish") - public Response publishDataverse(@PathParam("identifier") String dvIdtf) { + public Response publishDataverse(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { try { Dataverse dv = findDataverseOrDie(dvIdtf); - return ok(json(execCommand(new PublishDataverseCommand(createDataverseRequest(findAuthenticatedUserOrDie()), dv)))); + return ok(json(execCommand(new PublishDataverseCommand(createDataverseRequest(getRequestAuthenticatedUserOrDie(crc)), dv)))); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -1016,8 +1047,9 @@ public Response publishDataverse(@PathParam("identifier") String dvIdtf) { } @POST + @AuthRequired @Path("{identifier}/groups/") - public Response createExplicitGroup(ExplicitGroupDTO dto, @PathParam("identifier") String dvIdtf) { + public Response createExplicitGroup(@Context ContainerRequestContext crc, ExplicitGroupDTO dto, @PathParam("identifier") String dvIdtf) { return response(req -> { ExplicitGroupProvider prv = explicitGroupSvc.getProvider(); ExplicitGroup newGroup = dto.apply(prv.makeGroup()); @@ -1026,36 +1058,40 @@ public Response createExplicitGroup(ExplicitGroupDTO dto, @PathParam("identifier String groupUri = String.format("%s/groups/%s", dvIdtf, newGroup.getGroupAliasInOwner()); return created(groupUri, json(newGroup)); - }); + }, getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/groups/") - public Response listGroups(@PathParam("identifier") String dvIdtf, @QueryParam("key") String apiKey) { + public Response listGroups(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @QueryParam("key") String apiKey) { return response(req -> ok( execCommand(new ListExplicitGroupsCommand(req, findDataverseOrDie(dvIdtf))) .stream().map(eg -> json(eg)) .collect(toJsonArray()) - )); + ), getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/groups/{aliasInOwner}") - public Response getGroupByOwnerAndAliasInOwner(@PathParam("identifier") String dvIdtf, - @PathParam("aliasInOwner") String grpAliasInOwner) { + public Response getGroupByOwnerAndAliasInOwner(@Context ContainerRequestContext crc, + @PathParam("identifier") String dvIdtf, + @PathParam("aliasInOwner") String grpAliasInOwner) { return response(req -> ok(json(findExplicitGroupOrDie(findDataverseOrDie(dvIdtf), req, - grpAliasInOwner)))); + grpAliasInOwner))), getRequestUser(crc)); } @GET + @AuthRequired @Path("{identifier}/guestbookResponses/") - public Response getGuestbookResponsesByDataverse(@PathParam("identifier") String dvIdtf, + public Response getGuestbookResponsesByDataverse(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @QueryParam("guestbookId") Long gbId, @Context HttpServletResponse response) { try { Dataverse dv = findDataverseOrDie(dvIdtf); - User u = findUserOrDie(); + User u = getRequestUser(crc); DataverseRequest req = createDataverseRequest(u); if (permissionSvc.request(req) .on(dv) @@ -1091,18 +1127,21 @@ public void write(OutputStream os) throws IOException, } @PUT + @AuthRequired @Path("{identifier}/groups/{aliasInOwner}") - public Response updateGroup(ExplicitGroupDTO groupDto, + public Response updateGroup(@Context ContainerRequestContext crc, ExplicitGroupDTO groupDto, @PathParam("identifier") String dvIdtf, @PathParam("aliasInOwner") String grpAliasInOwner) { return response(req -> ok(json(execCommand( new UpdateExplicitGroupCommand(req, - groupDto.apply(findExplicitGroupOrDie(findDataverseOrDie(dvIdtf), req, grpAliasInOwner))))))); + groupDto.apply(findExplicitGroupOrDie(findDataverseOrDie(dvIdtf), req, grpAliasInOwner)))))), getRequestUser(crc)); } @PUT + @AuthRequired @Path("{identifier}/defaultContributorRole/{roleAlias}") public Response updateDefaultContributorRole( + @Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf, @PathParam("roleAlias") String roleAlias) { @@ -1138,7 +1177,7 @@ public Response updateDefaultContributorRole( List args = Arrays.asList(dv.getDisplayName(), defaultRoleName); String retString = BundleUtil.getStringFromBundle("dataverses.api.update.default.contributor.role.success", args); return ok(retString); - }); + }, getRequestUser(crc)); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -1147,47 +1186,55 @@ public Response updateDefaultContributorRole( } @DELETE + @AuthRequired @Path("{identifier}/groups/{aliasInOwner}") - public Response deleteGroup(@PathParam("identifier") String dvIdtf, - @PathParam("aliasInOwner") String grpAliasInOwner) { + public Response deleteGroup(@Context ContainerRequestContext crc, + @PathParam("identifier") String dvIdtf, + @PathParam("aliasInOwner") String grpAliasInOwner) { return response(req -> { execCommand(new DeleteExplicitGroupCommand(req, findExplicitGroupOrDie(findDataverseOrDie(dvIdtf), req, grpAliasInOwner))); return ok("Group " + dvIdtf + "/" + grpAliasInOwner + " deleted"); - }); + }, getRequestUser(crc)); } @POST + @AuthRequired @Path("{identifier}/groups/{aliasInOwner}/roleAssignees") @Consumes("application/json") - public Response addRoleAssingees(List roleAssingeeIdentifiers, - @PathParam("identifier") String dvIdtf, - @PathParam("aliasInOwner") String grpAliasInOwner) { + public Response addRoleAssingees(@Context ContainerRequestContext crc, + List roleAssingeeIdentifiers, + @PathParam("identifier") String dvIdtf, + @PathParam("aliasInOwner") String grpAliasInOwner) { return response(req -> ok( json( execCommand( new AddRoleAssigneesToExplicitGroupCommand(req, findExplicitGroupOrDie(findDataverseOrDie(dvIdtf), req, grpAliasInOwner), - new TreeSet<>(roleAssingeeIdentifiers)))))); + new TreeSet<>(roleAssingeeIdentifiers))))), getRequestUser(crc)); } @PUT + @AuthRequired @Path("{identifier}/groups/{aliasInOwner}/roleAssignees/{roleAssigneeIdentifier: .*}") - public Response addRoleAssingee(@PathParam("identifier") String dvIdtf, - @PathParam("aliasInOwner") String grpAliasInOwner, - @PathParam("roleAssigneeIdentifier") String roleAssigneeIdentifier) { - return addRoleAssingees(Collections.singletonList(roleAssigneeIdentifier), dvIdtf, grpAliasInOwner); + public Response addRoleAssingee(@Context ContainerRequestContext crc, + @PathParam("identifier") String dvIdtf, + @PathParam("aliasInOwner") String grpAliasInOwner, + @PathParam("roleAssigneeIdentifier") String roleAssigneeIdentifier) { + return addRoleAssingees(crc, Collections.singletonList(roleAssigneeIdentifier), dvIdtf, grpAliasInOwner); } @DELETE + @AuthRequired @Path("{identifier}/groups/{aliasInOwner}/roleAssignees/{roleAssigneeIdentifier: .*}") - public Response deleteRoleAssingee(@PathParam("identifier") String dvIdtf, - @PathParam("aliasInOwner") String grpAliasInOwner, - @PathParam("roleAssigneeIdentifier") String roleAssigneeIdentifier) { + public Response deleteRoleAssingee(@Context ContainerRequestContext crc, + @PathParam("identifier") String dvIdtf, + @PathParam("aliasInOwner") String grpAliasInOwner, + @PathParam("roleAssigneeIdentifier") String roleAssigneeIdentifier) { return response(req -> ok(json(execCommand( new RemoveRoleAssigneesFromExplicitGroupCommand(req, findExplicitGroupOrDie(findDataverseOrDie(dvIdtf), req, grpAliasInOwner), - Collections.singleton(roleAssigneeIdentifier)))))); + Collections.singleton(roleAssigneeIdentifier))))), getRequestUser(crc)); } private ExplicitGroup findExplicitGroupOrDie(DvObject dv, DataverseRequest req, String groupIdtf) throws WrappedResponse { @@ -1199,10 +1246,11 @@ private ExplicitGroup findExplicitGroupOrDie(DvObject dv, DataverseRequest req, } @GET + @AuthRequired @Path("{identifier}/links") - public Response listLinks(@PathParam("identifier") String dvIdtf) { + public Response listLinks(@Context ContainerRequestContext crc, @PathParam("identifier") String dvIdtf) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); Dataverse dv = findDataverseOrDie(dvIdtf); if (!u.isSuperuser()) { return error(Status.FORBIDDEN, "Not a superuser"); @@ -1238,10 +1286,11 @@ public Response listLinks(@PathParam("identifier") String dvIdtf) { } @POST + @AuthRequired @Path("{id}/move/{targetDataverseAlias}") - public Response moveDataverse(@PathParam("id") String id, @PathParam("targetDataverseAlias") String targetDataverseAlias, @QueryParam("forceMove") Boolean force) { + public Response moveDataverse(@Context ContainerRequestContext crc, @PathParam("id") String id, @PathParam("targetDataverseAlias") String targetDataverseAlias, @QueryParam("forceMove") Boolean force) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); Dataverse dv = findDataverseOrDie(id); Dataverse target = findDataverseOrDie(targetDataverseAlias); if (target == null) { @@ -1257,10 +1306,11 @@ public Response moveDataverse(@PathParam("id") String id, @PathParam("targetData } @PUT + @AuthRequired @Path("{linkedDataverseAlias}/link/{linkingDataverseAlias}") - public Response linkDataverse(@PathParam("linkedDataverseAlias") String linkedDataverseAlias, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { + public Response linkDataverse(@Context ContainerRequestContext crc, @PathParam("linkedDataverseAlias") String linkedDataverseAlias, @PathParam("linkingDataverseAlias") String linkingDataverseAlias) { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); Dataverse linked = findDataverseOrDie(linkedDataverseAlias); Dataverse linking = findDataverseOrDie(linkingDataverseAlias); if (linked == null) { diff --git a/src/main/java/edu/harvard/iq/dataverse/api/EditDDI.java b/src/main/java/edu/harvard/iq/dataverse/api/EditDDI.java index 7ca4cffd030..05fccab0e1a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/EditDDI.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/EditDDI.java @@ -1,5 +1,6 @@ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.Permission; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; @@ -33,11 +34,14 @@ import jakarta.inject.Inject; import jakarta.persistence.EntityManager; import jakarta.persistence.PersistenceContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import jakarta.ws.rs.Path; import jakarta.ws.rs.PUT; import jakarta.ws.rs.Consumes; import jakarta.ws.rs.PathParam; +import jakarta.ws.rs.container.ContainerRequestContext; + import javax.xml.stream.XMLStreamException; import javax.xml.stream.XMLInputFactory; import javax.xml.stream.XMLStreamReader; @@ -90,9 +94,10 @@ public class EditDDI extends AbstractApiBean { @PUT - @Consumes("application/xml") + @AuthRequired @Path("{fileId}") - public Response edit (InputStream body, @PathParam("fileId") String fileId) { + @Consumes("application/xml") + public Response edit(@Context ContainerRequestContext crc, InputStream body, @PathParam("fileId") String fileId) { DataFile dataFile = null; try { dataFile = findDataFileOrDie(fileId); @@ -100,7 +105,7 @@ public Response edit (InputStream body, @PathParam("fileId") String fileId) { } catch (WrappedResponse ex) { return ex.getResponse(); } - User apiTokenUser = checkAuth(dataFile); + User apiTokenUser = checkAuth(getRequestUser(crc), dataFile); if (apiTokenUser == null) { return unauthorized("Cannot edit metadata, access denied" ); @@ -421,27 +426,10 @@ private boolean AreDefaultValues(VariableMetadata varMet) { } - private User checkAuth(DataFile dataFile) { - - User apiTokenUser = null; - - try { - apiTokenUser = findUserOrDie(); - } catch (WrappedResponse wr) { - apiTokenUser = null; - logger.log(Level.FINE, "Message from findUserOrDie(): {0}", wr.getMessage()); + private User checkAuth(User requestUser, DataFile dataFile) { + if (!permissionService.requestOn(createDataverseRequest(requestUser), dataFile.getOwner()).has(Permission.EditDataset)) { + return null; } - - if (apiTokenUser != null) { - // used in an API context - if (!permissionService.requestOn(createDataverseRequest(apiTokenUser), dataFile.getOwner()).has(Permission.EditDataset)) { - apiTokenUser = null; - } - } - - return apiTokenUser; - + return requestUser; } } - - diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Files.java b/src/main/java/edu/harvard/iq/dataverse/api/Files.java index 6722266d63c..c8dc1d15df3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Files.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Files.java @@ -14,6 +14,7 @@ import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.TermsOfUseAndAccessValidator; import edu.harvard.iq.dataverse.UserNotificationServiceBean; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.users.ApiToken; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; @@ -43,6 +44,7 @@ import edu.harvard.iq.dataverse.util.FileUtil; import edu.harvard.iq.dataverse.util.StringUtil; import edu.harvard.iq.dataverse.util.SystemConfig; +import static edu.harvard.iq.dataverse.util.json.JsonPrinter.json; import edu.harvard.iq.dataverse.util.json.JsonUtil; import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; import java.io.InputStream; @@ -64,6 +66,7 @@ import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.HttpHeaders; import jakarta.ws.rs.core.MediaType; @@ -121,8 +124,9 @@ private void msgt(String m){ * @return */ @PUT + @AuthRequired @Path("{id}/restrict") - public Response restrictFileInDataset(@PathParam("id") String fileToRestrictId, String restrictStr) { + public Response restrictFileInDataset(@Context ContainerRequestContext crc, @PathParam("id") String fileToRestrictId, String restrictStr) { //create request DataverseRequest dataverseRequest = null; //get the datafile @@ -134,12 +138,8 @@ public Response restrictFileInDataset(@PathParam("id") String fileToRestrictId, } boolean restrict = Boolean.valueOf(restrictStr); - - try { - dataverseRequest = createDataverseRequest(findUserOrDie()); - } catch (WrappedResponse wr) { - return error(BAD_REQUEST, "Couldn't find user to execute command: " + wr.getLocalizedMessage()); - } + + dataverseRequest = createDataverseRequest(getRequestUser(crc)); // try to restrict the datafile try { @@ -178,9 +178,11 @@ public Response restrictFileInDataset(@PathParam("id") String fileToRestrictId, * @return */ @POST + @AuthRequired @Path("{id}/replace") @Consumes(MediaType.MULTIPART_FORM_DATA) public Response replaceFileInDataset( + @Context ContainerRequestContext crc, @PathParam("id") String fileIdOrPersistentId, @FormDataParam("jsonData") String jsonData, @FormDataParam("file") InputStream testFileInputStream, @@ -191,15 +193,8 @@ public Response replaceFileInDataset( if (!systemConfig.isHTTPUpload()) { return error(Response.Status.SERVICE_UNAVAILABLE, BundleUtil.getStringFromBundle("file.api.httpDisabled")); } - // (1) Get the user from the API key - User authUser; - try { - authUser = findUserOrDie(); - } catch (AbstractApiBean.WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, - BundleUtil.getStringFromBundle("file.addreplace.error.auth") - ); - } + // (1) Get the user from the ContainerRequestContext + User authUser = getRequestUser(crc); // (2) Check/Parse the JSON (if uploaded) Boolean forceReplace = false; @@ -322,8 +317,9 @@ public Response replaceFileInDataset( //Much of this code is taken from the replace command, //simplified as we aren't actually switching files @POST + @AuthRequired @Path("{id}/metadata") - public Response updateFileMetadata(@FormDataParam("jsonData") String jsonData, + public Response updateFileMetadata(@Context ContainerRequestContext crc, @FormDataParam("jsonData") String jsonData, @PathParam("id") String fileIdOrPersistentId ) throws DataFileTagException, CommandException { @@ -332,7 +328,7 @@ public Response updateFileMetadata(@FormDataParam("jsonData") String jsonData, try { DataverseRequest req; try { - req = createDataverseRequest(findUserOrDie()); + req = createDataverseRequest(getRequestUser(crc)); } catch (Exception e) { return error(BAD_REQUEST, "Error attempting to request information. Maybe a bad API token?"); } @@ -343,8 +339,6 @@ public Response updateFileMetadata(@FormDataParam("jsonData") String jsonData, return error(BAD_REQUEST, "Error attempting get the requested data file."); } - - User authUser = findUserOrDie(); //You shouldn't be trying to edit a datafile that has been replaced List result = em.createNamedQuery("DataFile.findDataFileThatReplacedId", Long.class) @@ -445,13 +439,86 @@ public Response updateFileMetadata(@FormDataParam("jsonData") String jsonData, .build(); } - @GET + @GET + @AuthRequired + @Path("{id}/draft") + public Response getFileDataDraft(@Context ContainerRequestContext crc, @PathParam("id") String fileIdOrPersistentId, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WrappedResponse, Exception { + return getFileDataResponse(getRequestUser(crc), fileIdOrPersistentId, uriInfo, headers, response, true); + } + + @GET + @AuthRequired + @Path("{id}") + public Response getFileData(@Context ContainerRequestContext crc, @PathParam("id") String fileIdOrPersistentId, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response) throws WrappedResponse, Exception { + return getFileDataResponse(getRequestUser(crc), fileIdOrPersistentId, uriInfo, headers, response, false); + } + + private Response getFileDataResponse(User user, String fileIdOrPersistentId, UriInfo uriInfo, HttpHeaders headers, HttpServletResponse response, boolean draft ){ + + DataverseRequest req; + try { + req = createDataverseRequest(user); + } catch (Exception e) { + return error(BAD_REQUEST, "Error attempting to request information. Maybe a bad API token?"); + } + final DataFile df; + try { + df = execCommand(new GetDataFileCommand(req, findDataFileOrDie(fileIdOrPersistentId))); + } catch (Exception e) { + return error(BAD_REQUEST, "Error attempting get the requested data file."); + } + + FileMetadata fm; + + if (draft) { + try { + fm = execCommand(new GetDraftFileMetadataIfAvailableCommand(req, df)); + } catch (WrappedResponse w) { + return error(BAD_REQUEST, "An error occurred getting a draft version, you may not have permission to access unpublished data on this dataset."); + } + if (null == fm) { + return error(BAD_REQUEST, BundleUtil.getStringFromBundle("files.api.no.draft")); + } + } else { + //first get latest published + //if not available get draft if permissible + + try { + fm = df.getLatestPublishedFileMetadata(); + + } catch (UnsupportedOperationException e) { + try { + fm = execCommand(new GetDraftFileMetadataIfAvailableCommand(req, df)); + } catch (WrappedResponse w) { + return error(BAD_REQUEST, "An error occurred getting a draft version, you may not have permission to access unpublished data on this dataset."); + } + if (null == fm) { + return error(BAD_REQUEST, BundleUtil.getStringFromBundle("files.api.no.draft")); + } + } + + } + + if (fm.getDatasetVersion().isReleased()) { + MakeDataCountLoggingServiceBean.MakeDataCountEntry entry = new MakeDataCountLoggingServiceBean.MakeDataCountEntry(uriInfo, headers, dvRequestService, df); + mdcLogService.logEntry(entry); + } + + return Response.ok(Json.createObjectBuilder() + .add("status", ApiConstants.STATUS_OK) + .add("data", json(fm)).build()) + .type(MediaType.APPLICATION_JSON) + .build(); + } + + @GET + @AuthRequired @Path("{id}/metadata") - public Response getFileMetadata(@PathParam("id") String fileIdOrPersistentId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response, Boolean getDraft) throws WrappedResponse, Exception { + public Response getFileMetadata(@Context ContainerRequestContext crc, @PathParam("id") String fileIdOrPersistentId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response, Boolean getDraft) throws WrappedResponse, Exception { //ToDo - versionId is not used - can't get metadata for earlier versions DataverseRequest req; try { - req = createDataverseRequest(findUserOrDie()); + req = createDataverseRequest(getRequestUser(crc)); } catch (Exception e) { return error(BAD_REQUEST, "Error attempting to request information. Maybe a bad API token?"); } @@ -470,7 +537,7 @@ public Response getFileMetadata(@PathParam("id") String fileIdOrPersistentId, @P return error(BAD_REQUEST, "An error occurred getting a draft version, you may not have permission to access unpublished data on this dataset." ); } if(null == fm) { - return error(BAD_REQUEST, "No draft availabile for this dataset"); + return error(BAD_REQUEST, BundleUtil.getStringFromBundle("files.api.no.draft")); } } else { fm = df.getLatestPublishedFileMetadata(); @@ -486,15 +553,18 @@ public Response getFileMetadata(@PathParam("id") String fileIdOrPersistentId, @P .type(MediaType.TEXT_PLAIN) //Our plain text string is already json .build(); } - @GET + + @GET + @AuthRequired @Path("{id}/metadata/draft") - public Response getFileMetadataDraft(@PathParam("id") String fileIdOrPersistentId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response, Boolean getDraft) throws WrappedResponse, Exception { - return getFileMetadata(fileIdOrPersistentId, versionId, uriInfo, headers, response, true); + public Response getFileMetadataDraft(@Context ContainerRequestContext crc, @PathParam("id") String fileIdOrPersistentId, @PathParam("versionId") String versionId, @Context UriInfo uriInfo, @Context HttpHeaders headers, @Context HttpServletResponse response, Boolean getDraft) throws WrappedResponse, Exception { + return getFileMetadata(crc, fileIdOrPersistentId, versionId, uriInfo, headers, response, true); } - @Path("{id}/uningest") @POST - public Response uningestDatafile(@PathParam("id") String id) { + @AuthRequired + @Path("{id}/uningest") + public Response uningestDatafile(@Context ContainerRequestContext crc, @PathParam("id") String id) { DataFile dataFile; try { @@ -511,7 +581,7 @@ public Response uningestDatafile(@PathParam("id") String id) { } try { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); execCommand(new UningestFileCommand(req, dataFile)); Long dataFileId = dataFile.getId(); dataFile = fileService.find(dataFileId); @@ -523,22 +593,22 @@ public Response uningestDatafile(@PathParam("id") String id) { } } - + // reingest attempts to queue an *existing* DataFile // for tabular ingest. It can be used on non-tabular datafiles; to try to // ingest a file that has previously failed ingest, or to ingest a file of a // type for which ingest was not previously supported. // We are considering making it possible, in the future, to reingest // a datafile that's already ingested as Tabular; for example, to address a - // bug that has been found in an ingest plugin. - - @Path("{id}/reingest") + // bug that has been found in an ingest plugin. @POST - public Response reingest(@PathParam("id") String id) { + @AuthRequired + @Path("{id}/reingest") + public Response reingest(@Context ContainerRequestContext crc, @PathParam("id") String id) { AuthenticatedUser u; try { - u = findAuthenticatedUserOrDie(); + u = getRequestAuthenticatedUserOrDie(crc); if (!u.isSuperuser()) { return error(Response.Status.FORBIDDEN, "This API call can be used by superusers only"); } @@ -601,13 +671,14 @@ public Response reingest(@PathParam("id") String id) { } - @Path("{id}/redetect") @POST - public Response redetectDatafile(@PathParam("id") String id, @QueryParam("dryRun") boolean dryRun) { + @AuthRequired + @Path("{id}/redetect") + public Response redetectDatafile(@Context ContainerRequestContext crc, @PathParam("id") String id, @QueryParam("dryRun") boolean dryRun) { try { DataFile dataFileIn = findDataFileOrDie(id); String originalContentType = dataFileIn.getContentType(); - DataFile dataFileOut = execCommand(new RedetectFileTypeCommand(createDataverseRequest(findUserOrDie()), dataFileIn, dryRun)); + DataFile dataFileOut = execCommand(new RedetectFileTypeCommand(createDataverseRequest(getRequestUser(crc)), dataFileIn, dryRun)); NullSafeJsonBuilder result = NullSafeJsonBuilder.jsonObjectBuilder() .add("dryRun", dryRun) .add("oldContentType", originalContentType) @@ -618,6 +689,28 @@ public Response redetectDatafile(@PathParam("id") String id, @QueryParam("dryRun } } + @POST + @AuthRequired + @Path("{id}/extractNcml") + public Response extractNcml(@Context ContainerRequestContext crc, @PathParam("id") String id) { + try { + AuthenticatedUser au = getRequestAuthenticatedUserOrDie(crc); + if (!au.isSuperuser()) { + // We can always make a command in the future if there's a need + // for non-superusers to call this API. + return error(Response.Status.FORBIDDEN, "This API call can be used by superusers only"); + } + DataFile dataFileIn = findDataFileOrDie(id); + java.nio.file.Path tempLocationPath = null; + boolean successOrFail = ingestService.extractMetadataNcml(dataFileIn, tempLocationPath); + NullSafeJsonBuilder result = NullSafeJsonBuilder.jsonObjectBuilder() + .add("result", successOrFail); + return ok(result); + } catch (WrappedResponse wr) { + return wr.getResponse(); + } + } + /** * Attempting to run metadata export, for all the formats for which we have * metadata Exporters. @@ -644,33 +737,30 @@ private void exportDatasetMetadata(SettingsServiceBean settingsServiceBean, Data // This supports the cases where a tool is accessing a restricted resource (e.g. // preview of a draft file), or public case. @GET + @AuthRequired @Path("{id}/metadata/{fmid}/toolparams/{tid}") - public Response getExternalToolFMParams(@PathParam("tid") long externalToolId, + public Response getExternalToolFMParams(@Context ContainerRequestContext crc, @PathParam("tid") long externalToolId, @PathParam("id") String fileId, @PathParam("fmid") long fmid, @QueryParam(value = "locale") String locale) { - try { - ExternalTool externalTool = externalToolService.findById(externalToolId); - if(externalTool == null) { - return error(BAD_REQUEST, "External tool not found."); - } - if (!ExternalTool.Scope.FILE.equals(externalTool.getScope())) { - return error(BAD_REQUEST, "External tool does not have file scope."); - } - ApiToken apiToken = null; - User u = findUserOrDie(); - if (u instanceof AuthenticatedUser) { - apiToken = authSvc.findApiTokenByUser((AuthenticatedUser) u); - } - FileMetadata target = fileSvc.findFileMetadata(fmid); - if (target == null) { - return error(BAD_REQUEST, "FileMetadata not found."); - } + ExternalTool externalTool = externalToolService.findById(externalToolId); + if(externalTool == null) { + return error(BAD_REQUEST, "External tool not found."); + } + if (!ExternalTool.Scope.FILE.equals(externalTool.getScope())) { + return error(BAD_REQUEST, "External tool does not have file scope."); + } + ApiToken apiToken = null; + User u = getRequestUser(crc); + if (u instanceof AuthenticatedUser) { + apiToken = authSvc.findApiTokenByUser((AuthenticatedUser) u); + } + FileMetadata target = fileSvc.findFileMetadata(fmid); + if (target == null) { + return error(BAD_REQUEST, "FileMetadata not found."); + } - ExternalToolHandler eth = null; + ExternalToolHandler eth = null; - eth = new ExternalToolHandler(externalTool, target.getDataFile(), apiToken, target, locale); - return ok(eth.createPostBody(eth.getParams(JsonUtil.getJsonObject(externalTool.getToolParameters())))); - } catch (WrappedResponse wr) { - return wr.getResponse(); - } + eth = new ExternalToolHandler(externalTool, target.getDataFile(), apiToken, target, locale); + return ok(eth.createPostBody(eth.getParams(JsonUtil.getJsonObject(externalTool.getToolParameters())))); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/HarvestingClients.java b/src/main/java/edu/harvard/iq/dataverse/api/HarvestingClients.java index 8fb3b319dff..b0de5542a0c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/HarvestingClients.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/HarvestingClients.java @@ -2,6 +2,7 @@ import edu.harvard.iq.dataverse.Dataverse; import edu.harvard.iq.dataverse.DataverseServiceBean; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.harvest.client.HarvestingClient; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; @@ -15,6 +16,7 @@ import edu.harvard.iq.dataverse.util.BundleUtil; import edu.harvard.iq.dataverse.util.StringUtil; import edu.harvard.iq.dataverse.util.json.JsonParseException; +import edu.harvard.iq.dataverse.util.json.JsonPrinter; import jakarta.json.JsonObjectBuilder; import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; import java.io.IOException; @@ -34,6 +36,8 @@ import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; @Path("harvest/clients") @@ -55,8 +59,9 @@ public class HarvestingClients extends AbstractApiBean { * optionally, plain text output may be provided as well. */ @GET + @AuthRequired @Path("/") - public Response harvestingClients(@QueryParam("key") String apiKey) throws IOException { + public Response harvestingClients(@Context ContainerRequestContext crc, @QueryParam("key") String apiKey) throws IOException { List harvestingClients = null; try { @@ -80,7 +85,7 @@ public Response harvestingClients(@QueryParam("key") String apiKey) throws IOExc // the permission to view this harvesting client config. -- L.A. 4.4 HarvestingClient retrievedHarvestingClient = null; try { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); retrievedHarvestingClient = execCommand( new GetHarvestingClientCommand(req, harvestingClient)); } catch (Exception ex) { // Don't do anything. @@ -89,7 +94,7 @@ public Response harvestingClients(@QueryParam("key") String apiKey) throws IOExc } if (retrievedHarvestingClient != null) { - hcArr.add(harvestingConfigAsJson(retrievedHarvestingClient)); + hcArr.add(JsonPrinter.json(retrievedHarvestingClient)); } } @@ -97,8 +102,9 @@ public Response harvestingClients(@QueryParam("key") String apiKey) throws IOExc } @GET + @AuthRequired @Path("{nickName}") - public Response harvestingClient(@PathParam("nickName") String nickName, @QueryParam("key") String apiKey) throws IOException { + public Response harvestingClient(@Context ContainerRequestContext crc, @PathParam("nickName") String nickName, @QueryParam("key") String apiKey) throws IOException { HarvestingClient harvestingClient = null; try { @@ -122,7 +128,7 @@ public Response harvestingClient(@PathParam("nickName") String nickName, @QueryP // findUserOrDie() and execCommand() both throw WrappedResponse // exception, that already has a proper HTTP response in it. - retrievedHarvestingClient = execCommand(new GetHarvestingClientCommand(createDataverseRequest(findUserOrDie()), harvestingClient)); + retrievedHarvestingClient = execCommand(new GetHarvestingClientCommand(createDataverseRequest(getRequestUser(crc)), harvestingClient)); logger.fine("retrieved Harvesting Client " + retrievedHarvestingClient.getName() + " with the GetHarvestingClient command."); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -137,7 +143,7 @@ public Response harvestingClient(@PathParam("nickName") String nickName, @QueryP } try { - return ok(harvestingConfigAsJson(retrievedHarvestingClient)); + return ok(JsonPrinter.json(retrievedHarvestingClient)); } catch (Exception ex) { logger.warning("Unknown exception caught while trying to format harvesting client config as json: "+ex.getMessage()); return error( Response.Status.BAD_REQUEST, @@ -146,12 +152,13 @@ public Response harvestingClient(@PathParam("nickName") String nickName, @QueryP } @POST + @AuthRequired @Path("{nickName}") - public Response createHarvestingClient(String jsonBody, @PathParam("nickName") String nickName, @QueryParam("key") String apiKey) throws IOException, JsonParseException { + public Response createHarvestingClient(@Context ContainerRequestContext crc, String jsonBody, @PathParam("nickName") String nickName, @QueryParam("key") String apiKey) throws IOException, JsonParseException { // Per the discussion during the QA of PR #9174, we decided to make // the create/edit APIs superuser-only (the delete API was already so) try { - User u = findUserOrDie(); + User u = getRequestUser(crc); if ((!(u instanceof AuthenticatedUser) || !u.isSuperuser())) { throw new WrappedResponse(error(Response.Status.UNAUTHORIZED, "Only superusers can create harvesting clients.")); } @@ -215,9 +222,9 @@ public Response createHarvestingClient(String jsonBody, @PathParam("nickName") S } ownerDataverse.getHarvestingClientConfigs().add(harvestingClient); - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); harvestingClient = execCommand(new CreateHarvestingClientCommand(req, harvestingClient)); - return created( "/harvest/clients/" + nickName, harvestingConfigAsJson(harvestingClient)); + return created( "/harvest/clients/" + nickName, JsonPrinter.json(harvestingClient)); } catch (JsonParseException ex) { return error( Response.Status.BAD_REQUEST, "Error parsing harvesting client: " + ex.getMessage() ); @@ -230,10 +237,11 @@ public Response createHarvestingClient(String jsonBody, @PathParam("nickName") S } @PUT + @AuthRequired @Path("{nickName}") - public Response modifyHarvestingClient(String jsonBody, @PathParam("nickName") String nickName, @QueryParam("key") String apiKey) throws IOException, JsonParseException { + public Response modifyHarvestingClient(@Context ContainerRequestContext crc, String jsonBody, @PathParam("nickName") String nickName, @QueryParam("key") String apiKey) throws IOException, JsonParseException { try { - User u = findUserOrDie(); + User u = getRequestUser(crc); if ((!(u instanceof AuthenticatedUser) || !u.isSuperuser())) { throw new WrappedResponse(error(Response.Status.UNAUTHORIZED, "Only superusers can modify harvesting clients.")); } @@ -256,7 +264,7 @@ public Response modifyHarvestingClient(String jsonBody, @PathParam("nickName") S String ownerDataverseAlias = harvestingClient.getDataverse().getAlias(); try ( StringReader rdr = new StringReader(jsonBody) ) { - DataverseRequest req = createDataverseRequest(findUserOrDie()); + DataverseRequest req = createDataverseRequest(getRequestUser(crc)); JsonObject json = Json.createReader(rdr).readObject(); HarvestingClient newHarvestingClient = new HarvestingClient(); @@ -269,6 +277,8 @@ public Response modifyHarvestingClient(String jsonBody, @PathParam("nickName") S } // Go through the supported editable fields and update the client accordingly: + // TODO: We may want to reevaluate whether we really want/need *all* + // of these fields to be editable. if (newHarvestingClient.getHarvestingUrl() != null) { harvestingClient.setHarvestingUrl(newHarvestingClient.getHarvestingUrl()); @@ -288,10 +298,13 @@ public Response modifyHarvestingClient(String jsonBody, @PathParam("nickName") S if (newHarvestingClient.getHarvestStyle() != null) { harvestingClient.setHarvestStyle(newHarvestingClient.getHarvestStyle()); } + if (newHarvestingClient.getCustomHttpHeaders() != null) { + harvestingClient.setCustomHttpHeaders(newHarvestingClient.getCustomHttpHeaders()); + } // TODO: Make schedule configurable via this API too. harvestingClient = execCommand( new UpdateHarvestingClientCommand(req, harvestingClient)); - return ok( "/harvest/clients/" + nickName, harvestingConfigAsJson(harvestingClient)); + return ok( "/harvest/clients/" + nickName, JsonPrinter.json(harvestingClient)); // harvestingConfigAsJson(harvestingClient)); } catch (JsonParseException ex) { return error( Response.Status.BAD_REQUEST, "Error parsing harvesting client: " + ex.getMessage() ); @@ -304,15 +317,16 @@ public Response modifyHarvestingClient(String jsonBody, @PathParam("nickName") S } @DELETE + @AuthRequired @Path("{nickName}") - public Response deleteHarvestingClient(@PathParam("nickName") String nickName) throws IOException { + public Response deleteHarvestingClient(@Context ContainerRequestContext crc, @PathParam("nickName") String nickName) throws IOException { // Deleting a client can take a while (if there's a large amnount of // harvested content associated with it). So instead of calling the command // directly, we will be calling an async. service bean method. try { - User u = findUserOrDie(); + User u = getRequestUser(crc); if ((!(u instanceof AuthenticatedUser) || !u.isSuperuser())) { throw new WrappedResponse(error(Response.Status.UNAUTHORIZED, "Only superusers can delete harvesting clients.")); } @@ -361,14 +375,15 @@ public Response deleteHarvestingClient(@PathParam("nickName") String nickName) t // This POST starts a new harvesting run: @POST + @AuthRequired @Path("{nickName}/run") - public Response startHarvestingJob(@PathParam("nickName") String clientNickname, @QueryParam("key") String apiKey) throws IOException { + public Response startHarvestingJob(@Context ContainerRequestContext crc, @PathParam("nickName") String clientNickname, @QueryParam("key") String apiKey) throws IOException { try { AuthenticatedUser authenticatedUser = null; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse wr) { return error(Response.Status.UNAUTHORIZED, "Authentication required to use this API method"); } @@ -391,32 +406,4 @@ public Response startHarvestingJob(@PathParam("nickName") String clientNickname, } return this.accepted(); } - - /* Auxiliary, helper methods: */ - - public static JsonObjectBuilder harvestingConfigAsJson(HarvestingClient harvestingConfig) { - if (harvestingConfig == null) { - return null; - } - - - return jsonObjectBuilder().add("nickName", harvestingConfig.getName()). - add("dataverseAlias", harvestingConfig.getDataverse().getAlias()). - add("type", harvestingConfig.getHarvestType()). - add("style", harvestingConfig.getHarvestStyle()). - add("harvestUrl", harvestingConfig.getHarvestingUrl()). - add("archiveUrl", harvestingConfig.getArchiveUrl()). - add("archiveDescription",harvestingConfig.getArchiveDescription()). - add("metadataFormat", harvestingConfig.getMetadataPrefix()). - add("set", harvestingConfig.getHarvestingSet() == null ? "N/A" : harvestingConfig.getHarvestingSet()). - add("schedule", harvestingConfig.isScheduled() ? harvestingConfig.getScheduleDescription() : "none"). - add("status", harvestingConfig.isHarvestingNow() ? "inProgress" : "inActive"). - add("lastHarvest", harvestingConfig.getLastHarvestTime() == null ? "N/A" : harvestingConfig.getLastHarvestTime().toString()). - add("lastResult", harvestingConfig.getLastResult()). - add("lastSuccessful", harvestingConfig.getLastSuccessfulHarvestTime() == null ? "N/A" : harvestingConfig.getLastSuccessfulHarvestTime().toString()). - add("lastNonEmpty", harvestingConfig.getLastNonEmptyHarvestTime() == null ? "N/A" : harvestingConfig.getLastNonEmptyHarvestTime().toString()). - add("lastDatasetsHarvested", harvestingConfig.getLastHarvestedDatasetCount() == null ? "N/A" : harvestingConfig.getLastHarvestedDatasetCount().toString()). - add("lastDatasetsDeleted", harvestingConfig.getLastDeletedDatasetCount() == null ? "N/A" : harvestingConfig.getLastDeletedDatasetCount().toString()). - add("lastDatasetsFailed", harvestingConfig.getLastFailedDatasetCount() == null ? "N/A" : harvestingConfig.getLastFailedDatasetCount().toString()); - } } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/HarvestingServer.java b/src/main/java/edu/harvard/iq/dataverse/api/HarvestingServer.java index b5c448b563a..308b910c425 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/HarvestingServer.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/HarvestingServer.java @@ -5,6 +5,7 @@ */ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.harvest.server.OAISet; import edu.harvard.iq.dataverse.harvest.server.OAISetServiceBean; @@ -30,6 +31,8 @@ import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import org.apache.commons.lang3.StringUtils; @@ -104,14 +107,15 @@ public Response oaiSet(@PathParam("specname") String spec, @QueryParam("key") St * "description":$optional_set_description,"definition":$set_search_query_string}. */ @POST + @AuthRequired @Path("/add") - public Response createOaiSet(String jsonBody, @QueryParam("key") String apiKey) throws IOException, JsonParseException { + public Response createOaiSet(@Context ContainerRequestContext crc, String jsonBody, @QueryParam("key") String apiKey) throws IOException, JsonParseException { /* * authorization modeled after the UI (aka HarvestingSetsPage) */ AuthenticatedUser dvUser; try { - dvUser = findAuthenticatedUserOrDie(); + dvUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -173,12 +177,13 @@ public Response createOaiSet(String jsonBody, @QueryParam("key") String apiKey) } @PUT + @AuthRequired @Path("{specname}") - public Response modifyOaiSet(String jsonBody, @PathParam("specname") String spec, @QueryParam("key") String apiKey) throws IOException, JsonParseException { + public Response modifyOaiSet(@Context ContainerRequestContext crc, String jsonBody, @PathParam("specname") String spec, @QueryParam("key") String apiKey) throws IOException, JsonParseException { AuthenticatedUser dvUser; try { - dvUser = findAuthenticatedUserOrDie(); + dvUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse wr) { return wr.getResponse(); } @@ -225,12 +230,13 @@ public Response modifyOaiSet(String jsonBody, @PathParam("specname") String spec } @DELETE + @AuthRequired @Path("{specname}") - public Response deleteOaiSet(@PathParam("specname") String spec, @QueryParam("key") String apiKey) { + public Response deleteOaiSet(@Context ContainerRequestContext crc, @PathParam("specname") String spec, @QueryParam("key") String apiKey) { AuthenticatedUser dvUser; try { - dvUser = findAuthenticatedUserOrDie(); + dvUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse wr) { return wr.getResponse(); } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Index.java b/src/main/java/edu/harvard/iq/dataverse/api/Index.java index 4cabeafc282..c82d3197ded 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Index.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Index.java @@ -15,6 +15,7 @@ import edu.harvard.iq.dataverse.DvObjectServiceBean; import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.RoleAssignment; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.GuestUser; import edu.harvard.iq.dataverse.search.SearchServiceBean; @@ -59,6 +60,8 @@ import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import jakarta.ws.rs.core.Response.Status; import org.apache.solr.client.solrj.SolrServerException; @@ -633,15 +636,16 @@ public Response deleteTimestamp(@PathParam("dvObjectId") long dvObjectId) { } @GET + @AuthRequired @Path("filesearch") - public Response filesearch(@QueryParam("persistentId") String persistentId, @QueryParam("semanticVersion") String semanticVersion, @QueryParam("q") String userSuppliedQuery) { + public Response filesearch(@Context ContainerRequestContext crc, @QueryParam("persistentId") String persistentId, @QueryParam("semanticVersion") String semanticVersion, @QueryParam("q") String userSuppliedQuery) { Dataset dataset = datasetService.findByGlobalId(persistentId); if (dataset == null) { return error(Status.BAD_REQUEST, "Could not find dataset with persistent id " + persistentId); } User user = GuestUser.get(); try { - AuthenticatedUser authenticatedUser = findAuthenticatedUserOrDie(); + AuthenticatedUser authenticatedUser = getRequestAuthenticatedUserOrDie(crc); if (authenticatedUser != null) { user = authenticatedUser; } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Info.java b/src/main/java/edu/harvard/iq/dataverse/api/Info.java index c82c8327a3e..ff0c6433d62 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Info.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Info.java @@ -1,5 +1,6 @@ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.SystemConfig; @@ -8,6 +9,8 @@ import jakarta.json.JsonValue; import jakarta.ws.rs.GET; import jakarta.ws.rs.Path; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; @Path("info") @@ -31,26 +34,29 @@ public Response getDatasetPublishPopupCustomText() { } @GET + @AuthRequired @Path("version") - public Response getInfo() { + public Response getInfo(@Context ContainerRequestContext crc) { String versionStr = systemConfig.getVersion(true); String[] comps = versionStr.split("build",2); String version = comps[0].trim(); JsonValue build = comps.length > 1 ? Json.createArrayBuilder().add(comps[1].trim()).build().get(0) : JsonValue.NULL; return response( req -> ok( Json.createObjectBuilder().add("version", version) - .add("build", build))); + .add("build", build)), getRequestUser(crc)); } @GET + @AuthRequired @Path("server") - public Response getServer() { - return response( req -> ok(JvmSettings.FQDN.lookup())); + public Response getServer(@Context ContainerRequestContext crc) { + return response( req -> ok(JvmSettings.FQDN.lookup()), getRequestUser(crc)); } @GET + @AuthRequired @Path("apiTermsOfUse") - public Response getTermsOfUse() { - return response( req -> ok(systemConfig.getApiTermsOfUse())); + public Response getTermsOfUse(@Context ContainerRequestContext crc) { + return response( req -> ok(systemConfig.getApiTermsOfUse()), getRequestUser(crc)); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Licenses.java b/src/main/java/edu/harvard/iq/dataverse/api/Licenses.java index 8c82f9ba462..ab50ebbf2e4 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Licenses.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Licenses.java @@ -10,11 +10,14 @@ import jakarta.ws.rs.PUT; import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import java.util.logging.Logger; import jakarta.ejb.Stateless; import jakarta.ws.rs.core.Response.Status; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.license.License; import edu.harvard.iq.dataverse.util.json.JsonPrinter; @@ -51,11 +54,12 @@ public Response getLicenseById(@PathParam("id") long id) { } @POST + @AuthRequired @Path("/") - public Response addLicense(License license) { + public Response addLicense(@Context ContainerRequestContext crc, License license) { User authenticatedUser; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); if (!authenticatedUser.isSuperuser()) { return error(Status.FORBIDDEN, "must be superuser"); } @@ -86,11 +90,12 @@ public Response getDefault() { } @PUT + @AuthRequired @Path("/default/{id}") - public Response setDefault(@PathParam("id") long id) { + public Response setDefault(@Context ContainerRequestContext crc, @PathParam("id") long id) { User authenticatedUser; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); if (!authenticatedUser.isSuperuser()) { return error(Status.FORBIDDEN, "must be superuser"); } @@ -117,11 +122,12 @@ public Response setDefault(@PathParam("id") long id) { } @PUT + @AuthRequired @Path("/{id}/:active/{activeState}") - public Response setActiveState(@PathParam("id") long id, @PathParam("activeState") boolean active) { + public Response setActiveState(@Context ContainerRequestContext crc, @PathParam("id") long id, @PathParam("activeState") boolean active) { User authenticatedUser; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); if (!authenticatedUser.isSuperuser()) { return error(Status.FORBIDDEN, "must be superuser"); } @@ -147,11 +153,12 @@ public Response setActiveState(@PathParam("id") long id, @PathParam("activeState } @PUT + @AuthRequired @Path("/{id}/:sortOrder/{sortOrder}") - public Response setSortOrder(@PathParam("id") long id, @PathParam("sortOrder") long sortOrder) { + public Response setSortOrder(@Context ContainerRequestContext crc, @PathParam("id") long id, @PathParam("sortOrder") long sortOrder) { User authenticatedUser; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); if (!authenticatedUser.isSuperuser()) { return error(Status.FORBIDDEN, "must be superuser"); } @@ -178,11 +185,12 @@ public Response setSortOrder(@PathParam("id") long id, @PathParam("sortOrder") l } @DELETE + @AuthRequired @Path("/{id}") - public Response deleteLicenseById(@PathParam("id") long id) { + public Response deleteLicenseById(@Context ContainerRequestContext crc, @PathParam("id") long id) { User authenticatedUser; try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); if (!authenticatedUser.isSuperuser()) { return error(Status.FORBIDDEN, "must be superuser"); } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Notifications.java b/src/main/java/edu/harvard/iq/dataverse/api/Notifications.java index 87607d87d84..3db305b738f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Notifications.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Notifications.java @@ -3,6 +3,7 @@ import edu.harvard.iq.dataverse.MailServiceBean; import edu.harvard.iq.dataverse.UserNotification; import edu.harvard.iq.dataverse.UserNotification.Type; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.util.MailUtil; @@ -19,6 +20,8 @@ import jakarta.ws.rs.PUT; import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import java.util.List; import java.util.Optional; @@ -32,17 +35,10 @@ public class Notifications extends AbstractApiBean { MailServiceBean mailService; @GET + @AuthRequired @Path("/all") - public Response getAllNotificationsForUser() { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response getAllNotificationsForUser(@Context ContainerRequestContext crc) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); @@ -86,17 +82,10 @@ private JsonArrayBuilder getReasonsForReturn(UserNotification notification) { } @DELETE + @AuthRequired @Path("/{id}") - public Response deleteNotificationForUser(@PathParam("id") long id) { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response deleteNotificationForUser(@Context ContainerRequestContext crc, @PathParam("id") long id) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); @@ -115,17 +104,10 @@ public Response deleteNotificationForUser(@PathParam("id") long id) { } @GET + @AuthRequired @Path("/mutedEmails") - public Response getMutedEmailsForUser() { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response getMutedEmailsForUser(@Context ContainerRequestContext crc) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); @@ -141,17 +123,10 @@ public Response getMutedEmailsForUser() { } @PUT + @AuthRequired @Path("/mutedEmails/{typeName}") - public Response muteEmailsForUser(@PathParam("typeName") String typeName) { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response muteEmailsForUser(@Context ContainerRequestContext crc, @PathParam("typeName") String typeName) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); @@ -172,17 +147,10 @@ public Response muteEmailsForUser(@PathParam("typeName") String typeName) { } @DELETE + @AuthRequired @Path("/mutedEmails/{typeName}") - public Response unmuteEmailsForUser(@PathParam("typeName") String typeName) { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response unmuteEmailsForUser(@Context ContainerRequestContext crc, @PathParam("typeName") String typeName) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); @@ -203,17 +171,10 @@ public Response unmuteEmailsForUser(@PathParam("typeName") String typeName) { } @GET + @AuthRequired @Path("/mutedNotifications") - public Response getMutedNotificationsForUser() { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response getMutedNotificationsForUser(@Context ContainerRequestContext crc) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); @@ -229,17 +190,10 @@ public Response getMutedNotificationsForUser() { } @PUT + @AuthRequired @Path("/mutedNotifications/{typeName}") - public Response muteNotificationsForUser(@PathParam("typeName") String typeName) { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response muteNotificationsForUser(@Context ContainerRequestContext crc, @PathParam("typeName") String typeName) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); @@ -260,17 +214,10 @@ public Response muteNotificationsForUser(@PathParam("typeName") String typeName) } @DELETE + @AuthRequired @Path("/mutedNotifications/{typeName}") - public Response unmuteNotificationsForUser(@PathParam("typeName") String typeName) { - User user; - try { - user = findUserOrDie(); - } catch (WrappedResponse ex) { - return error(Response.Status.UNAUTHORIZED, "You must supply an API token."); - } - if (user == null) { - return error(Response.Status.BAD_REQUEST, "A user could not be found based on the API token."); - } + public Response unmuteNotificationsForUser(@Context ContainerRequestContext crc, @PathParam("typeName") String typeName) { + User user = getRequestUser(crc); if (!(user instanceof AuthenticatedUser)) { // It's unlikely we'll reach this error. A Guest doesn't have an API token and would have been blocked above. return error(Response.Status.BAD_REQUEST, "Only an AuthenticatedUser can have notifications."); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Pids.java b/src/main/java/edu/harvard/iq/dataverse/api/Pids.java index 4613b6ee78f..5229fd55336 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Pids.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Pids.java @@ -1,6 +1,7 @@ package edu.harvard.iq.dataverse.api; import edu.harvard.iq.dataverse.Dataset; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.engine.command.impl.DeletePidCommand; import edu.harvard.iq.dataverse.engine.command.impl.ReservePidCommand; @@ -21,6 +22,8 @@ import jakarta.ws.rs.PathParam; import jakarta.ws.rs.Produces; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.MediaType; import jakarta.ws.rs.core.Response; @@ -36,15 +39,12 @@ public class Pids extends AbstractApiBean { @GET + @AuthRequired @Produces(MediaType.APPLICATION_JSON) - public Response getPid(@QueryParam("persistentId") String persistentId) { - try { - User user = findUserOrDie(); - if (!user.isSuperuser()) { - return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("admin.api.auth.mustBeSuperUser")); - } - } catch (WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("api.errors.invalidApiToken")); + public Response getPid(@Context ContainerRequestContext crc, @QueryParam("persistentId") String persistentId) { + User user = getRequestUser(crc); + if (!user.isSuperuser()) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("admin.api.auth.mustBeSuperUser")); } String baseUrl = systemConfig.getDataCiteRestApiUrlString(); String username = System.getProperty("doi.username"); @@ -60,16 +60,13 @@ public Response getPid(@QueryParam("persistentId") String persistentId) { } @GET + @AuthRequired @Produces(MediaType.APPLICATION_JSON) @Path("unreserved") - public Response getUnreserved(@QueryParam("persistentId") String persistentId) { - try { - User user = findUserOrDie(); - if (!user.isSuperuser()) { - return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("admin.api.auth.mustBeSuperUser")); - } - } catch (WrappedResponse ex) { - return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("api.errors.invalidApiToken")); + public Response getUnreserved(@Context ContainerRequestContext crc, @QueryParam("persistentId") String persistentId) { + User user = getRequestUser(crc); + if (!user.isSuperuser()) { + return error(Response.Status.FORBIDDEN, BundleUtil.getStringFromBundle("admin.api.auth.mustBeSuperUser")); } JsonArrayBuilder unreserved = Json.createArrayBuilder(); @@ -93,12 +90,13 @@ public Response getUnreserved(@QueryParam("persistentId") String persistentId) { } @POST + @AuthRequired @Produces(MediaType.APPLICATION_JSON) @Path("{id}/reserve") - public Response reservePid(@PathParam("id") String idSupplied) { + public Response reservePid(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { try { Dataset dataset = findDatasetOrDie(idSupplied); - execCommand(new ReservePidCommand(createDataverseRequest(findUserOrDie()), dataset)); + execCommand(new ReservePidCommand(createDataverseRequest(getRequestUser(crc)), dataset)); return ok(BundleUtil.getStringFromBundle("pids.api.reservePid.success", Arrays.asList(dataset.getGlobalId().asString()))); } catch (WrappedResponse ex) { return ex.getResponse(); @@ -106,9 +104,10 @@ public Response reservePid(@PathParam("id") String idSupplied) { } @DELETE + @AuthRequired @Produces(MediaType.APPLICATION_JSON) @Path("{id}/delete") - public Response deletePid(@PathParam("id") String idSupplied) { + public Response deletePid(@Context ContainerRequestContext crc, @PathParam("id") String idSupplied) { try { Dataset dataset = findDatasetOrDie(idSupplied); //Restrict to never-published datasets (that should have draft/nonpublic pids). The underlying code will invalidate @@ -117,7 +116,7 @@ public Response deletePid(@PathParam("id") String idSupplied) { if(dataset.isReleased()) { return badRequest("Not allowed for Datasets that have been published."); } - execCommand(new DeletePidCommand(createDataverseRequest(findUserOrDie()), dataset)); + execCommand(new DeletePidCommand(createDataverseRequest(getRequestUser(crc)), dataset)); return ok(BundleUtil.getStringFromBundle("pids.api.deletePid.success", Arrays.asList(dataset.getGlobalId().asString()))); } catch (WrappedResponse ex) { return ex.getResponse(); diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Prov.java b/src/main/java/edu/harvard/iq/dataverse/api/Prov.java index bb351188114..78ff0f937b7 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Prov.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Prov.java @@ -1,6 +1,7 @@ package edu.harvard.iq.dataverse.api; import edu.harvard.iq.dataverse.DataFile; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.provenance.ProvEntityFileData; import edu.harvard.iq.dataverse.provenance.ProvInvestigator; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; @@ -26,6 +27,8 @@ import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import static jakarta.ws.rs.core.Response.Status.BAD_REQUEST; import static jakarta.ws.rs.core.Response.Status.FORBIDDEN; @@ -39,9 +42,10 @@ public class Prov extends AbstractApiBean { /** Provenance JSON methods **/ @POST + @AuthRequired @Path("{id}/prov-json") @Consumes("application/json") - public Response addProvJson(String body, @PathParam("id") String idSupplied, @QueryParam("entityName") String entityName) { + public Response addProvJson(@Context ContainerRequestContext crc, String body, @PathParam("id") String idSupplied, @QueryParam("entityName") String entityName) { if(!systemConfig.isProvCollectionEnabled()) { return error(FORBIDDEN, BundleUtil.getStringFromBundle("api.prov.error.provDisabled")); } @@ -68,7 +72,7 @@ public Response addProvJson(String body, @PathParam("id") String idSupplied, @Qu return error(BAD_REQUEST, BundleUtil.getStringFromBundle("api.prov.error.entityMismatch")); } - execCommand(new PersistProvJsonCommand(createDataverseRequest(findUserOrDie()), dataFile , body, entityName, true)); + execCommand(new PersistProvJsonCommand(createDataverseRequest(getRequestUser(crc)), dataFile , body, entityName, true)); JsonObjectBuilder jsonResponse = Json.createObjectBuilder(); jsonResponse.add("message", BundleUtil.getStringFromBundle("api.prov.provJsonSaved") + " " + dataFile.getDisplayName()); return ok(jsonResponse); @@ -78,8 +82,9 @@ public Response addProvJson(String body, @PathParam("id") String idSupplied, @Qu } @DELETE + @AuthRequired @Path("{id}/prov-json") - public Response deleteProvJson(String body, @PathParam("id") String idSupplied) { + public Response deleteProvJson(@Context ContainerRequestContext crc, String body, @PathParam("id") String idSupplied) { if(!systemConfig.isProvCollectionEnabled()) { return error(FORBIDDEN, BundleUtil.getStringFromBundle("api.prov.error.provDisabled")); } @@ -88,7 +93,7 @@ public Response deleteProvJson(String body, @PathParam("id") String idSupplied) if(dataFile.isReleased()){ return error(FORBIDDEN, BundleUtil.getStringFromBundle("api.prov.error.jsonDeleteNotAllowed")); } - execCommand(new DeleteProvJsonCommand(createDataverseRequest(findUserOrDie()), dataFile, true)); + execCommand(new DeleteProvJsonCommand(createDataverseRequest(getRequestUser(crc)), dataFile, true)); return ok(BundleUtil.getStringFromBundle("api.prov.provJsonDeleted")); } catch (WrappedResponse ex) { return ex.getResponse(); @@ -97,9 +102,10 @@ public Response deleteProvJson(String body, @PathParam("id") String idSupplied) /** Provenance FreeForm methods **/ @POST + @AuthRequired @Path("{id}/prov-freeform") @Consumes("application/json") - public Response addProvFreeForm(String body, @PathParam("id") String idSupplied) { + public Response addProvFreeForm(@Context ContainerRequestContext crc, String body, @PathParam("id") String idSupplied) { if(!systemConfig.isProvCollectionEnabled()) { return error(FORBIDDEN, BundleUtil.getStringFromBundle("api.prov.error.provDisabled")); } @@ -118,7 +124,7 @@ public Response addProvFreeForm(String body, @PathParam("id") String idSupplied) return error(BAD_REQUEST, BundleUtil.getStringFromBundle("api.prov.error.freeformMissingJsonKey")); } try { - DataverseRequest dr= createDataverseRequest(findUserOrDie()); + DataverseRequest dr= createDataverseRequest(getRequestUser(crc)); DataFile dataFile = findDataFileOrDie(idSupplied); if (dataFile == null) { return error(BAD_REQUEST, BundleUtil.getStringFromBundle("api.prov.error.badDataFileId")); @@ -136,13 +142,14 @@ public Response addProvFreeForm(String body, @PathParam("id") String idSupplied) } @GET + @AuthRequired @Path("{id}/prov-freeform") - public Response getProvFreeForm(String body, @PathParam("id") String idSupplied) { + public Response getProvFreeForm(@Context ContainerRequestContext crc, String body, @PathParam("id") String idSupplied) { if(!systemConfig.isProvCollectionEnabled()) { return error(FORBIDDEN, BundleUtil.getStringFromBundle("api.prov.error.provDisabled")); } try { - String freeFormText = execCommand(new GetProvFreeFormCommand(createDataverseRequest(findUserOrDie()), findDataFileOrDie(idSupplied))); + String freeFormText = execCommand(new GetProvFreeFormCommand(createDataverseRequest(getRequestUser(crc)), findDataFileOrDie(idSupplied))); if(null == freeFormText) { return error(BAD_REQUEST, BundleUtil.getStringFromBundle("api.prov.error.freeformNoText")); } @@ -155,13 +162,14 @@ public Response getProvFreeForm(String body, @PathParam("id") String idSupplied) } @GET + @AuthRequired @Path("{id}/prov-json") - public Response getProvJson(String body, @PathParam("id") String idSupplied) { + public Response getProvJson(@Context ContainerRequestContext crc, String body, @PathParam("id") String idSupplied) { if(!systemConfig.isProvCollectionEnabled()) { return error(FORBIDDEN, BundleUtil.getStringFromBundle("api.prov.error.provDisabled")); } try { - JsonObject jsonText = execCommand(new GetProvJsonCommand(createDataverseRequest(findUserOrDie()), findDataFileOrDie(idSupplied))); + JsonObject jsonText = execCommand(new GetProvJsonCommand(createDataverseRequest(getRequestUser(crc)), findDataFileOrDie(idSupplied))); if(null == jsonText) { return error(BAD_REQUEST, BundleUtil.getStringFromBundle("api.prov.error.jsonNoContent")); } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Roles.java b/src/main/java/edu/harvard/iq/dataverse/api/Roles.java index 76d00114f5e..fcddaf108fe 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Roles.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Roles.java @@ -1,5 +1,6 @@ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.api.dto.RoleDTO; import edu.harvard.iq.dataverse.authorization.DataverseRole; import edu.harvard.iq.dataverse.authorization.Permission; @@ -17,21 +18,18 @@ import jakarta.ejb.Stateless; import jakarta.ws.rs.DELETE; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; - -/** - * Util API for managing roles. Might not make it to the production version. - * @author michael - */ -@Stateless @Path("roles") public class Roles extends AbstractApiBean { @GET + @AuthRequired @Path("{id}") - public Response viewRole( @PathParam("id") String id) { + public Response viewRole(@Context ContainerRequestContext crc, @PathParam("id") String id) { return response( ()-> { - final User user = findUserOrDie(); + final User user = getRequestUser(crc); final DataverseRole role = findRoleOrDie(id); return ( permissionSvc.userOn(user, role.getOwner()).has(Permission.ManageDataversePermissions) ) ? ok( json(role) ) : permissionError("Permission required to view roles."); @@ -39,8 +37,9 @@ public Response viewRole( @PathParam("id") String id) { } @DELETE + @AuthRequired @Path("{id}") - public Response deleteRole(@PathParam("id") String id) { + public Response deleteRole(@Context ContainerRequestContext crc, @PathParam("id") String id) { return response(req -> { DataverseRole role = findRoleOrDie(id); List args = Arrays.asList(role.getName()); @@ -49,15 +48,17 @@ public Response deleteRole(@PathParam("id") String id) { } execCommand(new DeleteRoleCommand(req, role)); return ok("role " + role.getName() + " deleted."); - }); + }, getRequestUser(crc)); } @POST - public Response createNewRole( RoleDTO roleDto, - @QueryParam("dvo") String dvoIdtf ) { + @AuthRequired + public Response createNewRole(@Context ContainerRequestContext crc, + RoleDTO roleDto, + @QueryParam("dvo") String dvoIdtf) { return response( req -> ok(json(execCommand( new CreateRoleCommand(roleDto.asRole(), - req,findDataverseOrDie(dvoIdtf)))))); + req,findDataverseOrDie(dvoIdtf))))), getRequestUser(crc)); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Search.java b/src/main/java/edu/harvard/iq/dataverse/api/Search.java index fc1a4d99f91..c760534ca7b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Search.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Search.java @@ -1,6 +1,7 @@ package edu.harvard.iq.dataverse.api; import edu.harvard.iq.dataverse.Dataverse; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.search.SearchFields; import edu.harvard.iq.dataverse.DataverseServiceBean; import edu.harvard.iq.dataverse.DvObjectServiceBean; @@ -32,6 +33,7 @@ import jakarta.ws.rs.GET; import jakarta.ws.rs.Path; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.Response; import org.apache.commons.lang3.StringUtils; @@ -55,7 +57,9 @@ public class Search extends AbstractApiBean { SolrIndexServiceBean SolrIndexService; @GET + @AuthRequired public Response search( + @Context ContainerRequestContext crc, @QueryParam("q") String query, @QueryParam("type") final List types, @QueryParam("subtree") final List subtrees, @@ -78,7 +82,7 @@ public Response search( User user; try { - user = getUser(); + user = getUser(crc); } catch (WrappedResponse ex) { return ex.getResponse(); } @@ -226,10 +230,10 @@ public Response search( } } - private User getUser() throws WrappedResponse { + private User getUser(ContainerRequestContext crc) throws WrappedResponse { User userToExecuteSearchAs = GuestUser.get(); try { - AuthenticatedUser authenticatedUser = findAuthenticatedUserOrDie(); + AuthenticatedUser authenticatedUser = getRequestAuthenticatedUserOrDie(crc); if (authenticatedUser != null) { userToExecuteSearchAs = authenticatedUser; } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/TestApi.java b/src/main/java/edu/harvard/iq/dataverse/api/TestApi.java index 2b2a120a18a..87be1f14e05 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/TestApi.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/TestApi.java @@ -62,7 +62,9 @@ public Response getExternalToolsForFile(@PathParam("id") String idSupplied, @Que ApiToken apiToken = externalToolService.getApiToken(getRequestApiKey()); ExternalToolHandler externalToolHandler = new ExternalToolHandler(tool, dataFile, apiToken, dataFile.getFileMetadata(), null); JsonObjectBuilder toolToJson = externalToolService.getToolAsJsonWithQueryParameters(externalToolHandler); - tools.add(toolToJson); + if (externalToolService.meetsRequirements(tool, dataFile)) { + tools.add(toolToJson); + } } return ok(tools); } catch (WrappedResponse wr) { diff --git a/src/main/java/edu/harvard/iq/dataverse/api/Users.java b/src/main/java/edu/harvard/iq/dataverse/api/Users.java index db8d94c8552..791fc7aa774 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/Users.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/Users.java @@ -5,6 +5,7 @@ */ package edu.harvard.iq.dataverse.api; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.authorization.users.ApiToken; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; @@ -30,6 +31,7 @@ import jakarta.ws.rs.Path; import jakarta.ws.rs.PathParam; import jakarta.ws.rs.Produces; +import jakarta.ws.rs.container.ContainerRequestContext; import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.MediaType; import jakarta.ws.rs.core.Request; @@ -47,11 +49,12 @@ public class Users extends AbstractApiBean { private static final Logger logger = Logger.getLogger(Users.class.getName()); @POST + @AuthRequired @Path("{consumedIdentifier}/mergeIntoUser/{baseIdentifier}") - public Response mergeInAuthenticatedUser(@PathParam("consumedIdentifier") String consumedIdentifier, @PathParam("baseIdentifier") String baseIdentifier) { + public Response mergeInAuthenticatedUser(@Context ContainerRequestContext crc, @PathParam("consumedIdentifier") String consumedIdentifier, @PathParam("baseIdentifier") String baseIdentifier) { User u; try { - u = findUserOrDie(); + u = getRequestUser(crc); if(!u.isSuperuser()) { throw new WrappedResponse(error(Response.Status.UNAUTHORIZED, "Only superusers can merge users")); } @@ -85,11 +88,12 @@ public Response mergeInAuthenticatedUser(@PathParam("consumedIdentifier") String } @POST + @AuthRequired @Path("{identifier}/changeIdentifier/{newIdentifier}") - public Response changeAuthenticatedUserIdentifier(@PathParam("identifier") String oldIdentifier, @PathParam("newIdentifier") String newIdentifier) { + public Response changeAuthenticatedUserIdentifier(@Context ContainerRequestContext crc, @PathParam("identifier") String oldIdentifier, @PathParam("newIdentifier") String newIdentifier) { User u; try { - u = findUserOrDie(); + u = getRequestUser(crc); if(!u.isSuperuser()) { throw new WrappedResponse(error(Response.Status.UNAUTHORIZED, "Only superusers can change userIdentifiers")); } @@ -118,16 +122,11 @@ public Response changeAuthenticatedUserIdentifier(@PathParam("identifier") Strin } @Path("token") + @AuthRequired @DELETE - public Response deleteToken() { - User u; - - try { - u = findUserOrDie(); - } catch (WrappedResponse ex) { - return ex.getResponse(); - } - AuthenticatedUser au; + public Response deleteToken(@Context ContainerRequestContext crc) { + User u = getRequestUser(crc); + AuthenticatedUser au; try{ au = (AuthenticatedUser) u; @@ -142,16 +141,9 @@ public Response deleteToken() { } @Path("token") + @AuthRequired @GET public Response getTokenExpirationDate() { - User u; - - try { - u = findUserOrDie(); - } catch (WrappedResponse ex) { - return ex.getResponse(); - } - ApiToken token = authSvc.findApiToken(getRequestApiKey()); if (token == null) { @@ -163,16 +155,11 @@ public Response getTokenExpirationDate() { } @Path("token/recreate") + @AuthRequired @POST - public Response recreateToken() { - User u; + public Response recreateToken(@Context ContainerRequestContext crc) { + User u = getRequestUser(crc); - try { - u = findUserOrDie(); - } catch (WrappedResponse ex) { - return ex.getResponse(); - } - AuthenticatedUser au; try{ au = (AuthenticatedUser) u; @@ -192,8 +179,9 @@ public Response recreateToken() { } @GET + @AuthRequired @Path(":me") - public Response getAuthenticatedUserByToken() { + public Response getAuthenticatedUserByToken(@Context ContainerRequestContext crc) { String tokenFromRequestAPI = getRequestApiKey(); @@ -202,7 +190,7 @@ public Response getAuthenticatedUserByToken() { // this is a good idea if (authenticatedUser == null) { try { - authenticatedUser = findAuthenticatedUserOrDie(); + authenticatedUser = getRequestAuthenticatedUserOrDie(crc); } catch (WrappedResponse ex) { Logger.getLogger(Users.class.getName()).log(Level.SEVERE, null, ex); return error(Response.Status.BAD_REQUEST, "User with token " + tokenFromRequestAPI + " not found."); @@ -212,14 +200,15 @@ public Response getAuthenticatedUserByToken() { } @POST + @AuthRequired @Path("{identifier}/removeRoles") - public Response removeUserRoles(@PathParam("identifier") String identifier) { + public Response removeUserRoles(@Context ContainerRequestContext crc, @PathParam("identifier") String identifier) { try { AuthenticatedUser userToModify = authSvc.getAuthenticatedUser(identifier); if (userToModify == null) { return error(Response.Status.BAD_REQUEST, "Cannot find user based on " + identifier + "."); } - execCommand(new RevokeAllRolesCommand(userToModify, createDataverseRequest(findUserOrDie()))); + execCommand(new RevokeAllRolesCommand(userToModify, createDataverseRequest(getRequestUser(crc)))); return ok("Roles removed for user " + identifier + "."); } catch (WrappedResponse wr) { return wr.getResponse(); @@ -227,11 +216,12 @@ public Response removeUserRoles(@PathParam("identifier") String identifier) { } @GET + @AuthRequired @Path("{identifier}/traces") - public Response getTraces(@PathParam("identifier") String identifier) { + public Response getTraces(@Context ContainerRequestContext crc, @PathParam("identifier") String identifier) { try { AuthenticatedUser userToQuery = authSvc.getAuthenticatedUser(identifier); - JsonObjectBuilder jsonObj = execCommand(new GetUserTracesCommand(createDataverseRequest(findUserOrDie()), userToQuery, null)); + JsonObjectBuilder jsonObj = execCommand(new GetUserTracesCommand(createDataverseRequest(getRequestUser(crc)), userToQuery, null)); return ok(jsonObj); } catch (WrappedResponse ex) { return ex.getResponse(); @@ -241,15 +231,16 @@ public Response getTraces(@PathParam("identifier") String identifier) { private List elements = Arrays.asList("roleAssignments","dataverseCreator", "dataversePublisher","datasetCreator", "datasetPublisher","dataFileCreator","dataFilePublisher","datasetVersionUsers","explicitGroups","guestbookEntries", "savedSearches"); @GET + @AuthRequired @Path("{identifier}/traces/{element}") @Produces("text/csv, application/json") - public Response getTraces(@Context Request req, @PathParam("identifier") String identifier, @PathParam("element") String element) { + public Response getTraces(@Context ContainerRequestContext crc, @Context Request req, @PathParam("identifier") String identifier, @PathParam("element") String element) { try { AuthenticatedUser userToQuery = authSvc.getAuthenticatedUser(identifier); if(!elements.contains(element)) { throw new BadRequestException("Not a valid element"); } - JsonObjectBuilder jsonObj = execCommand(new GetUserTracesCommand(createDataverseRequest(findUserOrDie()), userToQuery, element)); + JsonObjectBuilder jsonObj = execCommand(new GetUserTracesCommand(createDataverseRequest(getRequestUser(crc)), userToQuery, element)); List vars = Variant .mediaTypes(MediaType.valueOf(FileUtil.MIME_TYPE_CSV), MediaType.APPLICATION_JSON_TYPE) diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/ApiKeyAuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/ApiKeyAuthMechanism.java new file mode 100644 index 00000000000..0dd8a28baca --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/ApiKeyAuthMechanism.java @@ -0,0 +1,73 @@ +package edu.harvard.iq.dataverse.api.auth; + +import edu.harvard.iq.dataverse.UserServiceBean; +import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.authorization.users.PrivateUrlUser; +import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.privateurl.PrivateUrlServiceBean; + +import jakarta.inject.Inject; +import jakarta.ws.rs.container.ContainerRequestContext; +import java.util.logging.Logger; + +/** + * @author Guillermo Portas + * Authentication mechanism that attempts to authenticate a user from an API Key provided in an API request. + */ +public class ApiKeyAuthMechanism implements AuthMechanism { + + public static final String DATAVERSE_API_KEY_REQUEST_HEADER_NAME = "X-Dataverse-key"; + public static final String DATAVERSE_API_KEY_REQUEST_PARAM_NAME = "key"; + public static final String RESPONSE_MESSAGE_BAD_API_KEY = "Bad API key"; + public static final String ACCESS_DATAFILE_PATH_PREFIX = "/access/datafile/"; + + @Inject + protected PrivateUrlServiceBean privateUrlSvc; + + @Inject + protected AuthenticationServiceBean authSvc; + + @Inject + protected UserServiceBean userSvc; + + private static final Logger logger = Logger.getLogger(ApiKeyAuthMechanism.class.getName()); + + @Override + public User findUserFromRequest(ContainerRequestContext containerRequestContext) throws WrappedAuthErrorResponse { + String apiKey = getRequestApiKey(containerRequestContext); + if (apiKey == null) { + return null; + } + PrivateUrlUser privateUrlUser = privateUrlSvc.getPrivateUrlUserFromToken(apiKey); + if (privateUrlUser != null) { + checkAnonymizedAccessToRequestPath(containerRequestContext.getUriInfo().getPath(), privateUrlUser); + return privateUrlUser; + } + AuthenticatedUser authUser = authSvc.lookupUser(apiKey); + if (authUser != null) { + authUser = userSvc.updateLastApiUseTime(authUser); + return authUser; + } + throw new WrappedAuthErrorResponse(RESPONSE_MESSAGE_BAD_API_KEY); + } + + private String getRequestApiKey(ContainerRequestContext containerRequestContext) { + String headerParamApiKey = containerRequestContext.getHeaderString(DATAVERSE_API_KEY_REQUEST_HEADER_NAME); + String queryParamApiKey = containerRequestContext.getUriInfo().getQueryParameters().getFirst(DATAVERSE_API_KEY_REQUEST_PARAM_NAME); + + return headerParamApiKey != null ? headerParamApiKey : queryParamApiKey; + } + + private void checkAnonymizedAccessToRequestPath(String requestPath, PrivateUrlUser privateUrlUser) throws WrappedAuthErrorResponse { + if (!privateUrlUser.hasAnonymizedAccess()) { + return; + } + // For privateUrlUsers restricted to anonymized access, all api calls are off-limits except for those used in the UI + // to download the file or image thumbs + if (!(requestPath.startsWith(ACCESS_DATAFILE_PATH_PREFIX) && !requestPath.substring(ACCESS_DATAFILE_PATH_PREFIX.length()).contains("/"))) { + logger.info("Anonymized access request for " + requestPath); + throw new WrappedAuthErrorResponse(RESPONSE_MESSAGE_BAD_API_KEY); + } + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthFilter.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthFilter.java new file mode 100644 index 00000000000..34a72d718f0 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthFilter.java @@ -0,0 +1,35 @@ +package edu.harvard.iq.dataverse.api.auth; + +import edu.harvard.iq.dataverse.api.ApiConstants; +import edu.harvard.iq.dataverse.authorization.users.User; + +import jakarta.annotation.Priority; +import jakarta.inject.Inject; +import jakarta.ws.rs.Priorities; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.container.ContainerRequestFilter; +import jakarta.ws.rs.ext.Provider; +import java.io.IOException; + +/** + * @author Guillermo Portas + * Dedicated filter to authenticate the user requesting an API endpoint that requires user authentication. + */ +@AuthRequired +@Provider +@Priority(Priorities.AUTHENTICATION) +public class AuthFilter implements ContainerRequestFilter { + + @Inject + private CompoundAuthMechanism compoundAuthMechanism; + + @Override + public void filter(ContainerRequestContext containerRequestContext) throws IOException { + try { + User user = compoundAuthMechanism.findUserFromRequest(containerRequestContext); + containerRequestContext.setProperty(ApiConstants.CONTAINER_REQUEST_CONTEXT_USER, user); + } catch (WrappedAuthErrorResponse e) { + containerRequestContext.abortWith(e.getResponse()); + } + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthMechanism.java new file mode 100644 index 00000000000..bd34acbf702 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthMechanism.java @@ -0,0 +1,24 @@ +package edu.harvard.iq.dataverse.api.auth; + +import edu.harvard.iq.dataverse.authorization.users.User; + +import jakarta.ws.rs.container.ContainerRequestContext; + +/** + * @author Guillermo Portas + * This interface defines the common behavior for any kind of Dataverse API authentication mechanism. + * Any implementation must correspond to a particular Dataverse API authentication credential type. + */ +interface AuthMechanism { + + /** + * Returns the user associated with a particular authentication credential provided in a request. + * If the credential is not provided, it is expected to return a null user. + * If the credential is provided, but is invalid, it will throw a WrappedAuthErrorResponse exception. + * + * @param containerRequestContext a ContainerRequestContext implementation. + * @return a user that can be null. + * @throws edu.harvard.iq.dataverse.api.auth.WrappedAuthErrorResponse if there is a credential provided, but invalid. + */ + User findUserFromRequest(ContainerRequestContext containerRequestContext) throws WrappedAuthErrorResponse; +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthRequired.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthRequired.java new file mode 100644 index 00000000000..bf0d785eeb3 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/AuthRequired.java @@ -0,0 +1,19 @@ +package edu.harvard.iq.dataverse.api.auth; + +import jakarta.ws.rs.NameBinding; +import java.lang.annotation.ElementType; +import java.lang.annotation.Retention; +import java.lang.annotation.Target; + +import static java.lang.annotation.RetentionPolicy.RUNTIME; + +/** + * @author Guillermo Portas + * Annotation intended to be placed on any API method that requires user authentication. + * Marks the API methods whose related requests should be filtered by {@link edu.harvard.iq.dataverse.api.auth.AuthFilter}. + */ +@NameBinding +@Retention(RUNTIME) +@Target({ElementType.TYPE, ElementType.METHOD}) +public @interface AuthRequired { +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/CompoundAuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/CompoundAuthMechanism.java new file mode 100644 index 00000000000..938e7f2e729 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/CompoundAuthMechanism.java @@ -0,0 +1,50 @@ +package edu.harvard.iq.dataverse.api.auth; + +import edu.harvard.iq.dataverse.authorization.users.GuestUser; +import edu.harvard.iq.dataverse.authorization.users.User; + +import jakarta.inject.Inject; +import jakarta.ws.rs.container.ContainerRequestContext; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.List; + +/** + * @author Guillermo Portas + * Compound authentication mechanism that attempts to authenticate a user through the different authentication mechanisms (ordered by priority) of which it is composed. + * If no user is returned from any of the inner authentication mechanisms, a Guest user is returned. + */ +public class CompoundAuthMechanism implements AuthMechanism { + + private final List authMechanisms = new ArrayList<>(); + + @Inject + public CompoundAuthMechanism(ApiKeyAuthMechanism apiKeyAuthMechanism, WorkflowKeyAuthMechanism workflowKeyAuthMechanism, SignedUrlAuthMechanism signedUrlAuthMechanism) { + // Auth mechanisms should be ordered by priority here + add(apiKeyAuthMechanism, workflowKeyAuthMechanism, signedUrlAuthMechanism); + } + + public CompoundAuthMechanism(AuthMechanism... authMechanisms) { + add(authMechanisms); + } + + public void add(AuthMechanism... authMechanisms) { + this.authMechanisms.addAll(Arrays.asList(authMechanisms)); + } + + @Override + public User findUserFromRequest(ContainerRequestContext containerRequestContext) throws WrappedAuthErrorResponse { + User user = null; + for (AuthMechanism authMechanism : authMechanisms) { + User userFromRequest = authMechanism.findUserFromRequest(containerRequestContext); + if (userFromRequest != null) { + user = userFromRequest; + break; + } + } + if (user == null) { + user = GuestUser.get(); + } + return user; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java new file mode 100644 index 00000000000..f8572144236 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/SignedUrlAuthMechanism.java @@ -0,0 +1,70 @@ +package edu.harvard.iq.dataverse.api.auth; + +import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; +import edu.harvard.iq.dataverse.authorization.users.ApiToken; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.authorization.users.User; +import edu.harvard.iq.dataverse.settings.JvmSettings; +import edu.harvard.iq.dataverse.util.UrlSignerUtil; + +import jakarta.inject.Inject; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.UriInfo; + +import java.net.URLDecoder; +import java.nio.charset.StandardCharsets; + +import static edu.harvard.iq.dataverse.util.UrlSignerUtil.SIGNED_URL_TOKEN; +import static edu.harvard.iq.dataverse.util.UrlSignerUtil.SIGNED_URL_USER; + +/** + * @author Guillermo Portas + * Authentication mechanism that attempts to authenticate a user from a Signed URL provided in an API request. + */ +public class SignedUrlAuthMechanism implements AuthMechanism { + + public static final String RESPONSE_MESSAGE_BAD_SIGNED_URL = "Bad signed URL"; + + @Inject + protected AuthenticationServiceBean authSvc; + + @Override + public User findUserFromRequest(ContainerRequestContext containerRequestContext) throws WrappedAuthErrorResponse { + String signedUrlRequestParameter = getSignedUrlRequestParameter(containerRequestContext); + if (signedUrlRequestParameter == null) { + return null; + } + AuthenticatedUser authUser = getAuthenticatedUserFromSignedUrl(containerRequestContext); + if (authUser != null) { + return authUser; + } + throw new WrappedAuthErrorResponse(RESPONSE_MESSAGE_BAD_SIGNED_URL); + } + + private String getSignedUrlRequestParameter(ContainerRequestContext containerRequestContext) { + return containerRequestContext.getUriInfo().getQueryParameters().getFirst(SIGNED_URL_TOKEN); + } + + private AuthenticatedUser getAuthenticatedUserFromSignedUrl(ContainerRequestContext containerRequestContext) { + AuthenticatedUser authUser = null; + // The signedUrl contains a param telling which user this is supposed to be for. + // We don't trust this. So we lookup that user, and get their API key, and use + // that as a secret in validating the signedURL. If the signature can't be + // validated with their key, the user (or their API key) has been changed and + // we reject the request. + UriInfo uriInfo = containerRequestContext.getUriInfo(); + String userId = uriInfo.getQueryParameters().getFirst(SIGNED_URL_USER); + AuthenticatedUser targetUser = authSvc.getAuthenticatedUser(userId); + ApiToken userApiToken = authSvc.findApiTokenByUser(targetUser); + if (targetUser != null && userApiToken != null) { + String signedUrl = URLDecoder.decode(uriInfo.getRequestUri().toString(), StandardCharsets.UTF_8); + String requestMethod = containerRequestContext.getMethod(); + String signedUrlSigningKey = JvmSettings.API_SIGNING_SECRET.lookupOptional().orElse("") + userApiToken.getTokenString(); + boolean isSignedUrlValid = UrlSignerUtil.isValidUrl(signedUrl, userId, requestMethod, signedUrlSigningKey); + if (isSignedUrlValid) { + authUser = targetUser; + } + } + return authUser; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/WorkflowKeyAuthMechanism.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/WorkflowKeyAuthMechanism.java new file mode 100644 index 00000000000..bbd67713e85 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/WorkflowKeyAuthMechanism.java @@ -0,0 +1,42 @@ +package edu.harvard.iq.dataverse.api.auth; + +import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; +import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; +import edu.harvard.iq.dataverse.authorization.users.User; + +import jakarta.inject.Inject; +import jakarta.ws.rs.container.ContainerRequestContext; + +/** + * @author Guillermo Portas + * Authentication mechanism that attempts to authenticate a user from a Workflow Key provided in an API request. + */ +public class WorkflowKeyAuthMechanism implements AuthMechanism { + + public static final String DATAVERSE_WORKFLOW_KEY_REQUEST_HEADER_NAME = "X-Dataverse-invocationID"; + public static final String DATAVERSE_WORKFLOW_KEY_REQUEST_PARAM_NAME = "invocationID"; + public static final String RESPONSE_MESSAGE_BAD_WORKFLOW_KEY = "Bad workflow invocationID"; + + @Inject + protected AuthenticationServiceBean authSvc; + + @Override + public User findUserFromRequest(ContainerRequestContext containerRequestContext) throws WrappedAuthErrorResponse { + String workflowKey = getRequestWorkflowKey(containerRequestContext); + if (workflowKey == null) { + return null; + } + AuthenticatedUser authUser = authSvc.lookupUserForWorkflowInvocationID(workflowKey); + if (authUser != null) { + return authUser; + } + throw new WrappedAuthErrorResponse(RESPONSE_MESSAGE_BAD_WORKFLOW_KEY); + } + + private String getRequestWorkflowKey(ContainerRequestContext containerRequestContext) { + String headerParamWorkflowKey = containerRequestContext.getHeaderString(DATAVERSE_WORKFLOW_KEY_REQUEST_HEADER_NAME); + String queryParamWorkflowKey = containerRequestContext.getUriInfo().getQueryParameters().getFirst(DATAVERSE_WORKFLOW_KEY_REQUEST_PARAM_NAME); + + return headerParamWorkflowKey != null ? headerParamWorkflowKey : queryParamWorkflowKey; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/auth/WrappedAuthErrorResponse.java b/src/main/java/edu/harvard/iq/dataverse/api/auth/WrappedAuthErrorResponse.java new file mode 100644 index 00000000000..40431557261 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/auth/WrappedAuthErrorResponse.java @@ -0,0 +1,30 @@ +package edu.harvard.iq.dataverse.api.auth; + +import edu.harvard.iq.dataverse.api.ApiConstants; +import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; + +import jakarta.ws.rs.core.MediaType; +import jakarta.ws.rs.core.Response; + +public class WrappedAuthErrorResponse extends Exception { + + private final String message; + private final Response response; + + public WrappedAuthErrorResponse(String message) { + this.message = message; + this.response = Response.status(Response.Status.UNAUTHORIZED) + .entity(NullSafeJsonBuilder.jsonObjectBuilder() + .add("status", ApiConstants.STATUS_ERROR) + .add("message", message).build() + ).type(MediaType.APPLICATION_JSON_TYPE).build(); + } + + public String getMessage() { + return this.message; + } + + public Response getResponse() { + return response; + } +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/batchjob/FileRecordJobResource.java b/src/main/java/edu/harvard/iq/dataverse/api/batchjob/FileRecordJobResource.java index 6d495d9acf2..b7a6b7cfafd 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/batchjob/FileRecordJobResource.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/batchjob/FileRecordJobResource.java @@ -3,6 +3,7 @@ import edu.harvard.iq.dataverse.DatasetServiceBean; import edu.harvard.iq.dataverse.PermissionServiceBean; import edu.harvard.iq.dataverse.api.AbstractApiBean; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.batch.jobs.importer.ImportMode; import edu.harvard.iq.dataverse.engine.command.impl.ImportFromFileSystemCommand; import jakarta.ejb.EJB; @@ -13,6 +14,8 @@ import jakarta.ws.rs.PathParam; import jakarta.ws.rs.Produces; import jakarta.ws.rs.QueryParam; +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import jakarta.ws.rs.core.MediaType; import jakarta.ws.rs.core.Response; import java.util.logging.Logger; @@ -33,13 +36,15 @@ public class FileRecordJobResource extends AbstractApiBean { DatasetServiceBean datasetService; @POST + @AuthRequired @Path("import/datasets/files/{identifier}") @Produces(MediaType.APPLICATION_JSON) - public Response getFilesystemImport(@PathParam("identifier") String identifier, - @QueryParam("mode") @DefaultValue("MERGE") String mode, - /*@QueryParam("fileMode") @DefaultValue("package_file") String fileMode*/ - @QueryParam("uploadFolder") String uploadFolder, - @QueryParam("totalSize") Long totalSize) { + public Response getFilesystemImport(@Context ContainerRequestContext crc, + @PathParam("identifier") String identifier, + @QueryParam("mode") @DefaultValue("MERGE") String mode, + /*@QueryParam("fileMode") @DefaultValue("package_file") String fileMode*/ + @QueryParam("uploadFolder") String uploadFolder, + @QueryParam("totalSize") Long totalSize) { return response(req -> { ImportMode importMode = ImportMode.MERGE; // Switch to this if you ever need to use something other than MERGE. @@ -53,7 +58,7 @@ public Response getFilesystemImport(@PathParam("identifier") String identifier, .add("message", returnString) .add("executionId", jsonObject.getInt("executionId")) ); - }); + }, getRequestUser(crc)); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/CollectionDepositManagerImpl.java b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/CollectionDepositManagerImpl.java index 20163cb76f3..71726cbda00 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/CollectionDepositManagerImpl.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/CollectionDepositManagerImpl.java @@ -165,7 +165,7 @@ public DepositReceipt createNew(String collectionUri, Deposit deposit, AuthCrede } } else if (deposit.isBinaryOnly()) { // get here with this: - // curl --insecure -s --data-binary "@example.zip" -H "Content-Disposition: filename=example.zip" -H "Content-Type: application/zip" https://sword:sword@localhost:8181/dvn/api/data-deposit/v1/swordv2/collection/dataverse/sword/ + // curl --insecure -s --data-binary "@example.zip" -H "Content-Disposition: attachment;filename=example.zip" -H "Content-Type: application/zip" https://sword:sword@localhost:8181/dvn/api/data-deposit/v1/swordv2/collection/dataverse/sword/ throw new SwordError(UriRegistry.ERROR_BAD_REQUEST, "Binary deposit to the collection IRI via POST is not supported. Please POST an Atom entry instead."); } else if (deposit.isMultipart()) { // get here with this: diff --git a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/MediaResourceManagerImpl.java b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/MediaResourceManagerImpl.java index 98cdd11c6e3..1151287321e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/MediaResourceManagerImpl.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/MediaResourceManagerImpl.java @@ -259,7 +259,7 @@ DepositReceipt replaceOrAddFiles(String uri, Deposit deposit, AuthCredentials au * for us, the following *does* work: * * curl--data-binary @path/to/trees.png -H "Content-Disposition: - * filename=trees.png" -H "Content-Type: image/png" -H "Packaging: + * attachment;filename=trees.png" -H "Content-Type: image/png" -H "Packaging: * http://purl.org/net/sword/package/SimpleZip" * * We *might* want to continue to force API users to only upload zip diff --git a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SWORDv2ContainerServlet.java b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SWORDv2ContainerServlet.java index e89428d996e..53dce24c0fe 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SWORDv2ContainerServlet.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SWORDv2ContainerServlet.java @@ -17,9 +17,11 @@ public class SWORDv2ContainerServlet extends SwordServlet { ContainerManagerImpl containerManagerImpl; @Inject StatementManagerImpl statementManagerImpl; - private ContainerManager cm; + // this field can be replaced by local variable +// private ContainerManager cm; private ContainerAPI api; - private StatementManager sm; + // this field can be replaced by local variable +// private StatementManager sm; private final ReentrantLock lock = new ReentrantLock(); @@ -28,13 +30,15 @@ public void init() throws ServletException { super.init(); // load the container manager implementation - this.cm = containerManagerImpl; - - // load the statement manager implementation - this.sm = statementManagerImpl; +// this.cm = containerManagerImpl; + ContainerManager cm = containerManagerImpl; + // load the statement manager implementation +// this.sm = statementManagerImpl; + StatementManager sm = statementManagerImpl; // initialise the underlying servlet processor - this.api = new ContainerAPI(this.cm, this.sm, this.config); +// this.api = new ContainerAPI(this.cm, this.sm, this.config); + this.api = new ContainerAPI(cm, sm, this.config); } @Override diff --git a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordConfigurationImpl.java b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordConfigurationImpl.java index 2229372e3cd..a5564e9fbdb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordConfigurationImpl.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/datadeposit/SwordConfigurationImpl.java @@ -1,5 +1,6 @@ package edu.harvard.iq.dataverse.api.datadeposit; +import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.util.SystemConfig; import java.io.File; import java.util.Arrays; @@ -86,37 +87,32 @@ public boolean storeAndCheckBinary() { @Override public String getTempDirectory() { - String tmpFileDir = System.getProperty(SystemConfig.FILES_DIRECTORY); - if (tmpFileDir != null) { - String swordDirString = tmpFileDir + File.separator + "sword"; - File swordDirFile = new File(swordDirString); - /** - * @todo Do we really need this check? It seems like we do because - * if you create a dataset via the native API and then later try to - * upload a file via SWORD, the directory defined by - * dataverse.files.directory may not exist and we get errors deep in - * the SWORD library code. Could maybe use a try catch in the doPost - * method of our SWORDv2MediaResourceServlet. - */ - if (swordDirFile.exists()) { + // will throw a runtime exception when not found + String tmpFileDir = JvmSettings.FILES_DIRECTORY.lookup(); + + String swordDirString = tmpFileDir + File.separator + "sword"; + File swordDirFile = new File(swordDirString); + /** + * @todo Do we really need this check? It seems like we do because + * if you create a dataset via the native API and then later try to + * upload a file via SWORD, the directory defined by + * dataverse.files.directory may not exist and we get errors deep in + * the SWORD library code. Could maybe use a try catch in the doPost + * method of our SWORDv2MediaResourceServlet. + */ + if (swordDirFile.exists()) { + return swordDirString; + } else { + boolean mkdirSuccess = swordDirFile.mkdirs(); + if (mkdirSuccess) { + logger.info("Created directory " + swordDirString); return swordDirString; } else { - boolean mkdirSuccess = swordDirFile.mkdirs(); - if (mkdirSuccess) { - logger.info("Created directory " + swordDirString); - return swordDirString; - } else { - String msgForSwordUsers = ("Could not determine or create SWORD temp directory. Check logs for details."); - logger.severe(msgForSwordUsers + " Failed to create " + swordDirString); - // sadly, must throw RunTimeException to communicate with SWORD user - throw new RuntimeException(msgForSwordUsers); - } + String msgForSwordUsers = ("Could not determine or create SWORD temp directory. Check logs for details."); + logger.severe(msgForSwordUsers + " Failed to create " + swordDirString); + // sadly, must throw RunTimeException to communicate with SWORD user + throw new RuntimeException(msgForSwordUsers); } - } else { - String msgForSwordUsers = ("JVM option \"" + SystemConfig.FILES_DIRECTORY + "\" not defined. Check logs for details."); - logger.severe(msgForSwordUsers); - // sadly, must throw RunTimeException to communicate with SWORD user - throw new RuntimeException(msgForSwordUsers); } } diff --git a/src/main/java/edu/harvard/iq/dataverse/api/harvestingClients.java b/src/main/java/edu/harvard/iq/dataverse/api/harvestingClients.java new file mode 100644 index 00000000000..66643cca430 --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/api/harvestingClients.java @@ -0,0 +1,5 @@ +package edu.harvard.iq.dataverse.api; + +public class harvestingClients { + +} diff --git a/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java index 38094612e36..81ad5583f22 100644 --- a/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/api/imports/ImportDDIServiceBean.java @@ -1351,7 +1351,9 @@ private void processProdStmt(XMLStreamReader xmlr, MetadataBlockDTO citation) th } else if (xmlr.getLocalName().equals("prodDate")) { citation.getFields().add(FieldDTO.createPrimitiveFieldDTO("productionDate", parseDate(xmlr, "prodDate"))); } else if (xmlr.getLocalName().equals("prodPlac")) { - citation.getFields().add(FieldDTO.createPrimitiveFieldDTO("productionPlace", parseDate(xmlr, "prodPlac"))); + List prodPlac = new ArrayList<>(); + prodPlac.add(parseText(xmlr, "prodPlac")); + citation.getFields().add(FieldDTO.createMultiplePrimitiveFieldDTO(DatasetFieldConstant.productionPlace, prodPlac)); } else if (xmlr.getLocalName().equals("software")) { HashSet set = new HashSet<>(); addToSet(set,"softwareVersion", xmlr.getAttributeValue(null, "version")); diff --git a/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordJobListener.java b/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordJobListener.java index 8925e04a2bd..593a5cbfdc3 100644 --- a/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordJobListener.java +++ b/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordJobListener.java @@ -57,8 +57,10 @@ import jakarta.inject.Named; import jakarta.servlet.http.HttpServletRequest; +import edu.harvard.iq.dataverse.settings.JvmSettings; import org.apache.commons.io.IOUtils; +import java.io.File; import java.io.FileReader; import java.io.IOException; import java.sql.Timestamp; @@ -79,7 +81,7 @@ @Dependent public class FileRecordJobListener implements ItemReadListener, StepListener, JobListener { - public static final String SEP = System.getProperty("file.separator"); + public static final String SEP = File.separator; private static final UserNotification.Type notifyType = UserNotification.Type.FILESYSTEMIMPORT; @@ -433,8 +435,10 @@ private void loadChecksumManifest() { manifest = checksumManifest; getJobLogger().log(Level.INFO, "Checksum manifest = " + manifest + " (FileSystemImportJob.xml property)"); } - // construct full path - String manifestAbsolutePath = System.getProperty("dataverse.files.directory") + + // Construct full path - retrieve base dir via MPCONFIG. + // (Has sane default /tmp/dataverse from META-INF/microprofile-config.properties) + String manifestAbsolutePath = JvmSettings.FILES_DIRECTORY.lookup() + SEP + dataset.getAuthority() + SEP + dataset.getIdentifier() + SEP + uploadFolder diff --git a/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordReader.java b/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordReader.java index 7cf20d856dd..fb702c21df2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordReader.java +++ b/src/main/java/edu/harvard/iq/dataverse/batch/jobs/importer/filesystem/FileRecordReader.java @@ -24,6 +24,7 @@ import edu.harvard.iq.dataverse.authorization.AuthenticationServiceBean; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.batch.jobs.importer.ImportMode; +import edu.harvard.iq.dataverse.settings.JvmSettings; import org.apache.commons.io.filefilter.NotFileFilter; import org.apache.commons.io.filefilter.WildcardFileFilter; @@ -54,7 +55,7 @@ @Dependent public class FileRecordReader extends AbstractItemReader { - public static final String SEP = System.getProperty("file.separator"); + public static final String SEP = File.separator; @Inject JobContext jobContext; @@ -96,9 +97,11 @@ public void init() { @Override public void open(Serializable checkpoint) throws Exception { - - directory = new File(System.getProperty("dataverse.files.directory") - + SEP + dataset.getAuthority() + SEP + dataset.getIdentifier() + SEP + uploadFolder); + + // Retrieve via MPCONFIG. Has sane default /tmp/dataverse from META-INF/microprofile-config.properties + String baseDir = JvmSettings.FILES_DIRECTORY.lookup(); + + directory = new File(baseDir + SEP + dataset.getAuthority() + SEP + dataset.getIdentifier() + SEP + uploadFolder); // TODO: // The above goes directly to the filesystem directory configured by the // old "dataverse.files.directory" JVM option (otherwise used for temp diff --git a/src/main/java/edu/harvard/iq/dataverse/batch/util/LoggingUtil.java b/src/main/java/edu/harvard/iq/dataverse/batch/util/LoggingUtil.java index 3b212259874..19d1112ba54 100644 --- a/src/main/java/edu/harvard/iq/dataverse/batch/util/LoggingUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/batch/util/LoggingUtil.java @@ -154,8 +154,8 @@ public static Logger getJobLogger(String jobId) { try { Logger jobLogger = Logger.getLogger("job-"+jobId); FileHandler fh; - String logDir = System.getProperty("com.sun.aas.instanceRoot") + System.getProperty("file.separator") - + "logs" + System.getProperty("file.separator") + "batch-jobs" + System.getProperty("file.separator"); + String logDir = System.getProperty("com.sun.aas.instanceRoot") + File.separator + + "logs" + File.separator + "batch-jobs" + File.separator; checkCreateLogDirectory( logDir ); fh = new FileHandler(logDir + "job-" + jobId + ".log"); logger.log(Level.INFO, "JOB LOG: " + logDir + "job-" + jobId + ".log"); diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java index d5f00b9868f..8ee3f0cf53c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java @@ -33,9 +33,11 @@ import java.nio.file.Path; import java.nio.file.Paths; import java.util.List; +import java.util.function.Predicate; import java.util.logging.Logger; import java.util.regex.Matcher; import java.util.regex.Pattern; +import java.util.stream.Collectors; // Dataverse imports: import edu.harvard.iq.dataverse.DataFile; @@ -683,4 +685,56 @@ protected static boolean isValidIdentifier(String driverId, String storageId) { } return true; } + + private List listAllFiles() throws IOException { + Dataset dataset = this.getDataset(); + if (dataset == null) { + throw new IOException("This FileAccessIO object hasn't been properly initialized."); + } + + Path datasetDirectoryPath = Paths.get(dataset.getAuthorityForFileStorage(), dataset.getIdentifierForFileStorage()); + if (datasetDirectoryPath == null) { + throw new IOException("Could not determine the filesystem directory of the dataset."); + } + + DirectoryStream dirStream = Files.newDirectoryStream(Paths.get(this.getFilesRootDirectory(), datasetDirectoryPath.toString())); + + List res = new ArrayList<>(); + if (dirStream != null) { + for (Path filePath : dirStream) { + res.add(filePath.getFileName().toString()); + } + dirStream.close(); + } + + return res; + } + + private void deleteFile(String fileName) throws IOException { + Dataset dataset = this.getDataset(); + if (dataset == null) { + throw new IOException("This FileAccessIO object hasn't been properly initialized."); + } + + Path datasetDirectoryPath = Paths.get(dataset.getAuthorityForFileStorage(), dataset.getIdentifierForFileStorage()); + if (datasetDirectoryPath == null) { + throw new IOException("Could not determine the filesystem directory of the dataset."); + } + + Path p = Paths.get(this.getFilesRootDirectory(), datasetDirectoryPath.toString(), fileName); + Files.delete(p); + } + + @Override + public List cleanUp(Predicate filter, boolean dryRun) throws IOException { + List toDelete = this.listAllFiles().stream().filter(filter).collect(Collectors.toList()); + if (dryRun) { + return toDelete; + } + for (String f : toDelete) { + this.deleteFile(f); + } + return toDelete; + } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/InputStreamIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/InputStreamIO.java index c9796d24b27..be6f9df0254 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/InputStreamIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/InputStreamIO.java @@ -14,6 +14,7 @@ import java.nio.channels.WritableByteChannel; import java.nio.file.Path; import java.util.List; +import java.util.function.Predicate; import java.util.logging.Logger; /** @@ -159,5 +160,9 @@ public void revertBackupAsAux(String auxItemTag) throws IOException { throw new UnsupportedDataAccessOperationException("InputStreamIO: this method is not supported in this DataAccess driver."); } - + @Override + public List cleanUp(Predicate filter, boolean dryRun) throws IOException { + throw new UnsupportedDataAccessOperationException("InputStreamIO: tthis method is not supported in this DataAccess driver."); + } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIO.java index c8e42349318..66c6a4cc2ee 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/RemoteOverlayAccessIO.java @@ -24,6 +24,7 @@ import java.security.KeyStoreException; import java.security.NoSuchAlgorithmException; import java.util.List; +import java.util.function.Predicate; import java.util.logging.Logger; import org.apache.http.Header; @@ -630,5 +631,9 @@ protected static boolean isValidIdentifier(String driverId, String storageId) { public static String getBaseStoreIdFor(String driverId) { return System.getProperty("dataverse.files." + driverId + ".base-store"); } - + + @Override + public List cleanUp(Predicate filter, boolean dryRun) throws IOException { + return baseStore.cleanUp(filter, dryRun); + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java index ac3ba38b117..4103372c2cb 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java @@ -60,7 +60,10 @@ import java.util.HashMap; import java.util.List; import java.util.Random; +import java.util.function.Predicate; import java.util.logging.Logger; +import java.util.stream.Collectors; + import org.apache.commons.io.IOUtils; import org.eclipse.microprofile.config.Config; import org.eclipse.microprofile.config.ConfigProvider; @@ -1306,5 +1309,75 @@ protected static boolean isValidIdentifier(String driverId, String storageId) { return true; } + private List listAllFiles() throws IOException { + if (!this.canWrite()) { + open(); + } + Dataset dataset = this.getDataset(); + if (dataset == null) { + throw new IOException("This S3AccessIO object hasn't been properly initialized."); + } + String prefix = dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage() + "/"; + + List ret = new ArrayList<>(); + ListObjectsRequest req = new ListObjectsRequest().withBucketName(bucketName).withPrefix(prefix); + ObjectListing storedFilesList = null; + try { + storedFilesList = s3.listObjects(req); + } catch (SdkClientException sce) { + throw new IOException ("S3 listObjects: failed to get a listing for " + prefix); + } + if (storedFilesList == null) { + return ret; + } + List storedFilesSummary = storedFilesList.getObjectSummaries(); + try { + while (storedFilesList.isTruncated()) { + logger.fine("S3 listObjects: going to next page of list"); + storedFilesList = s3.listNextBatchOfObjects(storedFilesList); + if (storedFilesList != null) { + storedFilesSummary.addAll(storedFilesList.getObjectSummaries()); + } + } + } catch (AmazonClientException ase) { + //logger.warning("Caught an AmazonServiceException in S3AccessIO.listObjects(): " + ase.getMessage()); + throw new IOException("S3AccessIO: Failed to get objects for listing."); + } -} + for (S3ObjectSummary item : storedFilesSummary) { + String fileName = item.getKey().substring(prefix.length()); + ret.add(fileName); + } + return ret; + } + + private void deleteFile(String fileName) throws IOException { + if (!this.canWrite()) { + open(); + } + Dataset dataset = this.getDataset(); + if (dataset == null) { + throw new IOException("This S3AccessIO object hasn't been properly initialized."); + } + String prefix = dataset.getAuthorityForFileStorage() + "/" + dataset.getIdentifierForFileStorage() + "/"; + + try { + DeleteObjectRequest dor = new DeleteObjectRequest(bucketName, prefix + fileName); + s3.deleteObject(dor); + } catch (AmazonClientException ase) { + logger.warning("S3AccessIO: Unable to delete object " + ase.getMessage()); + } + } + + @Override + public List cleanUp(Predicate filter, boolean dryRun) throws IOException { + List toDelete = this.listAllFiles().stream().filter(filter).collect(Collectors.toList()); + if (dryRun) { + return toDelete; + } + for (String f : toDelete) { + this.deleteFile(f); + } + return toDelete; + } +} \ No newline at end of file diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java index 90e4a54dbe8..bfd5c5f0d8f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/StorageIO.java @@ -39,6 +39,7 @@ import java.util.Iterator; import java.util.List; import java.util.Map; +import java.util.function.Predicate; import java.util.regex.Matcher; import java.util.regex.Pattern; @@ -622,4 +623,6 @@ protected static boolean usesStandardNamePattern(String identifier) { return m.find(); } + public abstract List cleanUp(Predicate filter, boolean dryRun) throws IOException; + } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java index b1725b040a3..6c84009de3e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java @@ -22,7 +22,10 @@ import java.util.Formatter; import java.util.List; import java.util.Properties; +import java.util.function.Predicate; import java.util.logging.Logger; +import java.util.stream.Collectors; + import javax.crypto.Mac; import javax.crypto.spec.SecretKeySpec; import org.javaswift.joss.client.factory.AccountFactory; @@ -864,13 +867,16 @@ public InputStream getAuxFileAsInputStream(String auxItemTag) throws IOException } } + private String getSwiftContainerName(Dataset dataset) { + String authorityNoSlashes = dataset.getAuthorityForFileStorage().replace("/", swiftFolderPathSeparator); + return dataset.getProtocolForFileStorage() + swiftFolderPathSeparator + authorityNoSlashes.replace(".", swiftFolderPathSeparator) + + swiftFolderPathSeparator + dataset.getIdentifierForFileStorage(); + } + @Override public String getSwiftContainerName() { if (dvObject instanceof DataFile) { - String authorityNoSlashes = this.getDataFile().getOwner().getAuthorityForFileStorage().replace("/", swiftFolderPathSeparator); - return this.getDataFile().getOwner().getProtocolForFileStorage() + swiftFolderPathSeparator - + authorityNoSlashes.replace(".", swiftFolderPathSeparator) + - swiftFolderPathSeparator + this.getDataFile().getOwner().getIdentifierForFileStorage(); + return getSwiftContainerName(this.getDataFile().getOwner()); } return null; } @@ -893,5 +899,59 @@ public static String calculateRFC2104HMAC(String data, String key) mac.init(signingKey); return toHexString(mac.doFinal(data.getBytes())); } - + + private List listAllFiles() throws IOException { + if (!this.canWrite()) { + open(DataAccessOption.WRITE_ACCESS); + } + Dataset dataset = this.getDataset(); + if (dataset == null) { + throw new IOException("This SwiftAccessIO object hasn't been properly initialized."); + } + String prefix = getSwiftContainerName(dataset) + swiftFolderPathSeparator; + + Collection items; + String lastItemName = null; + List ret = new ArrayList<>(); + + while ((items = this.swiftContainer.list(prefix, lastItemName, LIST_PAGE_LIMIT)) != null && items.size() > 0) { + for (StoredObject item : items) { + lastItemName = item.getName().substring(prefix.length()); + ret.add(lastItemName); + } + } + + return ret; + } + + private void deleteFile(String fileName) throws IOException { + if (!this.canWrite()) { + open(DataAccessOption.WRITE_ACCESS); + } + Dataset dataset = this.getDataset(); + if (dataset == null) { + throw new IOException("This SwiftAccessIO object hasn't been properly initialized."); + } + String prefix = getSwiftContainerName(dataset) + swiftFolderPathSeparator; + + StoredObject fileObject = this.swiftContainer.getObject(prefix + fileName); + + if (!fileObject.exists()) { + throw new FileNotFoundException("SwiftAccessIO/Direct Access: " + fileName + " does not exist"); + } + + fileObject.delete(); + } + + @Override + public List cleanUp(Predicate filter, boolean dryRun) throws IOException { + List toDelete = this.listAllFiles().stream().filter(filter).collect(Collectors.toList()); + if (dryRun) { + return toDelete; + } + for (String f : toDelete) { + this.deleteFile(f); + } + return toDelete; + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/dataaccess/TabularSubsetGenerator.java b/src/main/java/edu/harvard/iq/dataverse/dataaccess/TabularSubsetGenerator.java index 0b6b37af9f0..782f7f3a52d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/dataaccess/TabularSubsetGenerator.java +++ b/src/main/java/edu/harvard/iq/dataverse/dataaccess/TabularSubsetGenerator.java @@ -365,8 +365,8 @@ public void subsetFile(String infile, String outfile, List columns, Lon public void subsetFile(String infile, String outfile, List columns, Long numCases, String delimiter) { - try { - subsetFile(new FileInputStream(new File(infile)), outfile, columns, numCases, delimiter); + try (FileInputStream fis = new FileInputStream(new File(infile))){ + subsetFile(fis, outfile, columns, numCases, delimiter); } catch (IOException ex) { throw new RuntimeException("Could not open file "+infile); } @@ -375,33 +375,28 @@ public void subsetFile(String infile, String outfile, List columns, Lon public void subsetFile(InputStream in, String outfile, List columns, Long numCases, String delimiter) { - try { - Scanner scanner = new Scanner(in); - scanner.useDelimiter("\\n"); - - BufferedWriter out = new BufferedWriter(new FileWriter(outfile)); - for (long caseIndex = 0; caseIndex < numCases; caseIndex++) { - if (scanner.hasNext()) { - String[] line = (scanner.next()).split(delimiter,-1); - List ln = new ArrayList(); - for (Integer i : columns) { - ln.add(line[i]); + try (Scanner scanner = new Scanner(in); BufferedWriter out = new BufferedWriter(new FileWriter(outfile))) { + scanner.useDelimiter("\\n"); + + for (long caseIndex = 0; caseIndex < numCases; caseIndex++) { + if (scanner.hasNext()) { + String[] line = (scanner.next()).split(delimiter,-1); + List ln = new ArrayList(); + for (Integer i : columns) { + ln.add(line[i]); + } + out.write(StringUtils.join(ln,"\t")+"\n"); + } else { + throw new RuntimeException("Tab file has fewer rows than the determined number of cases."); } - out.write(StringUtils.join(ln,"\t")+"\n"); - } else { - throw new RuntimeException("Tab file has fewer rows than the determined number of cases."); } - } - while (scanner.hasNext()) { - if (!"".equals(scanner.next()) ) { - throw new RuntimeException("Tab file has extra nonempty rows than the determined number of cases."); + while (scanner.hasNext()) { + if (!"".equals(scanner.next()) ) { + throw new RuntimeException("Tab file has extra nonempty rows than the determined number of cases."); + } } - } - - scanner.close(); - out.close(); } catch (FileNotFoundException e) { e.printStackTrace(); @@ -418,50 +413,48 @@ public void subsetFile(InputStream in, String outfile, List columns, Lo public static Double[] subsetDoubleVector(InputStream in, int column, int numCases) { Double[] retVector = new Double[numCases]; - Scanner scanner = new Scanner(in); - scanner.useDelimiter("\\n"); - - for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { - if (scanner.hasNext()) { - String[] line = (scanner.next()).split("\t", -1); - - // Verified: new Double("nan") works correctly, - // resulting in Double.NaN; - // Double("[+-]Inf") doesn't work however; - // (the constructor appears to be expecting it - // to be spelled as "Infinity", "-Infinity", etc. - if ("inf".equalsIgnoreCase(line[column]) || "+inf".equalsIgnoreCase(line[column])) { - retVector[caseIndex] = java.lang.Double.POSITIVE_INFINITY; - } else if ("-inf".equalsIgnoreCase(line[column])) { - retVector[caseIndex] = java.lang.Double.NEGATIVE_INFINITY; - } else if (line[column] == null || line[column].equals("")) { - // missing value: - retVector[caseIndex] = null; - } else { - try { - retVector[caseIndex] = new Double(line[column]); - } catch (NumberFormatException ex) { - retVector[caseIndex] = null; // missing value + try (Scanner scanner = new Scanner(in)) { + scanner.useDelimiter("\\n"); + + for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { + if (scanner.hasNext()) { + String[] line = (scanner.next()).split("\t", -1); + + // Verified: new Double("nan") works correctly, + // resulting in Double.NaN; + // Double("[+-]Inf") doesn't work however; + // (the constructor appears to be expecting it + // to be spelled as "Infinity", "-Infinity", etc. + if ("inf".equalsIgnoreCase(line[column]) || "+inf".equalsIgnoreCase(line[column])) { + retVector[caseIndex] = java.lang.Double.POSITIVE_INFINITY; + } else if ("-inf".equalsIgnoreCase(line[column])) { + retVector[caseIndex] = java.lang.Double.NEGATIVE_INFINITY; + } else if (line[column] == null || line[column].equals("")) { + // missing value: + retVector[caseIndex] = null; + } else { + try { + retVector[caseIndex] = new Double(line[column]); + } catch (NumberFormatException ex) { + retVector[caseIndex] = null; // missing value + } } - } - } else { - scanner.close(); - throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); + } else { + throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); + } } - } - int tailIndex = numCases; - while (scanner.hasNext()) { - String nextLine = scanner.next(); - if (!"".equals(nextLine)) { - scanner.close(); - throw new RuntimeException("Column " + column + ": tab file has more nonempty rows than the stored number of cases (" + numCases + ")! current index: " + tailIndex + ", line: " + nextLine); + int tailIndex = numCases; + while (scanner.hasNext()) { + String nextLine = scanner.next(); + if (!"".equals(nextLine)) { + throw new RuntimeException("Column " + column + ": tab file has more nonempty rows than the stored number of cases (" + numCases + ")! current index: " + tailIndex + ", line: " + nextLine); + } + tailIndex++; } - tailIndex++; - } - scanner.close(); + } return retVector; } @@ -472,48 +465,46 @@ public static Double[] subsetDoubleVector(InputStream in, int column, int numCas */ public static Float[] subsetFloatVector(InputStream in, int column, int numCases) { Float[] retVector = new Float[numCases]; - Scanner scanner = new Scanner(in); - scanner.useDelimiter("\\n"); - - for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { - if (scanner.hasNext()) { - String[] line = (scanner.next()).split("\t", -1); - // Verified: new Float("nan") works correctly, - // resulting in Float.NaN; - // Float("[+-]Inf") doesn't work however; - // (the constructor appears to be expecting it - // to be spelled as "Infinity", "-Infinity", etc. - if ("inf".equalsIgnoreCase(line[column]) || "+inf".equalsIgnoreCase(line[column])) { - retVector[caseIndex] = java.lang.Float.POSITIVE_INFINITY; - } else if ("-inf".equalsIgnoreCase(line[column])) { - retVector[caseIndex] = java.lang.Float.NEGATIVE_INFINITY; - } else if (line[column] == null || line[column].equals("")) { - // missing value: - retVector[caseIndex] = null; - } else { - try { - retVector[caseIndex] = new Float(line[column]); - } catch (NumberFormatException ex) { - retVector[caseIndex] = null; // missing value + try (Scanner scanner = new Scanner(in)) { + scanner.useDelimiter("\\n"); + + for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { + if (scanner.hasNext()) { + String[] line = (scanner.next()).split("\t", -1); + // Verified: new Float("nan") works correctly, + // resulting in Float.NaN; + // Float("[+-]Inf") doesn't work however; + // (the constructor appears to be expecting it + // to be spelled as "Infinity", "-Infinity", etc. + if ("inf".equalsIgnoreCase(line[column]) || "+inf".equalsIgnoreCase(line[column])) { + retVector[caseIndex] = java.lang.Float.POSITIVE_INFINITY; + } else if ("-inf".equalsIgnoreCase(line[column])) { + retVector[caseIndex] = java.lang.Float.NEGATIVE_INFINITY; + } else if (line[column] == null || line[column].equals("")) { + // missing value: + retVector[caseIndex] = null; + } else { + try { + retVector[caseIndex] = new Float(line[column]); + } catch (NumberFormatException ex) { + retVector[caseIndex] = null; // missing value + } } + } else { + throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); } - } else { - scanner.close(); - throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); } - } - int tailIndex = numCases; - while (scanner.hasNext()) { - String nextLine = scanner.next(); - if (!"".equals(nextLine)) { - scanner.close(); - throw new RuntimeException("Column "+column+": tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + int tailIndex = numCases; + while (scanner.hasNext()) { + String nextLine = scanner.next(); + if (!"".equals(nextLine)) { + throw new RuntimeException("Column "+column+": tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + } + tailIndex++; } - tailIndex++; - } - scanner.close(); + } return retVector; } @@ -524,34 +515,32 @@ public static Float[] subsetFloatVector(InputStream in, int column, int numCases */ public static Long[] subsetLongVector(InputStream in, int column, int numCases) { Long[] retVector = new Long[numCases]; - Scanner scanner = new Scanner(in); - scanner.useDelimiter("\\n"); + try (Scanner scanner = new Scanner(in)) { + scanner.useDelimiter("\\n"); - for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { - if (scanner.hasNext()) { - String[] line = (scanner.next()).split("\t", -1); - try { - retVector[caseIndex] = new Long(line[column]); - } catch (NumberFormatException ex) { - retVector[caseIndex] = null; // assume missing value + for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { + if (scanner.hasNext()) { + String[] line = (scanner.next()).split("\t", -1); + try { + retVector[caseIndex] = new Long(line[column]); + } catch (NumberFormatException ex) { + retVector[caseIndex] = null; // assume missing value + } + } else { + throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); } - } else { - scanner.close(); - throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); } - } - int tailIndex = numCases; - while (scanner.hasNext()) { - String nextLine = scanner.next(); - if (!"".equals(nextLine)) { - scanner.close(); - throw new RuntimeException("Column "+column+": tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + int tailIndex = numCases; + while (scanner.hasNext()) { + String nextLine = scanner.next(); + if (!"".equals(nextLine)) { + throw new RuntimeException("Column "+column+": tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + } + tailIndex++; } - tailIndex++; - } - scanner.close(); + } return retVector; } @@ -562,75 +551,72 @@ public static Long[] subsetLongVector(InputStream in, int column, int numCases) */ public static String[] subsetStringVector(InputStream in, int column, int numCases) { String[] retVector = new String[numCases]; - Scanner scanner = new Scanner(in); - scanner.useDelimiter("\\n"); + try (Scanner scanner = new Scanner(in)) { + scanner.useDelimiter("\\n"); - for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { - if (scanner.hasNext()) { - String[] line = (scanner.next()).split("\t", -1); - retVector[caseIndex] = line[column]; - - - if ("".equals(line[column])) { - // An empty string is a string missing value! - // An empty string in quotes is an empty string! - retVector[caseIndex] = null; - } else { - // Strip the outer quotes: - line[column] = line[column].replaceFirst("^\\\"", ""); - line[column] = line[column].replaceFirst("\\\"$", ""); - - // We need to restore the special characters that - // are stored in tab files escaped - quotes, new lines - // and tabs. Before we do that however, we need to - // take care of any escaped backslashes stored in - // the tab file. I.e., "foo\t" should be transformed - // to "foo"; but "foo\\t" should be transformed - // to "foo\t". This way new lines and tabs that were - // already escaped in the original data are not - // going to be transformed to unescaped tab and - // new line characters! - String[] splitTokens = line[column].split(Matcher.quoteReplacement("\\\\"), -2); - - // (note that it's important to use the 2-argument version - // of String.split(), and set the limit argument to a - // negative value; otherwise any trailing backslashes - // are lost.) - for (int i = 0; i < splitTokens.length; i++) { - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\\""), "\""); - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\t"), "\t"); - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\n"), "\n"); - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\r"), "\r"); - } - // TODO: - // Make (some of?) the above optional; for ex., we - // do need to restore the newlines when calculating UNFs; - // But if we are subsetting these vectors in order to - // create a new tab-delimited file, they will - // actually break things! -- L.A. Jul. 28 2014 + for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { + if (scanner.hasNext()) { + String[] line = (scanner.next()).split("\t", -1); + retVector[caseIndex] = line[column]; - line[column] = StringUtils.join(splitTokens, '\\'); + if ("".equals(line[column])) { + // An empty string is a string missing value! + // An empty string in quotes is an empty string! + retVector[caseIndex] = null; + } else { + // Strip the outer quotes: + line[column] = line[column].replaceFirst("^\\\"", ""); + line[column] = line[column].replaceFirst("\\\"$", ""); + + // We need to restore the special characters that + // are stored in tab files escaped - quotes, new lines + // and tabs. Before we do that however, we need to + // take care of any escaped backslashes stored in + // the tab file. I.e., "foo\t" should be transformed + // to "foo"; but "foo\\t" should be transformed + // to "foo\t". This way new lines and tabs that were + // already escaped in the original data are not + // going to be transformed to unescaped tab and + // new line characters! + String[] splitTokens = line[column].split(Matcher.quoteReplacement("\\\\"), -2); + + // (note that it's important to use the 2-argument version + // of String.split(), and set the limit argument to a + // negative value; otherwise any trailing backslashes + // are lost.) + for (int i = 0; i < splitTokens.length; i++) { + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\\""), "\""); + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\t"), "\t"); + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\n"), "\n"); + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\r"), "\r"); + } + // TODO: + // Make (some of?) the above optional; for ex., we + // do need to restore the newlines when calculating UNFs; + // But if we are subsetting these vectors in order to + // create a new tab-delimited file, they will + // actually break things! -- L.A. Jul. 28 2014 - retVector[caseIndex] = line[column]; - } + line[column] = StringUtils.join(splitTokens, '\\'); - } else { - scanner.close(); - throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); + retVector[caseIndex] = line[column]; + } + + } else { + throw new RuntimeException("Tab file has fewer rows than the stored number of cases!"); + } } - } - int tailIndex = numCases; - while (scanner.hasNext()) { - String nextLine = scanner.next(); - if (!"".equals(nextLine)) { - scanner.close(); - throw new RuntimeException("Column "+column+": tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + int tailIndex = numCases; + while (scanner.hasNext()) { + String nextLine = scanner.next(); + if (!"".equals(nextLine)) { + throw new RuntimeException("Column "+column+": tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + } + tailIndex++; } - tailIndex++; - } - scanner.close(); + } return retVector; } @@ -643,42 +629,40 @@ public static String[] subsetStringVector(InputStream in, int column, int numCas */ public static Double[][] subsetDoubleVectors(InputStream in, Set columns, int numCases) throws IOException { Double[][] retVector = new Double[columns.size()][numCases]; - Scanner scanner = new Scanner(in); - scanner.useDelimiter("\\n"); - - for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { - if (scanner.hasNext()) { - String[] line = (scanner.next()).split("\t", -1); - int j = 0; - for (Integer i : columns) { - try { - // TODO: verify that NaN and +-Inf are going to be - // handled correctly here! -- L.A. - // NO, "+-Inf" is not handled correctly; see the - // comment further down below. - retVector[j][caseIndex] = new Double(line[i]); - } catch (NumberFormatException ex) { - retVector[j][caseIndex] = null; // missing value + try (Scanner scanner = new Scanner(in)) { + scanner.useDelimiter("\\n"); + + for (int caseIndex = 0; caseIndex < numCases; caseIndex++) { + if (scanner.hasNext()) { + String[] line = (scanner.next()).split("\t", -1); + int j = 0; + for (Integer i : columns) { + try { + // TODO: verify that NaN and +-Inf are going to be + // handled correctly here! -- L.A. + // NO, "+-Inf" is not handled correctly; see the + // comment further down below. + retVector[j][caseIndex] = new Double(line[i]); + } catch (NumberFormatException ex) { + retVector[j][caseIndex] = null; // missing value + } + j++; } - j++; + } else { + throw new IOException("Tab file has fewer rows than the stored number of cases!"); } - } else { - scanner.close(); - throw new IOException("Tab file has fewer rows than the stored number of cases!"); } - } - int tailIndex = numCases; - while (scanner.hasNext()) { - String nextLine = scanner.next(); - if (!"".equals(nextLine)) { - scanner.close(); - throw new IOException("Tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + int tailIndex = numCases; + while (scanner.hasNext()) { + String nextLine = scanner.next(); + if (!"".equals(nextLine)) { + throw new IOException("Tab file has more nonempty rows than the stored number of cases ("+numCases+")! current index: "+tailIndex+", line: "+nextLine); + } + tailIndex++; } - tailIndex++; - } - scanner.close(); + } return retVector; } @@ -839,237 +823,238 @@ public Object[] subsetObjectVector(File tabfile, int column, int varcount, int c columnOffset = varcount * 8; columnLength = columnEndOffsets[0] - varcount * 8; } + int caseindex = 0; - FileChannel fc = (FileChannel.open(Paths.get(rotatedImageFile.getAbsolutePath()), StandardOpenOption.READ)); - fc.position(columnOffset); - int MAX_COLUMN_BUFFER = 8192; - - ByteBuffer in = ByteBuffer.allocate(MAX_COLUMN_BUFFER); - - if (columnLength < MAX_COLUMN_BUFFER) { - in.limit((int)(columnLength)); - } - - long bytesRead = 0; - long bytesReadTotal = 0; - int caseindex = 0; - int byteoffset = 0; - byte[] leftover = null; - - while (bytesReadTotal < columnLength) { - bytesRead = fc.read(in); - byte[] columnBytes = in.array(); - int bytecount = 0; + try (FileChannel fc = (FileChannel.open(Paths.get(rotatedImageFile.getAbsolutePath()), + StandardOpenOption.READ))) { + fc.position(columnOffset); + int MAX_COLUMN_BUFFER = 8192; - - while (bytecount < bytesRead) { - if (columnBytes[bytecount] == '\n') { - /* - String token = new String(columnBytes, byteoffset, bytecount-byteoffset, "UTF8"); - - if (leftover != null) { - String leftoverString = new String (leftover, "UTF8"); - token = leftoverString + token; - leftover = null; - } - */ - /* - * Note that the way I was doing it at first - above - - * was not quite the correct way - because I was creating UTF8 - * strings from the leftover bytes, and the bytes in the - * current buffer *separately*; which means, if a multi-byte - * UTF8 character got split in the middle between one buffer - * and the next, both chunks of it would become junk - * characters, on each side! - * The correct way of doing it, of course, is to create a - * merged byte buffer, and then turn it into a UTF8 string. - * -- L.A. 4.0 - */ - String token = null; - - if (leftover == null) { - token = new String(columnBytes, byteoffset, bytecount-byteoffset, "UTF8"); - } else { - byte[] merged = new byte[leftover.length + bytecount-byteoffset]; - - System.arraycopy(leftover, 0, merged, 0, leftover.length); - System.arraycopy(columnBytes, byteoffset, merged, leftover.length, bytecount-byteoffset); - token = new String (merged, "UTF8"); - leftover = null; - merged = null; - } - - if (isString) { - if ("".equals(token)) { - // An empty string is a string missing value! - // An empty string in quotes is an empty string! - retVector[caseindex] = null; + ByteBuffer in = ByteBuffer.allocate(MAX_COLUMN_BUFFER); + + if (columnLength < MAX_COLUMN_BUFFER) { + in.limit((int) (columnLength)); + } + + long bytesRead = 0; + long bytesReadTotal = 0; + + int byteoffset = 0; + byte[] leftover = null; + + while (bytesReadTotal < columnLength) { + bytesRead = fc.read(in); + byte[] columnBytes = in.array(); + int bytecount = 0; + + while (bytecount < bytesRead) { + if (columnBytes[bytecount] == '\n') { + /* + String token = new String(columnBytes, byteoffset, bytecount-byteoffset, "UTF8"); + + if (leftover != null) { + String leftoverString = new String (leftover, "UTF8"); + token = leftoverString + token; + leftover = null; + } + */ + /* + * Note that the way I was doing it at first - above - + * was not quite the correct way - because I was creating UTF8 + * strings from the leftover bytes, and the bytes in the + * current buffer *separately*; which means, if a multi-byte + * UTF8 character got split in the middle between one buffer + * and the next, both chunks of it would become junk + * characters, on each side! + * The correct way of doing it, of course, is to create a + * merged byte buffer, and then turn it into a UTF8 string. + * -- L.A. 4.0 + */ + String token = null; + + if (leftover == null) { + token = new String(columnBytes, byteoffset, bytecount - byteoffset, "UTF8"); } else { - // Strip the outer quotes: - token = token.replaceFirst("^\\\"", ""); - token = token.replaceFirst("\\\"$", ""); - - // We need to restore the special characters that - // are stored in tab files escaped - quotes, new lines - // and tabs. Before we do that however, we need to - // take care of any escaped backslashes stored in - // the tab file. I.e., "foo\t" should be transformed - // to "foo"; but "foo\\t" should be transformed - // to "foo\t". This way new lines and tabs that were - // already escaped in the original data are not - // going to be transformed to unescaped tab and - // new line characters! - - String[] splitTokens = token.split(Matcher.quoteReplacement("\\\\"), -2); - - // (note that it's important to use the 2-argument version - // of String.split(), and set the limit argument to a - // negative value; otherwise any trailing backslashes - // are lost.) - - for (int i = 0; i < splitTokens.length; i++) { - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\\""), "\""); - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\t"), "\t"); - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\n"), "\n"); - splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\r"), "\r"); - } - // TODO: - // Make (some of?) the above optional; for ex., we - // do need to restore the newlines when calculating UNFs; - // But if we are subsetting these vectors in order to - // create a new tab-delimited file, they will - // actually break things! -- L.A. Jul. 28 2014 - - token = StringUtils.join(splitTokens, '\\'); - - // "compatibility mode" - a hack, to be able to produce - // unfs identical to those produced by the "early" - // unf5 jar; will be removed in production 4.0. - // -- L.A. (TODO: ...) - if (compatmode && !"".equals(token)) { - if (token.length() > 128) { - if ("".equals(token.trim())) { - // don't ask... - token = token.substring(0, 129); + byte[] merged = new byte[leftover.length + bytecount - byteoffset]; + + System.arraycopy(leftover, 0, merged, 0, leftover.length); + System.arraycopy(columnBytes, byteoffset, merged, leftover.length, bytecount - byteoffset); + token = new String(merged, "UTF8"); + leftover = null; + merged = null; + } + + if (isString) { + if ("".equals(token)) { + // An empty string is a string missing value! + // An empty string in quotes is an empty string! + retVector[caseindex] = null; + } else { + // Strip the outer quotes: + token = token.replaceFirst("^\\\"", ""); + token = token.replaceFirst("\\\"$", ""); + + // We need to restore the special characters that + // are stored in tab files escaped - quotes, new lines + // and tabs. Before we do that however, we need to + // take care of any escaped backslashes stored in + // the tab file. I.e., "foo\t" should be transformed + // to "foo"; but "foo\\t" should be transformed + // to "foo\t". This way new lines and tabs that were + // already escaped in the original data are not + // going to be transformed to unescaped tab and + // new line characters! + + String[] splitTokens = token.split(Matcher.quoteReplacement("\\\\"), -2); + + // (note that it's important to use the 2-argument version + // of String.split(), and set the limit argument to a + // negative value; otherwise any trailing backslashes + // are lost.) + + for (int i = 0; i < splitTokens.length; i++) { + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\\""), "\""); + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\t"), "\t"); + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\n"), "\n"); + splitTokens[i] = splitTokens[i].replaceAll(Matcher.quoteReplacement("\\r"), "\r"); + } + // TODO: + // Make (some of?) the above optional; for ex., we + // do need to restore the newlines when calculating UNFs; + // But if we are subsetting these vectors in order to + // create a new tab-delimited file, they will + // actually break things! -- L.A. Jul. 28 2014 + + token = StringUtils.join(splitTokens, '\\'); + + // "compatibility mode" - a hack, to be able to produce + // unfs identical to those produced by the "early" + // unf5 jar; will be removed in production 4.0. + // -- L.A. (TODO: ...) + if (compatmode && !"".equals(token)) { + if (token.length() > 128) { + if ("".equals(token.trim())) { + // don't ask... + token = token.substring(0, 129); + } else { + token = token.substring(0, 128); + // token = String.format(loc, "%.128s", token); + token = token.trim(); + // dbgLog.info("formatted and trimmed: "+token); + } } else { - token = token.substring(0, 128); - //token = String.format(loc, "%.128s", token); - token = token.trim(); - //dbgLog.info("formatted and trimmed: "+token); + if ("".equals(token.trim())) { + // again, don't ask; + // - this replicates some bugginness + // that happens inside unf5; + token = "null"; + } else { + token = token.trim(); + } } + } + + retVector[caseindex] = token; + } + } else if (isDouble) { + try { + // TODO: verify that NaN and +-Inf are + // handled correctly here! -- L.A. + // Verified: new Double("nan") works correctly, + // resulting in Double.NaN; + // Double("[+-]Inf") doesn't work however; + // (the constructor appears to be expecting it + // to be spelled as "Infinity", "-Infinity", etc. + if ("inf".equalsIgnoreCase(token) || "+inf".equalsIgnoreCase(token)) { + retVector[caseindex] = java.lang.Double.POSITIVE_INFINITY; + } else if ("-inf".equalsIgnoreCase(token)) { + retVector[caseindex] = java.lang.Double.NEGATIVE_INFINITY; + } else if (token == null || token.equals("")) { + // missing value: + retVector[caseindex] = null; } else { - if ("".equals(token.trim())) { - // again, don't ask; - // - this replicates some bugginness - // that happens inside unf5; - token = "null"; - } else { - token = token.trim(); - } + retVector[caseindex] = new Double(token); } + } catch (NumberFormatException ex) { + dbgLog.warning("NumberFormatException thrown for " + token + " as Double"); + + retVector[caseindex] = null; // missing value + // TODO: ? } - - retVector[caseindex] = token; - } - } else if (isDouble) { - try { - // TODO: verify that NaN and +-Inf are - // handled correctly here! -- L.A. - // Verified: new Double("nan") works correctly, - // resulting in Double.NaN; - // Double("[+-]Inf") doesn't work however; - // (the constructor appears to be expecting it - // to be spelled as "Infinity", "-Infinity", etc. - if ("inf".equalsIgnoreCase(token) || "+inf".equalsIgnoreCase(token)) { - retVector[caseindex] = java.lang.Double.POSITIVE_INFINITY; - } else if ("-inf".equalsIgnoreCase(token)) { - retVector[caseindex] = java.lang.Double.NEGATIVE_INFINITY; - } else if (token == null || token.equals("")) { - // missing value: - retVector[caseindex] = null; - } else { - retVector[caseindex] = new Double(token); + } else if (isLong) { + try { + retVector[caseindex] = new Long(token); + } catch (NumberFormatException ex) { + retVector[caseindex] = null; // assume missing value } - } catch (NumberFormatException ex) { - dbgLog.warning("NumberFormatException thrown for "+token+" as Double"); - - retVector[caseindex] = null; // missing value - // TODO: ? - } - } else if (isLong) { - try { - retVector[caseindex] = new Long(token); - } catch (NumberFormatException ex) { - retVector[caseindex] = null; // assume missing value - } - } else if (isFloat) { - try { - if ("inf".equalsIgnoreCase(token) || "+inf".equalsIgnoreCase(token)) { - retVector[caseindex] = java.lang.Float.POSITIVE_INFINITY; - } else if ("-inf".equalsIgnoreCase(token)) { - retVector[caseindex] = java.lang.Float.NEGATIVE_INFINITY; - } else if (token == null || token.equals("")) { - // missing value: - retVector[caseindex] = null; - } else { - retVector[caseindex] = new Float(token); + } else if (isFloat) { + try { + if ("inf".equalsIgnoreCase(token) || "+inf".equalsIgnoreCase(token)) { + retVector[caseindex] = java.lang.Float.POSITIVE_INFINITY; + } else if ("-inf".equalsIgnoreCase(token)) { + retVector[caseindex] = java.lang.Float.NEGATIVE_INFINITY; + } else if (token == null || token.equals("")) { + // missing value: + retVector[caseindex] = null; + } else { + retVector[caseindex] = new Float(token); + } + } catch (NumberFormatException ex) { + dbgLog.warning("NumberFormatException thrown for " + token + " as Float"); + retVector[caseindex] = null; // assume missing value (TODO: ?) } - } catch (NumberFormatException ex) { - dbgLog.warning("NumberFormatException thrown for "+token+" as Float"); - retVector[caseindex] = null; // assume missing value (TODO: ?) } - } - caseindex++; - - if (bytecount == bytesRead - 1) { - byteoffset = 0; - } else { - byteoffset = bytecount + 1; - } - } else { - if (bytecount == bytesRead - 1) { - // We've reached the end of the buffer; - // This means we'll save whatever unused bytes left in - // it - i.e., the bytes between the last new line - // encountered and the end - in the leftover buffer. - - // *EXCEPT*, there may be a case of a very long String - // that is actually longer than MAX_COLUMN_BUFFER, in - // which case it is possible that we've read through - // an entire buffer of bytes without finding any - // new lines... in this case we may need to add this - // entire byte buffer to an already existing leftover - // buffer! - if (leftover == null) { - leftover = new byte[(int)bytesRead - byteoffset]; - System.arraycopy(columnBytes, byteoffset, leftover, 0, (int)bytesRead - byteoffset); + caseindex++; + + if (bytecount == bytesRead - 1) { + byteoffset = 0; } else { - if (byteoffset != 0) { + byteoffset = bytecount + 1; + } + } else { + if (bytecount == bytesRead - 1) { + // We've reached the end of the buffer; + // This means we'll save whatever unused bytes left in + // it - i.e., the bytes between the last new line + // encountered and the end - in the leftover buffer. + + // *EXCEPT*, there may be a case of a very long String + // that is actually longer than MAX_COLUMN_BUFFER, in + // which case it is possible that we've read through + // an entire buffer of bytes without finding any + // new lines... in this case we may need to add this + // entire byte buffer to an already existing leftover + // buffer! + if (leftover == null) { + leftover = new byte[(int) bytesRead - byteoffset]; + System.arraycopy(columnBytes, byteoffset, leftover, 0, (int) bytesRead - byteoffset); + } else { + if (byteoffset != 0) { throw new IOException("Reached the end of the byte buffer, with some leftover left from the last read; yet the offset is not zero!"); + } + byte[] merged = new byte[leftover.length + (int) bytesRead]; + + System.arraycopy(leftover, 0, merged, 0, leftover.length); + System.arraycopy(columnBytes, byteoffset, merged, leftover.length, (int) bytesRead); + // leftover = null; + leftover = merged; + merged = null; } - byte[] merged = new byte[leftover.length + (int)bytesRead]; + byteoffset = 0; - System.arraycopy(leftover, 0, merged, 0, leftover.length); - System.arraycopy(columnBytes, byteoffset, merged, leftover.length, (int)bytesRead); - //leftover = null; - leftover = merged; - merged = null; } - byteoffset = 0; - } + bytecount++; + } + + bytesReadTotal += bytesRead; + in.clear(); + if (columnLength - bytesReadTotal < MAX_COLUMN_BUFFER) { + in.limit((int) (columnLength - bytesReadTotal)); } - bytecount++; - } - - bytesReadTotal += bytesRead; - in.clear(); - if (columnLength - bytesReadTotal < MAX_COLUMN_BUFFER) { - in.limit((int)(columnLength - bytesReadTotal)); } + } - - fc.close(); if (caseindex != casecount) { throw new IOException("Faile to read "+casecount+" tokens for column "+column); @@ -1080,31 +1065,31 @@ public Object[] subsetObjectVector(File tabfile, int column, int varcount, int c } private long[] extractColumnOffsets (File rotatedImageFile, int varcount, int casecount) throws IOException { - BufferedInputStream rotfileStream = new BufferedInputStream(new FileInputStream(rotatedImageFile)); - - byte[] offsetHeader = new byte[varcount * 8]; long[] byteOffsets = new long[varcount]; - - int readlen = rotfileStream.read(offsetHeader); - - if (readlen != varcount * 8) { - throw new IOException ("Could not read "+varcount*8+" header bytes from the rotated file."); - } - - for (int varindex = 0; varindex < varcount; varindex++) { - byte[] offsetBytes = new byte[8]; - System.arraycopy(offsetHeader, varindex*8, offsetBytes, 0, 8); - - ByteBuffer offsetByteBuffer = ByteBuffer.wrap(offsetBytes); - byteOffsets[varindex] = offsetByteBuffer.getLong(); - - //System.out.println(byteOffsets[varindex]); + try (BufferedInputStream rotfileStream = new BufferedInputStream(new FileInputStream(rotatedImageFile))) { + + byte[] offsetHeader = new byte[varcount * 8]; + + int readlen = rotfileStream.read(offsetHeader); + + if (readlen != varcount * 8) { + throw new IOException("Could not read " + varcount * 8 + " header bytes from the rotated file."); + } + + for (int varindex = 0; varindex < varcount; varindex++) { + byte[] offsetBytes = new byte[8]; + System.arraycopy(offsetHeader, varindex * 8, offsetBytes, 0, 8); + + ByteBuffer offsetByteBuffer = ByteBuffer.wrap(offsetBytes); + byteOffsets[varindex] = offsetByteBuffer.getLong(); + + // System.out.println(byteOffsets[varindex]); + } + } - - rotfileStream.close(); - - return byteOffsets; + + return byteOffsets; } private File getRotatedImage(File tabfile, int varcount, int casecount) throws IOException { @@ -1149,85 +1134,84 @@ private File generateRotatedImage (File tabfile, int varcount, int casecount) th // read the tab-delimited file: - FileInputStream tabfileStream = new FileInputStream(tabfile); - - Scanner scanner = new Scanner(tabfileStream); - scanner.useDelimiter("\\n"); - - for (int caseindex = 0; caseindex < casecount; caseindex++) { - if (scanner.hasNext()) { - String[] line = (scanner.next()).split("\t", -1); - // TODO: throw an exception if there are fewer tab-delimited - // tokens than the number of variables specified. - String token = ""; - int tokensize = 0; - for (int varindex = 0; varindex < varcount; varindex++) { - // TODO: figure out the safest way to convert strings to - // bytes here. Is it going to be safer to use getBytes("UTF8")? - // we are already making the assumption that the values - // in the tab file are in UTF8. -- L.A. - token = line[varindex] + "\n"; - tokensize = token.getBytes().length; - if (bufferedSizes[varindex]+tokensize > MAX_COLUMN_BUFFER) { - // fill the buffer and dump its contents into the temp file: - // (do note that there may be *several* MAX_COLUMN_BUFFERs - // worth of bytes in the token!) - - int tokenoffset = 0; - - if (bufferedSizes[varindex] != MAX_COLUMN_BUFFER) { - tokenoffset = MAX_COLUMN_BUFFER-bufferedSizes[varindex]; - System.arraycopy(token.getBytes(), 0, bufferedColumns[varindex], bufferedSizes[varindex], tokenoffset); - } // (otherwise the buffer is already full, and we should - // simply dump it into the temp file, without adding any - // extra bytes to it) - - File bufferTempFile = columnTempFiles[varindex]; - if (bufferTempFile == null) { - bufferTempFile = File.createTempFile("columnBufferFile", "bytes"); - columnTempFiles[varindex] = bufferTempFile; - } - - // *append* the contents of the buffer to the end of the - // temp file, if already exists: - BufferedOutputStream outputStream = new BufferedOutputStream(new FileOutputStream (bufferTempFile, true)); - outputStream.write(bufferedColumns[varindex], 0, MAX_COLUMN_BUFFER); - cachedfileSizes[varindex] += MAX_COLUMN_BUFFER; - - // keep writing MAX_COLUMN_BUFFER-size chunks of bytes into - // the temp file, for as long as there's more than MAX_COLUMN_BUFFER - // bytes left in the token: - - while (tokensize - tokenoffset > MAX_COLUMN_BUFFER) { - outputStream.write(token.getBytes(), tokenoffset, MAX_COLUMN_BUFFER); - cachedfileSizes[varindex] += MAX_COLUMN_BUFFER; - tokenoffset += MAX_COLUMN_BUFFER; + try (FileInputStream tabfileStream = new FileInputStream(tabfile); + Scanner scanner = new Scanner(tabfileStream)) { + scanner.useDelimiter("\\n"); + + for (int caseindex = 0; caseindex < casecount; caseindex++) { + if (scanner.hasNext()) { + String[] line = (scanner.next()).split("\t", -1); + // TODO: throw an exception if there are fewer tab-delimited + // tokens than the number of variables specified. + String token = ""; + int tokensize = 0; + for (int varindex = 0; varindex < varcount; varindex++) { + // TODO: figure out the safest way to convert strings to + // bytes here. Is it going to be safer to use getBytes("UTF8")? + // we are already making the assumption that the values + // in the tab file are in UTF8. -- L.A. + token = line[varindex] + "\n"; + tokensize = token.getBytes().length; + if (bufferedSizes[varindex] + tokensize > MAX_COLUMN_BUFFER) { + // fill the buffer and dump its contents into the temp file: + // (do note that there may be *several* MAX_COLUMN_BUFFERs + // worth of bytes in the token!) + + int tokenoffset = 0; + + if (bufferedSizes[varindex] != MAX_COLUMN_BUFFER) { + tokenoffset = MAX_COLUMN_BUFFER - bufferedSizes[varindex]; + System.arraycopy(token.getBytes(), 0, bufferedColumns[varindex], bufferedSizes[varindex], tokenoffset); + } // (otherwise the buffer is already full, and we should + // simply dump it into the temp file, without adding any + // extra bytes to it) + + File bufferTempFile = columnTempFiles[varindex]; + if (bufferTempFile == null) { + bufferTempFile = File.createTempFile("columnBufferFile", "bytes"); + columnTempFiles[varindex] = bufferTempFile; + } + + // *append* the contents of the buffer to the end of the + // temp file, if already exists: + try (BufferedOutputStream outputStream = new BufferedOutputStream( + new FileOutputStream(bufferTempFile, true))) { + outputStream.write(bufferedColumns[varindex], 0, MAX_COLUMN_BUFFER); + cachedfileSizes[varindex] += MAX_COLUMN_BUFFER; + + // keep writing MAX_COLUMN_BUFFER-size chunks of bytes into + // the temp file, for as long as there's more than MAX_COLUMN_BUFFER + // bytes left in the token: + + while (tokensize - tokenoffset > MAX_COLUMN_BUFFER) { + outputStream.write(token.getBytes(), tokenoffset, MAX_COLUMN_BUFFER); + cachedfileSizes[varindex] += MAX_COLUMN_BUFFER; + tokenoffset += MAX_COLUMN_BUFFER; + } + + } + + // buffer the remaining bytes and reset the buffered + // byte counter: + + System.arraycopy(token.getBytes(), + tokenoffset, + bufferedColumns[varindex], + 0, + tokensize - tokenoffset); + + bufferedSizes[varindex] = tokensize - tokenoffset; + + } else { + // continue buffering + System.arraycopy(token.getBytes(), 0, bufferedColumns[varindex], bufferedSizes[varindex], tokensize); + bufferedSizes[varindex] += tokensize; } - - outputStream.close(); - - // buffer the remaining bytes and reset the buffered - // byte counter: - - System.arraycopy(token.getBytes(), - tokenoffset, - bufferedColumns[varindex], - 0, - tokensize - tokenoffset); - - bufferedSizes[varindex] = tokensize - tokenoffset; - - } else { - // continue buffering - System.arraycopy(token.getBytes(), 0, bufferedColumns[varindex], bufferedSizes[varindex], tokensize); - bufferedSizes[varindex] += tokensize; } + } else { + throw new IOException("Tab file has fewer rows than the stored number of cases!"); } - } else { - scanner.close(); - throw new IOException("Tab file has fewer rows than the stored number of cases!"); } - } // OK, we've created the individual byte vectors of the tab file columns; @@ -1235,60 +1219,61 @@ private File generateRotatedImage (File tabfile, int varcount, int casecount) th // We now need to go through all these buffers and create the final // rotated image file. - BufferedOutputStream finalOut = new BufferedOutputStream(new FileOutputStream (new File(rotatedImageFileName))); - - // but first we should create the offset header and write it out into - // the final file; because it should be at the head, doh! - - long columnOffset = varcount * 8; - // (this is the offset of the first column vector; it is equal to the - // size of the offset header, i.e. varcount * 8 bytes) - - for (int varindex = 0; varindex < varcount; varindex++) { - long totalColumnBytes = cachedfileSizes[varindex] + bufferedSizes[varindex]; - columnOffset+=totalColumnBytes; - //totalColumnBytes; - byte[] columnOffsetByteArray = ByteBuffer.allocate(8).putLong(columnOffset).array(); - System.arraycopy(columnOffsetByteArray, 0, offsetHeader, varindex * 8, 8); - } - - finalOut.write(offsetHeader, 0, varcount * 8); - - for (int varindex = 0; varindex < varcount; varindex++) { - long cachedBytesRead = 0; - - // check if there is a cached temp file: - - File cachedTempFile = columnTempFiles[varindex]; - if (cachedTempFile != null) { - byte[] cachedBytes = new byte[MAX_COLUMN_BUFFER]; - BufferedInputStream cachedIn = new BufferedInputStream(new FileInputStream(cachedTempFile)); - int readlen = 0; - while ((readlen = cachedIn.read(cachedBytes)) > -1) { - finalOut.write(cachedBytes, 0, readlen); - cachedBytesRead += readlen; - } - cachedIn.close(); - // delete the temp file: - cachedTempFile.delete(); - + try (BufferedOutputStream finalOut = new BufferedOutputStream( + new FileOutputStream(new File(rotatedImageFileName)))) { + + // but first we should create the offset header and write it out into + // the final file; because it should be at the head, doh! + + long columnOffset = varcount * 8; + // (this is the offset of the first column vector; it is equal to the + // size of the offset header, i.e. varcount * 8 bytes) + + for (int varindex = 0; varindex < varcount; varindex++) { + long totalColumnBytes = cachedfileSizes[varindex] + bufferedSizes[varindex]; + columnOffset += totalColumnBytes; + // totalColumnBytes; + byte[] columnOffsetByteArray = ByteBuffer.allocate(8).putLong(columnOffset).array(); + System.arraycopy(columnOffsetByteArray, 0, offsetHeader, varindex * 8, 8); } - - if (cachedBytesRead != cachedfileSizes[varindex]) { - finalOut.close(); - throw new IOException("Could not read the correct number of bytes cached for column "+varindex+"; "+ + + finalOut.write(offsetHeader, 0, varcount * 8); + + for (int varindex = 0; varindex < varcount; varindex++) { + long cachedBytesRead = 0; + + // check if there is a cached temp file: + + File cachedTempFile = columnTempFiles[varindex]; + if (cachedTempFile != null) { + byte[] cachedBytes = new byte[MAX_COLUMN_BUFFER]; + try (BufferedInputStream cachedIn = new BufferedInputStream(new FileInputStream(cachedTempFile))) { + int readlen = 0; + while ((readlen = cachedIn.read(cachedBytes)) > -1) { + finalOut.write(cachedBytes, 0, readlen); + cachedBytesRead += readlen; + } + } + + // delete the temp file: + cachedTempFile.delete(); + + } + + if (cachedBytesRead != cachedfileSizes[varindex]) { + throw new IOException("Could not read the correct number of bytes cached for column "+varindex+"; "+ cachedfileSizes[varindex] + " bytes expected, "+cachedBytesRead+" read."); + } + + // then check if there are any bytes buffered for this column: + + if (bufferedSizes[varindex] > 0) { + finalOut.write(bufferedColumns[varindex], 0, bufferedSizes[varindex]); + } + } - - // then check if there are any bytes buffered for this column: - - if (bufferedSizes[varindex] > 0) { - finalOut.write(bufferedColumns[varindex], 0, bufferedSizes[varindex]); - } - } - finalOut.close(); return new File(rotatedImageFileName); } @@ -1305,88 +1290,87 @@ private File generateRotatedImage (File tabfile, int varcount, int casecount) th */ private void reverseRotatedImage (File rotfile, int varcount, int casecount) throws IOException { // open the file, read in the offset header: - BufferedInputStream rotfileStream = new BufferedInputStream(new FileInputStream(rotfile)); - - byte[] offsetHeader = new byte[varcount * 8]; - long[] byteOffsets = new long[varcount]; - - int readlen = rotfileStream.read(offsetHeader); - - if (readlen != varcount * 8) { - throw new IOException ("Could not read "+varcount*8+" header bytes from the rotated file."); - } - - for (int varindex = 0; varindex < varcount; varindex++) { - byte[] offsetBytes = new byte[8]; - System.arraycopy(offsetHeader, varindex*8, offsetBytes, 0, 8); - - ByteBuffer offsetByteBuffer = ByteBuffer.wrap(offsetBytes); - byteOffsets[varindex] = offsetByteBuffer.getLong(); - - //System.out.println(byteOffsets[varindex]); - } - - String [][] reversedMatrix = new String[casecount][varcount]; - - long offset = varcount * 8; - byte[] columnBytes; - - for (int varindex = 0; varindex < varcount; varindex++) { - long columnLength = byteOffsets[varindex] - offset; + try (BufferedInputStream rotfileStream = new BufferedInputStream(new FileInputStream(rotfile))) { + byte[] offsetHeader = new byte[varcount * 8]; + long[] byteOffsets = new long[varcount]; + int readlen = rotfileStream.read(offsetHeader); - - columnBytes = new byte[(int)columnLength]; - readlen = rotfileStream.read(columnBytes); - - if (readlen != columnLength) { - throw new IOException ("Could not read "+columnBytes+" bytes for column "+varindex); + if (readlen != varcount * 8) { + throw new IOException ("Could not read "+varcount*8+" header bytes from the rotated file."); } - /* - String columnString = new String(columnBytes); - //System.out.print(columnString); - String[] values = columnString.split("\n", -1); - if (values.length < casecount) { - throw new IOException("count mismatch: "+values.length+" tokens found for column "+varindex); + for (int varindex = 0; varindex < varcount; varindex++) { + byte[] offsetBytes = new byte[8]; + System.arraycopy(offsetHeader, varindex*8, offsetBytes, 0, 8); + + ByteBuffer offsetByteBuffer = ByteBuffer.wrap(offsetBytes); + byteOffsets[varindex] = offsetByteBuffer.getLong(); + + //System.out.println(byteOffsets[varindex]); } - for (int caseindex = 0; caseindex < casecount; caseindex++) { - reversedMatrix[caseindex][varindex] = values[caseindex]; - }*/ + String [][] reversedMatrix = new String[casecount][varcount]; + + long offset = varcount * 8; + byte[] columnBytes; - int bytecount = 0; - int byteoffset = 0; - int caseindex = 0; - //System.out.println("generating value vector for column "+varindex); - while (bytecount < columnLength) { - if (columnBytes[bytecount] == '\n') { - String token = new String(columnBytes, byteoffset, bytecount-byteoffset); - reversedMatrix[caseindex++][varindex] = token; - byteoffset = bytecount + 1; + for (int varindex = 0; varindex < varcount; varindex++) { + long columnLength = byteOffsets[varindex] - offset; + + + + columnBytes = new byte[(int)columnLength]; + readlen = rotfileStream.read(columnBytes); + + if (readlen != columnLength) { + throw new IOException ("Could not read "+columnBytes+" bytes for column "+varindex); + } + /* + String columnString = new String(columnBytes); + //System.out.print(columnString); + String[] values = columnString.split("\n", -1); + + if (values.length < casecount) { + throw new IOException("count mismatch: "+values.length+" tokens found for column "+varindex); } - bytecount++; + + for (int caseindex = 0; caseindex < casecount; caseindex++) { + reversedMatrix[caseindex][varindex] = values[caseindex]; + }*/ + + int bytecount = 0; + int byteoffset = 0; + int caseindex = 0; + //System.out.println("generating value vector for column "+varindex); + while (bytecount < columnLength) { + if (columnBytes[bytecount] == '\n') { + String token = new String(columnBytes, byteoffset, bytecount-byteoffset); + reversedMatrix[caseindex++][varindex] = token; + byteoffset = bytecount + 1; + } + bytecount++; + } + + if (caseindex != casecount) { + throw new IOException("count mismatch: "+caseindex+" tokens found for column "+varindex); + } + offset = byteOffsets[varindex]; } - if (caseindex != casecount) { - throw new IOException("count mismatch: "+caseindex+" tokens found for column "+varindex); - } - offset = byteOffsets[varindex]; - } - - for (int caseindex = 0; caseindex < casecount; caseindex++) { - for (int varindex = 0; varindex < varcount; varindex++) { - System.out.print(reversedMatrix[caseindex][varindex]); - if (varindex < varcount-1) { - System.out.print("\t"); - } else { - System.out.print("\n"); + for (int caseindex = 0; caseindex < casecount; caseindex++) { + for (int varindex = 0; varindex < varcount; varindex++) { + System.out.print(reversedMatrix[caseindex][varindex]); + if (varindex < varcount-1) { + System.out.print("\t"); + } else { + System.out.print("\n"); + } } } + } - rotfileStream.close(); - } diff --git a/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java b/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java index 5daa09597d4..a6b3925a7b2 100644 --- a/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java +++ b/src/main/java/edu/harvard/iq/dataverse/datasetutility/AddReplaceFileHelper.java @@ -15,8 +15,8 @@ import edu.harvard.iq.dataverse.EjbDataverseEngine; import edu.harvard.iq.dataverse.FileMetadata; import edu.harvard.iq.dataverse.PermissionServiceBean; +import edu.harvard.iq.dataverse.api.ApiConstants; import edu.harvard.iq.dataverse.api.Util; -import edu.harvard.iq.dataverse.api.Files; import edu.harvard.iq.dataverse.authorization.users.AuthenticatedUser; import edu.harvard.iq.dataverse.authorization.users.User; import edu.harvard.iq.dataverse.dataaccess.DataAccess; @@ -62,8 +62,6 @@ import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; import org.apache.commons.io.IOUtils; -import static edu.harvard.iq.dataverse.api.AbstractApiBean.STATUS_ERROR; -import static edu.harvard.iq.dataverse.api.AbstractApiBean.STATUS_OK; import static jakarta.ws.rs.core.Response.Status.BAD_REQUEST; /** @@ -2173,7 +2171,7 @@ public Response addFiles(String jsonData, Dataset dataset, User authUser) { return Response.ok().entity(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", Json.createObjectBuilder().add("Files", jarr).add("Result", result)).build() ).build(); } @@ -2341,14 +2339,14 @@ public Response replaceFiles(String jsonData, Dataset ds, User authUser) { .add("Number of files successfully replaced", successNumberofFiles); return Response.ok().entity(Json.createObjectBuilder() - .add("status", STATUS_OK) + .add("status", ApiConstants.STATUS_OK) .add("data", Json.createObjectBuilder().add("Files", jarr).add("Result", result)).build() ).build(); } protected static Response error(Response.Status sts, String msg ) { return Response.status(sts) .entity( NullSafeJsonBuilder.jsonObjectBuilder() - .add("status", STATUS_ERROR) + .add("status", ApiConstants.STATUS_ERROR) .add( "message", msg ).build() ).type(MediaType.APPLICATION_JSON_TYPE).build(); } diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java index 66ba00bcf55..ca5bf1d3f2c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/CuratePublishedDatasetVersionCommand.java @@ -99,6 +99,10 @@ public Dataset execute(CommandContext ctxt) throws CommandException { logger.severe("Draft version of dataset: " + tempDataset.getId() + " has: " + newFileCount + " while last published version has " + pubFileCount); throw new IllegalCommandException(BundleUtil.getStringFromBundle("datasetversion.update.failure"), this); } + Long thumbId = null; + if(tempDataset.getThumbnailFile()!=null) { + thumbId = tempDataset.getThumbnailFile().getId(); + }; for (FileMetadata publishedFmd : pubFmds) { DataFile dataFile = publishedFmd.getDataFile(); FileMetadata draftFmd = dataFile.getLatestFileMetadata(); @@ -136,6 +140,10 @@ public Dataset execute(CommandContext ctxt) throws CommandException { for (DataFileCategory cat : tempDataset.getCategories()) { cat.getFileMetadatas().remove(draftFmd); } + //And any thumbnail reference + if(publishedFmd.getDataFile().getId()==thumbId) { + tempDataset.setThumbnailFile(publishedFmd.getDataFile()); + } } // Update modification time on the published version and the dataset diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DRSSubmitToArchiveCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DRSSubmitToArchiveCommand.java index 05724e1be50..594d4fe25ba 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DRSSubmitToArchiveCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/DRSSubmitToArchiveCommand.java @@ -305,7 +305,10 @@ public static String createJWTString(Algorithm algorithmRSA, String installation String canonicalBody = new JsonCanonicalizer(body).getEncodedString(); logger.fine("Canonical body: " + canonicalBody); String digest = DigestUtils.sha256Hex(canonicalBody); - return JWT.create().withIssuer(BrandingUtil.getInstallationBrandName()).withIssuedAt(Date.from(Instant.now())) + if(installationBrandName==null) { + installationBrandName = BrandingUtil.getInstallationBrandName(); + } + return JWT.create().withIssuer(installationBrandName).withIssuedAt(Date.from(Instant.now())) .withExpiresAt(Date.from(Instant.now().plusSeconds(60 * expirationInMinutes))) .withKeyId("defaultDataverse").withClaim("bodySHA256Hash", digest).sign(algorithmRSA); } diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GoogleCloudSubmitToArchiveCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GoogleCloudSubmitToArchiveCommand.java index 3a370eddf8d..17aed3ad0ca 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GoogleCloudSubmitToArchiveCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/GoogleCloudSubmitToArchiveCommand.java @@ -1,16 +1,24 @@ package edu.harvard.iq.dataverse.engine.command.impl; +import com.google.auth.oauth2.ServiceAccountCredentials; +import com.google.cloud.storage.Blob; +import com.google.cloud.storage.Bucket; +import com.google.cloud.storage.Storage; +import com.google.cloud.storage.StorageException; +import com.google.cloud.storage.StorageOptions; import edu.harvard.iq.dataverse.Dataset; -import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.DatasetLock.Reason; +import edu.harvard.iq.dataverse.DatasetVersion; import edu.harvard.iq.dataverse.authorization.Permission; import edu.harvard.iq.dataverse.authorization.users.ApiToken; import edu.harvard.iq.dataverse.engine.command.Command; import edu.harvard.iq.dataverse.engine.command.DataverseRequest; import edu.harvard.iq.dataverse.engine.command.RequiredPermissions; +import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.workflow.step.Failure; import edu.harvard.iq.dataverse.workflow.step.WorkflowStepResult; - +import org.apache.commons.codec.binary.Hex; +import java.io.File; import java.io.FileInputStream; import java.io.IOException; import java.io.PipedInputStream; @@ -56,10 +64,11 @@ public WorkflowStepResult performArchiveSubmission(DatasetVersion dv, ApiToken t statusObject.add(DatasetVersion.ARCHIVAL_STATUS, DatasetVersion.ARCHIVAL_STATUS_FAILURE); statusObject.add(DatasetVersion.ARCHIVAL_STATUS_MESSAGE, "Bag not transferred"); - try { - FileInputStream fis = new FileInputStream(System.getProperty("dataverse.files.directory") + System.getProperty("file.separator") + "googlecloudkey.json"); + String cloudKeyFile = JvmSettings.FILES_DIRECTORY.lookup() + File.separator + "googlecloudkey.json"; + + try (FileInputStream cloudKeyStream = new FileInputStream(cloudKeyFile)) { storage = StorageOptions.newBuilder() - .setCredentials(ServiceAccountCredentials.fromStream(fis)) + .setCredentials(ServiceAccountCredentials.fromStream(cloudKeyStream)) .setProjectId(projectName) .build() .getService(); diff --git a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/ImportFromFileSystemCommand.java b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/ImportFromFileSystemCommand.java index 9c3c1bc229c..f093b6aa416 100644 --- a/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/ImportFromFileSystemCommand.java +++ b/src/main/java/edu/harvard/iq/dataverse/engine/command/impl/ImportFromFileSystemCommand.java @@ -12,6 +12,8 @@ import edu.harvard.iq.dataverse.engine.command.RequiredPermissions; import edu.harvard.iq.dataverse.engine.command.exception.CommandException; import edu.harvard.iq.dataverse.engine.command.exception.IllegalCommandException; +import edu.harvard.iq.dataverse.settings.JvmSettings; + import static edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder.jsonObjectBuilder; import java.io.File; import java.util.Properties; @@ -69,18 +71,20 @@ public JsonObject execute(CommandContext ctxt) throws CommandException { logger.info(error); throw new IllegalCommandException(error, this); } - File directory = new File(System.getProperty("dataverse.files.directory") - + File.separator + dataset.getAuthority() + File.separator + dataset.getIdentifier()); - // TODO: - // The above goes directly to the filesystem directory configured by the - // old "dataverse.files.directory" JVM option (otherwise used for temp - // files only, after the Multistore implementation (#6488). - // We probably want package files to be able to use specific stores instead. - // More importantly perhaps, the approach above does not take into account - // if the dataset may have an AlternativePersistentIdentifier, that may be - // designated isStorageLocationDesignator() - i.e., if a different identifer - // needs to be used to name the storage directory, instead of the main/current - // persistent identifier above. + + File directory = new File( + String.join(File.separator, JvmSettings.FILES_DIRECTORY.lookup(), + dataset.getAuthority(), dataset.getIdentifier())); + + // TODO: The above goes directly to the filesystem directory configured by the + // old "dataverse.files.directory" JVM option (otherwise used for temp + // files only, after the Multistore implementation (#6488). + // We probably want package files to be able to use specific stores instead. + // More importantly perhaps, the approach above does not take into account + // if the dataset may have an AlternativePersistentIdentifier, that may be + // designated isStorageLocationDesignator() - i.e., if a different identifer + // needs to be used to name the storage directory, instead of the main/current + // persistent identifier above. if (!isValidDirectory(directory)) { String error = "Dataset directory is invalid. " + directory; logger.info(error); @@ -93,11 +97,10 @@ public JsonObject execute(CommandContext ctxt) throws CommandException { throw new IllegalCommandException(error, this); } - File uploadDirectory = new File(System.getProperty("dataverse.files.directory") - + File.separator + dataset.getAuthority() + File.separator + dataset.getIdentifier() - + File.separator + uploadFolder); - // TODO: - // see the comment above. + File uploadDirectory = new File(String.join(File.separator, JvmSettings.FILES_DIRECTORY.lookup(), + dataset.getAuthority(), dataset.getIdentifier(), uploadFolder)); + + // TODO: see the comment above. if (!isValidDirectory(uploadDirectory)) { String error = "Upload folder is not a valid directory."; logger.info(error); diff --git a/src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAireExportUtil.java b/src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAireExportUtil.java index a777ec1154e..eae9811f70b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAireExportUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/export/openaire/OpenAireExportUtil.java @@ -255,7 +255,10 @@ public static void writeCreatorsElement(XMLStreamWriter xmlw, DatasetVersionDTO creator_map.put("nameType", "Personal"); nameType_check = true; } - + // ToDo - the algorithm to determine if this is a Person or Organization here + // has been abstracted into a separate + // edu.harvard.iq.dataverse.util.PersonOrOrgUtil class that could be used here + // to avoid duplication/variants of the algorithm creatorName = Cleanup.normalize(creatorName); // Datacite algorithm, https://github.com/IQSS/dataverse/issues/2243#issuecomment-358615313 if (creatorName.contains(",")) { @@ -705,6 +708,11 @@ public static void writeContributorElement(XMLStreamWriter xmlw, String contribu boolean nameType_check = false; Map contributor_map = new HashMap(); + // ToDo - the algorithm to determine if this is a Person or Organization here + // has been abstracted into a separate + // edu.harvard.iq.dataverse.util.PersonOrOrgUtil class that could be used here + // to avoid duplication/variants of the algorithm + contributorName = Cleanup.normalize(contributorName); // Datacite algorithm, https://github.com/IQSS/dataverse/issues/2243#issuecomment-358615313 if (contributorName.contains(",")) { @@ -716,6 +724,9 @@ public static void writeContributorElement(XMLStreamWriter xmlw, String contribu // givenName ok contributor_map.put("nameType", "Personal"); nameType_check = true; + // re: the above toDo - the ("ContactPerson".equals(contributorType) && + // !isValidEmailAddress(contributorName)) clause in the next line could/should + // be sent as the OrgIfTied boolean parameter } else if (isOrganization || ("ContactPerson".equals(contributorType) && !isValidEmailAddress(contributorName))) { contributor_map.put("nameType", "Organizational"); } diff --git a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java index 6ac1201a3bc..37f705a5a31 100644 --- a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java +++ b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalTool.java @@ -38,6 +38,7 @@ public class ExternalTool implements Serializable { public static final String CONTENT_TYPE = "contentType"; public static final String TOOL_NAME = "toolName"; public static final String ALLOWED_API_CALLS = "allowedApiCalls"; + public static final String REQUIREMENTS = "requirements"; @Id @GeneratedValue(strategy = GenerationType.IDENTITY) @@ -102,6 +103,15 @@ public class ExternalTool implements Serializable { @Column(nullable = true, columnDefinition = "TEXT") private String allowedApiCalls; + /** + * When non-null, the tool has indicated that it has certain requirements + * that must be met before it should be shown to the user. This + * functionality was added for tools that operate on aux files rather than + * data files so "auxFilesExist" is one of the possible values. + */ + @Column(nullable = true, columnDefinition = "TEXT") + private String requirements; + /** * This default constructor is only here to prevent this error at * deployment: @@ -117,10 +127,10 @@ public ExternalTool() { } public ExternalTool(String displayName, String toolName, String description, List externalToolTypes, Scope scope, String toolUrl, String toolParameters, String contentType) { - this(displayName, toolName, description, externalToolTypes, scope, toolUrl, toolParameters, contentType, null); + this(displayName, toolName, description, externalToolTypes, scope, toolUrl, toolParameters, contentType, null, null); } - public ExternalTool(String displayName, String toolName, String description, List externalToolTypes, Scope scope, String toolUrl, String toolParameters, String contentType, String allowedApiCalls) { + public ExternalTool(String displayName, String toolName, String description, List externalToolTypes, Scope scope, String toolUrl, String toolParameters, String contentType, String allowedApiCalls, String requirements) { this.displayName = displayName; this.toolName = toolName; this.description = description; @@ -130,6 +140,7 @@ public ExternalTool(String displayName, String toolName, String description, Lis this.toolParameters = toolParameters; this.contentType = contentType; this.allowedApiCalls = allowedApiCalls; + this.requirements = requirements; } public enum Type { @@ -325,5 +336,12 @@ public void setAllowedApiCalls(String allowedApiCalls) { this.allowedApiCalls = allowedApiCalls; } + public String getRequirements() { + return requirements; + } + + public void setRequirements(String requirements) { + this.requirements = requirements; + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java index 4d715be1195..d6cc04b59a1 100644 --- a/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/externaltools/ExternalToolServiceBean.java @@ -1,5 +1,7 @@ package edu.harvard.iq.dataverse.externaltools; +import edu.harvard.iq.dataverse.AuxiliaryFile; +import edu.harvard.iq.dataverse.AuxiliaryFileServiceBean; import edu.harvard.iq.dataverse.DataFile; import edu.harvard.iq.dataverse.DataFileServiceBean; import edu.harvard.iq.dataverse.authorization.users.ApiToken; @@ -14,6 +16,8 @@ import java.util.List; import java.util.Set; import java.util.logging.Logger; + +import jakarta.ejb.EJB; import jakarta.ejb.Stateless; import jakarta.inject.Named; import jakarta.json.Json; @@ -21,6 +25,7 @@ import jakarta.json.JsonObject; import jakarta.json.JsonObjectBuilder; import jakarta.json.JsonReader; +import jakarta.json.JsonValue; import jakarta.persistence.EntityManager; import jakarta.persistence.NoResultException; import jakarta.persistence.NonUniqueResultException; @@ -38,6 +43,9 @@ public class ExternalToolServiceBean { @PersistenceContext(unitName = "VDCNet-ejbPU") private EntityManager em; + @EJB + AuxiliaryFileServiceBean auxiliaryFileService; + public List findAll() { TypedQuery typedQuery = em.createQuery("SELECT OBJECT(o) FROM ExternalTool AS o ORDER BY o.id", ExternalTool.class); return typedQuery.getResultList(); @@ -131,13 +139,13 @@ public ExternalTool save(ExternalTool externalTool) { * file supports The list of tools is passed in so it doesn't hit the * database each time */ - public static List findExternalToolsByFile(List allExternalTools, DataFile file) { + public List findExternalToolsByFile(List allExternalTools, DataFile file) { List externalTools = new ArrayList<>(); //Map tabular data to it's mimetype (the isTabularData() check assures that this code works the same as before, but it may need to change if tabular data is split into subtypes with differing mimetypes) final String contentType = file.isTabularData() ? DataFileServiceBean.MIME_TYPE_TSV_ALT : file.getContentType(); allExternalTools.forEach((externalTool) -> { - //Match tool and file type - if (contentType.equals(externalTool.getContentType())) { + //Match tool and file type, then check requirements + if (contentType.equals(externalTool.getContentType()) && meetsRequirements(externalTool, file)) { externalTools.add(externalTool); } }); @@ -145,6 +153,31 @@ public static List findExternalToolsByFile(List allE return externalTools; } + public boolean meetsRequirements(ExternalTool externalTool, DataFile dataFile) { + String requirements = externalTool.getRequirements(); + if (requirements == null) { + logger.fine("Data file id" + dataFile.getId() + ": no requirements for tool id " + externalTool.getId()); + return true; + } + boolean meetsRequirements = true; + JsonObject requirementsObj = JsonUtil.getJsonObject(requirements); + JsonArray auxFilesExist = requirementsObj.getJsonArray("auxFilesExist"); + for (JsonValue jsonValue : auxFilesExist) { + String formatTag = jsonValue.asJsonObject().getString("formatTag"); + String formatVersion = jsonValue.asJsonObject().getString("formatVersion"); + AuxiliaryFile auxFile = auxiliaryFileService.lookupAuxiliaryFile(dataFile, formatTag, formatVersion); + if (auxFile == null) { + logger.fine("Data file id" + dataFile.getId() + ": cannot find required aux file. formatTag=" + formatTag + ". formatVersion=" + formatVersion); + meetsRequirements = false; + break; + } else { + logger.fine("Data file id" + dataFile.getId() + ": found required aux file. formatTag=" + formatTag + ". formatVersion=" + formatVersion); + meetsRequirements = true; + } + } + return meetsRequirements; + } + public static ExternalTool parseAddExternalToolManifest(String manifest) { if (manifest == null || manifest.isEmpty()) { @@ -168,6 +201,7 @@ public static ExternalTool parseAddExternalToolManifest(String manifest) { JsonObject toolParametersObj = jsonObject.getJsonObject(TOOL_PARAMETERS); JsonArray queryParams = toolParametersObj.getJsonArray("queryParameters"); JsonArray allowedApiCallsArray = jsonObject.getJsonArray(ALLOWED_API_CALLS); + JsonObject requirementsObj = jsonObject.getJsonObject(REQUIREMENTS); boolean allRequiredReservedWordsFound = false; if (scope.equals(Scope.FILE)) { @@ -225,8 +259,12 @@ public static ExternalTool parseAddExternalToolManifest(String manifest) { if(allowedApiCallsArray !=null) { allowedApiCalls = allowedApiCallsArray.toString(); } + String requirements = null; + if (requirementsObj != null) { + requirements = requirementsObj.toString(); + } - return new ExternalTool(displayName, toolName, description, externalToolTypes, scope, toolUrl, toolParameters, contentType, allowedApiCalls); + return new ExternalTool(displayName, toolName, description, externalToolTypes, scope, toolUrl, toolParameters, contentType, allowedApiCalls, requirements); } private static String getRequiredTopLevelField(JsonObject jsonObject, String key) { diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java index c5e3a93e2df..402d0d8ef91 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/FastGetRecord.java @@ -19,8 +19,8 @@ */ package edu.harvard.iq.dataverse.harvest.client; +import edu.harvard.iq.dataverse.harvest.client.oai.OaiHandler; import java.io.IOException; -import java.io.FileNotFoundException; import java.io.InputStream; import java.io.StringReader; @@ -31,9 +31,14 @@ import java.io.FileOutputStream; import java.io.PrintWriter; -import java.net.HttpURLConnection; +import static java.net.HttpURLConnection.HTTP_OK; import java.net.MalformedURLException; -import java.net.URL; +import java.net.URI; +import java.net.http.HttpClient; +import java.net.http.HttpRequest; +import java.net.http.HttpResponse; +import java.util.Map; +import java.util.Optional; import java.util.zip.GZIPInputStream; import java.util.zip.InflaterInputStream; @@ -84,17 +89,18 @@ public class FastGetRecord { /** * Client-side GetRecord verb constructor * - * @param baseURL the baseURL of the server to be queried + * @param oaiHandler the configured OaiHande running this harvest + * @param identifier Record identifier + * @param httpClient jdk HttpClient used to make http requests * @exception MalformedURLException the baseURL is bad * @exception SAXException the xml response is bad * @exception IOException an I/O error occurred + * @exception TransformerException if it fails to parse the service portion of the record */ - public FastGetRecord(String baseURL, String identifier, String metadataPrefix) - throws IOException, ParserConfigurationException, SAXException, + public FastGetRecord(OaiHandler oaiHandler, String identifier, HttpClient httpClient) throws IOException, ParserConfigurationException, SAXException, TransformerException { - harvestRecord (baseURL, identifier, metadataPrefix); - + harvestRecord (oaiHandler.getBaseOaiUrl(), identifier, oaiHandler.getMetadataPrefix(), oaiHandler.getCustomHeaders(), httpClient); } private String errorMessage = null; @@ -117,57 +123,63 @@ public boolean isDeleted () { } - public void harvestRecord(String baseURL, String identifier, String metadataPrefix) throws IOException, - ParserConfigurationException, SAXException, TransformerException { + public void harvestRecord(String baseURL, String identifier, String metadataPrefix, Map customHeaders, HttpClient httpClient) throws IOException, + ParserConfigurationException, SAXException, TransformerException{ xmlInputFactory = javax.xml.stream.XMLInputFactory.newInstance(); - String requestURL = getRequestURL(baseURL, identifier, metadataPrefix); + InputStream in; + + // This was one other place where the Harvester code was still using + // the obsolete java.net.ttpUrlConnection that didn't get replaced with + // the new java.net.http.HttpClient during the first pas of the XOAI + // rewrite. (L.A.) - InputStream in = null; - URL url = new URL(requestURL); - HttpURLConnection con = null; - int responseCode = 0; - - con = (HttpURLConnection) url.openConnection(); - con.setRequestProperty("User-Agent", "Dataverse Harvesting Client v5"); - con.setRequestProperty("Accept-Encoding", - "compress, gzip, identify"); - try { - responseCode = con.getResponseCode(); - //logger.debug("responseCode=" + responseCode); - } catch (FileNotFoundException e) { - //logger.info(requestURL, e); - responseCode = HttpURLConnection.HTTP_UNAVAILABLE; - } - - // TODO: -- L.A. - // - // support for cookies; - // support for limited retry attempts -- ? - // implement reading of the stream as filterinputstream -- ? - // -- that could make it a little faster still. -- L.A. - - - - if (responseCode == 200) { - - String contentEncoding = con.getHeaderField("Content-Encoding"); - //logger.debug("contentEncoding=" + contentEncoding); - - // support for the standard compress/gzip/deflate compression - // schemes: - if ("compress".equals(contentEncoding)) { - ZipInputStream zis = new ZipInputStream(con.getInputStream()); - zis.getNextEntry(); - in = zis; - } else if ("gzip".equals(contentEncoding)) { - in = new GZIPInputStream(con.getInputStream()); - } else if ("deflate".equals(contentEncoding)) { - in = new InflaterInputStream(con.getInputStream()); - } else { - in = con.getInputStream(); + if (httpClient == null) { + throw new IOException("Null Http Client, cannot make a GetRecord call to obtain the metadata."); + } + + HttpRequest.Builder requestBuilder = HttpRequest.newBuilder() + .uri(URI.create(requestURL)) + .GET() + .header("User-Agent", "XOAI Service Provider v5 (Dataverse)") + .header("Accept-Encoding", "compress, gzip"); + + if (customHeaders != null) { + for (String headerName : customHeaders.keySet()) { + requestBuilder.header(headerName, customHeaders.get(headerName)); + } + } + + HttpRequest request = requestBuilder.build(); + HttpResponse response; + + try { + response = httpClient.send(request, HttpResponse.BodyHandlers.ofInputStream()); + } catch (InterruptedException ex) { + Thread.currentThread().interrupt(); + throw new IOException("Failed to connect to the remote dataverse server to obtain GetRecord metadata"); + } + + int responseCode = response.statusCode(); + + if (responseCode == HTTP_OK) { + InputStream inputStream = response.body(); + Optional contentEncoding = response.headers().firstValue("Content-Encoding"); + + // support for the standard gzip encoding: + in = inputStream; + if (contentEncoding.isPresent()) { + if (contentEncoding.get().equals("compress")) { + ZipInputStream zis = new ZipInputStream(inputStream); + zis.getNextEntry(); + in = zis; + } else if (contentEncoding.get().equals("gzip")) { + in = new GZIPInputStream(inputStream); + } else if (contentEncoding.get().equals("deflate")) { + in = new InflaterInputStream(inputStream); + } } // We are going to read the OAI header and SAX-parse it for the @@ -185,9 +197,7 @@ public void harvestRecord(String baseURL, String identifier, String metadataPref FileOutputStream tempFileStream = null; PrintWriter metadataOut = null; - savedMetadataFile = File.createTempFile("meta", ".tmp"); - - + savedMetadataFile = File.createTempFile("meta", ".tmp"); int mopen = 0; int mclose = 0; diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java index dce7465181d..a8aa7e47125 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvesterServiceBean.java @@ -229,11 +229,9 @@ private void harvestOAI(DataverseRequest dataverseRequest, HarvestingClient harv throw new IOException(errorMessage); } - if (DATAVERSE_PROPRIETARY_METADATA_FORMAT.equals(oaiHandler.getMetadataPrefix())) { - // If we are harvesting native Dataverse json, we'll also need this - // jdk http client to make direct calls to the remote Dataverse API: - httpClient = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.ALWAYS).build(); - } + // We will use this jdk http client to make direct calls to the remote + // OAI (or remote Dataverse API) to obtain the metadata records + httpClient = HttpClient.newBuilder().followRedirects(HttpClient.Redirect.ALWAYS).build(); try { for (Iterator
idIter = oaiHandler.runListIdentifiers(); idIter.hasNext();) { @@ -296,7 +294,7 @@ private Long processRecord(DataverseRequest dataverseRequest, Logger hdLogger, P tempFile = retrieveProprietaryDataverseMetadata(httpClient, metadataApiUrl); } else { - FastGetRecord record = oaiHandler.runGetRecord(identifier); + FastGetRecord record = oaiHandler.runGetRecord(identifier, httpClient); errMessage = record.getErrorMessage(); deleted = record.isDeleted(); tempFile = record.getMetadataFile(); @@ -361,7 +359,7 @@ File retrieveProprietaryDataverseMetadata (HttpClient client, String remoteApiUr HttpRequest request = HttpRequest.newBuilder() .uri(URI.create(remoteApiUrl)) .GET() - .header("User-Agent", "Dataverse Harvesting Client v5") + .header("User-Agent", "XOAI Service Provider v5 (Dataverse)") .build(); HttpResponse response; diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvestingClient.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvestingClient.java index 9b00fd66844..06d7e58e1ce 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvestingClient.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/HarvestingClient.java @@ -231,6 +231,16 @@ public void setMetadataPrefix(String metadataPrefix) { this.metadataPrefix = metadataPrefix; } + private String customHttpHeaders; + + public String getCustomHttpHeaders() { + return customHttpHeaders; + } + + public void setCustomHttpHeaders(String customHttpHeaders) { + this.customHttpHeaders = customHttpHeaders; + } + // TODO: do we need "orphanRemoval=true"? -- L.A. 4.4 // TODO: should it be @OrderBy("startTime")? -- L.A. 4.4 @OneToMany(mappedBy="harvestingClient", cascade={CascadeType.REMOVE, CascadeType.MERGE, CascadeType.PERSIST}) @@ -342,95 +352,7 @@ public Long getLastDeletedDatasetCount() { return lastNonEmptyHarvest.getDeletedDatasetCount(); } return null; - } - - /* move the fields below to the new HarvestingClientRun class: - private String harvestResult; - - public String getResult() { - return harvestResult; - } - - public void setResult(String harvestResult) { - this.harvestResult = harvestResult; - } - - // "Last Harvest Time" is the last time we *attempted* to harvest - // from this remote resource. - // It wasn't necessarily a successful attempt! - - @Temporal(value = TemporalType.TIMESTAMP) - private Date lastHarvestTime; - - public Date getLastHarvestTime() { - return lastHarvestTime; - } - - public void setLastHarvestTime(Date lastHarvestTime) { - this.lastHarvestTime = lastHarvestTime; - } - - // This is the last "successful harvest" - i.e., the last time we - // tried to harvest, and got a response from the remote server. - // We may not have necessarily harvested any useful content though; - // the result may have been a "no content" or "no changes since the last harvest" - // response. - - @Temporal(value = TemporalType.TIMESTAMP) - private Date lastSuccessfulHarvestTime; - - public Date getLastSuccessfulHarvestTime() { - return lastSuccessfulHarvestTime; - } - - public void setLastSuccessfulHarvestTime(Date lastSuccessfulHarvestTime) { - this.lastSuccessfulHarvestTime = lastSuccessfulHarvestTime; - } - - // Finally, this is the time stamp from the last "non-empty" harvest. - // I.e. the last time we ran a harvest that actually resulted in - // some Datasets created, updated or deleted: - - @Temporal(value = TemporalType.TIMESTAMP) - private Date lastNonEmptyHarvestTime; - - public Date getLastNonEmptyHarvestTime() { - return lastNonEmptyHarvestTime; - } - - public void setLastNonEmptyHarvestTime(Date lastNonEmptyHarvestTime) { - this.lastNonEmptyHarvestTime = lastNonEmptyHarvestTime; - } - - // And these are the Dataset counts from that last "non-empty" harvest: - private Long harvestedDatasetCount; - private Long failedDatasetCount; - private Long deletedDatasetCount; - - public Long getLastHarvestedDatasetCount() { - return harvestedDatasetCount; - } - - public void setHarvestedDatasetCount(Long harvestedDatasetCount) { - this.harvestedDatasetCount = harvestedDatasetCount; - } - - public Long getLastFailedDatasetCount() { - return failedDatasetCount; - } - - public void setFailedDatasetCount(Long failedDatasetCount) { - this.failedDatasetCount = failedDatasetCount; - } - - public Long getLastDeletedDatasetCount() { - return deletedDatasetCount; - } - - public void setDeletedDatasetCount(Long deletedDatasetCount) { - this.deletedDatasetCount = deletedDatasetCount; - } - */ + } private boolean scheduled; diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java b/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java index c0a039e2d2b..bb3dc06972c 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/client/oai/OaiHandler.java @@ -5,7 +5,6 @@ import io.gdcc.xoai.model.oaipmh.results.MetadataFormat; import io.gdcc.xoai.model.oaipmh.results.Set; import io.gdcc.xoai.serviceprovider.ServiceProvider; -import io.gdcc.xoai.serviceprovider.client.JdkHttpOaiClient; import io.gdcc.xoai.serviceprovider.exceptions.BadArgumentException; import io.gdcc.xoai.serviceprovider.exceptions.InvalidOAIResponse; import io.gdcc.xoai.serviceprovider.exceptions.NoSetHierarchyException; @@ -15,8 +14,10 @@ import edu.harvard.iq.dataverse.harvest.client.FastGetRecord; import static edu.harvard.iq.dataverse.harvest.client.HarvesterServiceBean.DATAVERSE_PROPRIETARY_METADATA_API; import edu.harvard.iq.dataverse.harvest.client.HarvestingClient; +import io.gdcc.xoai.serviceprovider.client.JdkHttpOaiClient; import java.io.IOException; import java.io.Serializable; +import java.net.http.HttpClient; import javax.xml.parsers.ParserConfigurationException; import org.apache.commons.lang3.StringUtils; @@ -24,14 +25,18 @@ import javax.xml.transform.TransformerException; import java.util.ArrayList; import java.util.Date; +import java.util.HashMap; import java.util.Iterator; import java.util.List; +import java.util.Map; +import java.util.logging.Logger; /** * * @author Leonid Andreev */ public class OaiHandler implements Serializable { + private static final Logger logger = Logger.getLogger("edu.harvard.iq.dataverse.harvest.client.oai.OaiHandler"); public OaiHandler() { @@ -65,6 +70,8 @@ public OaiHandler(HarvestingClient harvestingClient) throws OaiHandlerException this.fromDate = harvestingClient.getLastNonEmptyHarvestTime(); + this.customHeaders = makeCustomHeaders(harvestingClient.getCustomHttpHeaders()); + this.harvestingClient = harvestingClient; } @@ -74,6 +81,7 @@ public OaiHandler(HarvestingClient harvestingClient) throws OaiHandlerException private String setName; private Date fromDate; private Boolean setListTruncated = false; + private Map customHeaders = null; private ServiceProvider serviceProvider; @@ -119,6 +127,14 @@ public boolean isSetListTruncated() { return setListTruncated; } + public Map getCustomHeaders() { + return this.customHeaders; + } + + public void setCustomHeaders(Map customHeaders) { + this.customHeaders = customHeaders; + } + public ServiceProvider getServiceProvider() throws OaiHandlerException { if (serviceProvider == null) { if (baseOaiUrl == null) { @@ -128,8 +144,15 @@ public ServiceProvider getServiceProvider() throws OaiHandlerException { context.withBaseUrl(baseOaiUrl); context.withGranularity(Granularity.Second); - // builds the client with the default parameters and the JDK http client: - context.withOAIClient(JdkHttpOaiClient.newBuilder().withBaseUrl(baseOaiUrl).build()); + + JdkHttpOaiClient.Builder xoaiClientBuilder = JdkHttpOaiClient.newBuilder().withBaseUrl(getBaseOaiUrl()); + if (getCustomHeaders() != null) { + for (String headerName : getCustomHeaders().keySet()) { + logger.fine("adding custom header; name: "+headerName+", value: "+getCustomHeaders().get(headerName)); + } + xoaiClientBuilder = xoaiClientBuilder.withCustomHeaders(getCustomHeaders()); + } + context.withOAIClient(xoaiClientBuilder.build()); serviceProvider = new ServiceProvider(context); } @@ -235,7 +258,7 @@ public Iterator
runListIdentifiers() throws OaiHandlerException { } - public FastGetRecord runGetRecord(String identifier) throws OaiHandlerException { + public FastGetRecord runGetRecord(String identifier, HttpClient httpClient) throws OaiHandlerException { if (StringUtils.isEmpty(this.baseOaiUrl)) { throw new OaiHandlerException("Attempted to execute GetRecord without server URL specified."); } @@ -244,7 +267,7 @@ public FastGetRecord runGetRecord(String identifier) throws OaiHandlerException } try { - return new FastGetRecord(this.baseOaiUrl, identifier, this.metadataPrefix); + return new FastGetRecord(this, identifier, httpClient); } catch (ParserConfigurationException pce) { throw new OaiHandlerException("ParserConfigurationException executing GetRecord: "+pce.getMessage()); } catch (SAXException se) { @@ -293,4 +316,28 @@ public void runIdentify() { // (we will need it, both for validating the remote server, // and to learn about its extended capabilities) } + + public Map makeCustomHeaders(String headersString) { + if (headersString != null) { + String[] parts = headersString.split("\\\\n"); + HashMap ret = new HashMap<>(); + logger.info("found "+parts.length+" parts"); + int count = 0; + for (int i = 0; i < parts.length; i++) { + if (parts[i].indexOf(':') > 0) { + String headerName = parts[i].substring(0, parts[i].indexOf(':')); + String headerValue = parts[i].substring(parts[i].indexOf(':')+1).strip(); + + ret.put(headerName, headerValue); + count++; + } + // simply skipping it if malformed; or we could throw an exception - ? + } + if (ret.size() > 0) { + logger.info("returning the array with "+ret.size()+" name/value pairs"); + return ret; + } + } + return null; + } } diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java index 6e9099259a4..0845a7b738e 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAIRecordServiceBean.java @@ -371,4 +371,16 @@ public List findDeletedOaiRecordsBySetName(String setName) { } } + public Instant getEarliestDate() { + String queryString = "SELECT min(r.lastUpdateTime) FROM OAIRecord r"; + TypedQuery query = em.createQuery(queryString, Date.class); + Date retDate = query.getSingleResult(); + if (retDate != null) { + return retDate.toInstant(); + } + + // if there are no records yet, return the default "now" + return new Date().toInstant(); + } + } diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java index ee4475e12c8..2bd666401c7 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/OAISetServiceBean.java @@ -27,7 +27,7 @@ import jakarta.persistence.PersistenceContext; import org.apache.solr.client.solrj.SolrQuery; import org.apache.solr.client.solrj.SolrServerException; -import org.apache.solr.client.solrj.impl.HttpSolrClient.RemoteSolrException; +import org.apache.solr.client.solrj.impl.BaseHttpSolrClient.RemoteSolrException; import org.apache.solr.client.solrj.response.QueryResponse; import org.apache.solr.common.SolrDocument; import org.apache.solr.common.SolrDocumentList; diff --git a/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java b/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java index 3f0f3ef2494..b1f68e84659 100644 --- a/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java +++ b/src/main/java/edu/harvard/iq/dataverse/harvest/server/web/servlet/OAIServlet.java @@ -15,8 +15,9 @@ import io.gdcc.xoai.dataprovider.repository.ItemRepository; import io.gdcc.xoai.dataprovider.repository.SetRepository; import io.gdcc.xoai.model.oaipmh.DeletedRecord; +import io.gdcc.xoai.model.oaipmh.Granularity; import io.gdcc.xoai.model.oaipmh.OAIPMH; - +import io.gdcc.xoai.services.impl.SimpleResumptionTokenFormat; import io.gdcc.xoai.xml.XmlWriter; import edu.harvard.iq.dataverse.DatasetServiceBean; import edu.harvard.iq.dataverse.DataverseServiceBean; @@ -36,6 +37,7 @@ import java.io.IOException; +import java.time.Instant; import java.util.logging.Logger; import jakarta.ejb.EJB; import jakarta.mail.internet.InternetAddress; @@ -127,10 +129,9 @@ public void init(ServletConfig config) throws ServletException { repositoryConfiguration = createRepositoryConfiguration(); - xoaiRepository = new Repository() + xoaiRepository = new Repository(repositoryConfiguration) .withSetRepository(setRepository) - .withItemRepository(itemRepository) - .withConfiguration(repositoryConfiguration); + .withItemRepository(itemRepository); dataProvider = new DataProvider(getXoaiContext(), getXoaiRepository()); } @@ -193,23 +194,30 @@ private RepositoryConfiguration createRepositoryConfiguration() { } // The admin email address associated with this installation: // (Note: if the setting does not exist, we are going to assume that they - // have a reason not to want to advertise their email address, so no - // email will be shown in the output of Identify. + // have a reason not to want to configure their email address, if it is + // a developer's instance, for example; or a reason not to want to + // advertise it to the world.) InternetAddress systemEmailAddress = MailUtil.parseSystemAddress(settingsService.getValueForKey(SettingsServiceBean.Key.SystemEmail)); - - RepositoryConfiguration repositoryConfiguration = RepositoryConfiguration.defaults() - .withEnableMetadataAttributes(true) - .withRepositoryName(repositoryName) - .withBaseUrl(systemConfig.getDataverseSiteUrl()+"/oai") + String systemEmailLabel = systemEmailAddress != null ? systemEmailAddress.getAddress() : "donotreply@localhost"; + + RepositoryConfiguration configuration = new RepositoryConfiguration.RepositoryConfigurationBuilder() + .withAdminEmail(systemEmailLabel) .withCompression("gzip") .withCompression("deflate") - .withAdminEmail(systemEmailAddress != null ? systemEmailAddress.getAddress() : null) - .withDeleteMethod(DeletedRecord.TRANSIENT) + .withGranularity(Granularity.Lenient) + .withResumptionTokenFormat(new SimpleResumptionTokenFormat().withGranularity(Granularity.Second)) + .withRepositoryName(repositoryName) + .withBaseUrl(systemConfig.getDataverseSiteUrl()+"/oai") + .withEarliestDate(recordService.getEarliestDate()) .withMaxListIdentifiers(maxListIdentifiers) + .withMaxListSets(maxListSets) .withMaxListRecords(maxListRecords) - .withMaxListSets(maxListSets); + .withDeleteMethod(DeletedRecord.TRANSIENT) + .withEnableMetadataAttributes(true) + .withRequireFromAfterEarliest(false) + .build(); - return repositoryConfiguration; + return configuration; } /** diff --git a/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java index 3b723c06d60..0ee6afdc87f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/ingest/IngestServiceBean.java @@ -20,6 +20,8 @@ package edu.harvard.iq.dataverse.ingest; +import edu.harvard.iq.dataverse.AuxiliaryFile; +import edu.harvard.iq.dataverse.AuxiliaryFileServiceBean; import edu.harvard.iq.dataverse.ControlledVocabularyValue; import edu.harvard.iq.dataverse.datavariable.VariableCategory; import edu.harvard.iq.dataverse.datavariable.VariableServiceBean; @@ -72,6 +74,7 @@ //import edu.harvard.iq.dvn.unf.*; import org.dataverse.unf.*; import java.io.BufferedInputStream; +import java.io.ByteArrayInputStream; import java.io.File; import java.io.FileInputStream; import java.io.IOException; @@ -81,6 +84,7 @@ import java.nio.channels.FileChannel; import java.nio.channels.ReadableByteChannel; import java.nio.channels.WritableByteChannel; +import java.nio.charset.StandardCharsets; import java.nio.file.DirectoryStream; import java.nio.file.Files; import java.nio.file.Path; @@ -111,6 +115,9 @@ import jakarta.jms.QueueConnection; import jakarta.jms.QueueSender; import jakarta.jms.QueueSession; +import jakarta.ws.rs.core.MediaType; +import ucar.nc2.NetcdfFile; +import ucar.nc2.NetcdfFiles; import jakarta.jms.Message; import jakarta.faces.application.FacesMessage; @@ -134,6 +141,8 @@ public class IngestServiceBean { @EJB DataFileServiceBean fileService; @EJB + AuxiliaryFileServiceBean auxiliaryFileService; + @EJB SystemConfig systemConfig; @Resource(lookup = "java:app/jms/queue/ingest") @@ -232,6 +241,9 @@ public List saveAndAddFilesToDataset(DatasetVersion version, savedSuccess = true; logger.fine("Success: permanently saved file " + dataFile.getFileMetadata().getLabel()); + // TODO: reformat this file to remove the many tabs added in cc08330 + extractMetadataNcml(dataFile, tempLocationPath); + } catch (IOException ioex) { logger.warning("Failed to save the file, storage id " + dataFile.getStorageIdentifier() + " (" + ioex.getMessage() + ")"); } finally { @@ -343,6 +355,7 @@ public List saveAndAddFilesToDataset(DatasetVersion version, try { // FITS is the only type supported for metadata // extraction, as of now. -- L.A. 4.0 + // Note that extractMetadataNcml() is used for NetCDF/HDF5. dataFile.setContentType("application/fits"); metadataExtracted = extractMetadata(tempFileLocation, dataFile, version); } catch (IOException mex) { @@ -565,7 +578,6 @@ public int compare(DataFile d1, DataFile d2) { return sb.toString(); } - public void produceSummaryStatistics(DataFile dataFile, File generatedTabularFile) throws IOException { /* logger.info("Skipping summary statistics and UNF."); @@ -1206,7 +1218,104 @@ public boolean extractMetadata(String tempFileLocation, DataFile dataFile, Datas return ingestSuccessful; } - + /** + * @param dataFile The DataFile from which to attempt NcML extraction + * (NetCDF or HDF5 format) + * @param tempLocationPath Null if the file is already saved to permanent + * storage. Otherwise, the path to the temp location of the files, as during + * initial upload. + * @return True if the Ncml files was created. False on any error or if the + * NcML file already exists. + */ + public boolean extractMetadataNcml(DataFile dataFile, Path tempLocationPath) { + boolean ncmlFileCreated = false; + logger.fine("extractMetadataNcml: dataFileIn: " + dataFile + ". tempLocationPath: " + tempLocationPath); + InputStream inputStream = null; + String dataFileLocation = null; + if (tempLocationPath != null) { + // This file was just uploaded and hasn't been saved to S3 or local storage. + dataFileLocation = tempLocationPath.toString(); + } else { + // This file is already on S3 or local storage. + File tempFile = null; + File localFile; + StorageIO storageIO; + try { + storageIO = dataFile.getStorageIO(); + storageIO.open(); + if (storageIO.isLocalFile()) { + localFile = storageIO.getFileSystemPath().toFile(); + dataFileLocation = localFile.getAbsolutePath(); + logger.fine("extractMetadataNcml: file is local. Path: " + dataFileLocation); + } else { + // Need to create a temporary local file: + tempFile = File.createTempFile("tempFileExtractMetadataNcml", ".tmp"); + try ( ReadableByteChannel targetFileChannel = (ReadableByteChannel) storageIO.getReadChannel(); FileChannel tempFileChannel = new FileOutputStream(tempFile).getChannel();) { + tempFileChannel.transferFrom(targetFileChannel, 0, storageIO.getSize()); + } + dataFileLocation = tempFile.getAbsolutePath(); + logger.fine("extractMetadataNcml: file is on S3. Downloaded and saved to temp path: " + dataFileLocation); + } + } catch (IOException ex) { + logger.info("While attempting to extract NcML, could not use storageIO for data file id " + dataFile.getId() + ". Exception: " + ex); + } + } + if (dataFileLocation != null) { + try ( NetcdfFile netcdfFile = NetcdfFiles.open(dataFileLocation)) { + logger.fine("trying to open " + dataFileLocation); + if (netcdfFile != null) { + // For now, empty string. What should we pass as a URL to toNcml()? The filename (including the path) most commonly at https://docs.unidata.ucar.edu/netcdf-java/current/userguide/ncml_cookbook.html + // With an empty string the XML will show 'location="file:"'. + String ncml = netcdfFile.toNcml(""); + inputStream = new ByteArrayInputStream(ncml.getBytes(StandardCharsets.UTF_8)); + } else { + logger.info("NetcdfFiles.open() could not open file id " + dataFile.getId() + " (null returned)."); + } + } catch (IOException ex) { + logger.info("NetcdfFiles.open() could not open file id " + dataFile.getId() + ". Exception caught: " + ex); + } + } else { + logger.info("dataFileLocation is null for file id " + dataFile.getId() + ". Can't extract NcML."); + } + if (inputStream != null) { + // If you change NcML, you must also change the previewer. + String formatTag = "NcML"; + // 0.1 is arbitrary. It's our first attempt to put out NcML so we're giving it a low number. + // If you bump the number here, be sure the bump the number in the previewer as well. + // We could use 2.2 here since that's the current version of NcML. + String formatVersion = "0.1"; + String origin = "netcdf-java"; + boolean isPublic = true; + // See also file.auxfiles.types.NcML in Bundle.properties. Used to group aux files in UI. + String type = "NcML"; + // XML because NcML doesn't have its own MIME/content type at https://www.iana.org/assignments/media-types/media-types.xhtml + MediaType mediaType = new MediaType("text", "xml"); + try { + // Let the cascade do the save if the file isn't yet on permanent storage. + boolean callSave = false; + if (tempLocationPath == null) { + callSave = true; + // Check for an existing NcML file + logger.fine("Checking for existing NcML aux file for file id " + dataFile.getId()); + AuxiliaryFile existingAuxiliaryFile = auxiliaryFileService.lookupAuxiliaryFile(dataFile, formatTag, formatVersion); + if (existingAuxiliaryFile != null) { + logger.fine("Aux file already exists for NetCDF/HDF5 file for file id " + dataFile.getId()); + return false; + } + } + AuxiliaryFile auxFile = auxiliaryFileService.processAuxiliaryFile(inputStream, dataFile, formatTag, formatVersion, origin, isPublic, type, mediaType, callSave); + logger.fine("Aux file extracted from NetCDF/HDF5 file saved to storage (but not to the database yet) from file id " + dataFile.getId()); + ncmlFileCreated = true; + } catch (Exception ex) { + logger.info("exception throw calling processAuxiliaryFile: " + ex); + } + } else { + logger.info("extractMetadataNcml: input stream is null! dataFileLocation was " + dataFileLocation); + } + + return ncmlFileCreated; + } + private void processDatasetMetadata(FileMetadataIngest fileMetadataIngest, DatasetVersion editVersion) throws IOException { diff --git a/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsServiceBean.java index 99791569a3d..32af50b6731 100644 --- a/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsServiceBean.java @@ -512,7 +512,9 @@ public JsonArray fileDownloads(String yyyymm, Dataverse d, boolean uniqueCounts) for (Object[] result : results) { JsonObjectBuilder job = Json.createObjectBuilder(); job.add(MetricsUtil.ID, (int) result[0]); - job.add(MetricsUtil.PID, (String) result[1]); + if(result[1]!=null) { + job.add(MetricsUtil.PID, (String) result[1]); + } job.add(MetricsUtil.COUNT, (long) result[2]); jab.add(job); } diff --git a/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsUtil.java b/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsUtil.java index f4ef7a81b23..74bb53e1191 100644 --- a/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/metrics/MetricsUtil.java @@ -225,7 +225,9 @@ public static JsonArray timeSeriesByIDAndPIDToJson(List results) { JsonObjectBuilder job = Json.createObjectBuilder(); job.add(MetricsUtil.DATE, date); job.add(ID, id); - job.add(PID, pids.get(id)); + if(pids.get(id)!=null) { + job.add(PID, pids.get(id)); + } job.add(COUNT, totals.get(id)); jab.add(job); } diff --git a/src/main/java/edu/harvard/iq/dataverse/mydata/DataRetrieverAPI.java b/src/main/java/edu/harvard/iq/dataverse/mydata/DataRetrieverAPI.java index a834e75f97f..46569ee8e7a 100644 --- a/src/main/java/edu/harvard/iq/dataverse/mydata/DataRetrieverAPI.java +++ b/src/main/java/edu/harvard/iq/dataverse/mydata/DataRetrieverAPI.java @@ -8,6 +8,7 @@ import edu.harvard.iq.dataverse.DataverseSession; import edu.harvard.iq.dataverse.DvObjectServiceBean; import edu.harvard.iq.dataverse.RoleAssigneeServiceBean; +import edu.harvard.iq.dataverse.api.auth.AuthRequired; import edu.harvard.iq.dataverse.search.SearchServiceBean; import edu.harvard.iq.dataverse.search.SolrQueryResponse; import edu.harvard.iq.dataverse.search.SolrSearchResult; @@ -38,7 +39,8 @@ import jakarta.ws.rs.Path; import jakarta.ws.rs.Produces; import jakarta.ws.rs.QueryParam; - +import jakarta.ws.rs.container.ContainerRequestContext; +import jakarta.ws.rs.core.Context; import edu.harvard.iq.dataverse.util.BundleUtil; import org.apache.commons.lang3.StringUtils; @@ -254,78 +256,49 @@ private String getJSONErrorString(String jsonMsg, String optionalLoggerMsg){ return jsonData.build().toString(); } - - /** - * @todo This should support the "X-Dataverse-key" header like the other - * APIs. - */ - @Path(retrieveDataPartialAPIPath) + @GET + @AuthRequired + @Path(retrieveDataPartialAPIPath) @Produces({"application/json"}) - public String retrieveMyDataAsJsonString(@QueryParam("dvobject_types") List dvobject_types, - @QueryParam("published_states") List published_states, - @QueryParam("selected_page") Integer selectedPage, - @QueryParam("mydata_search_term") String searchTerm, - @QueryParam("role_ids") List roleIds, - @QueryParam("userIdentifier") String userIdentifier, - @QueryParam("key") String apiToken) { //String myDataParams) { - //System.out.println("_YE_OLDE_QUERY_COUNTER_"); - //msgt("_YE_OLDE_QUERY_COUNTER_"); // for debug purposes - boolean DEBUG_MODE = false; + public String retrieveMyDataAsJsonString( + @Context ContainerRequestContext crc, + @QueryParam("dvobject_types") List dvobject_types, + @QueryParam("published_states") List published_states, + @QueryParam("selected_page") Integer selectedPage, + @QueryParam("mydata_search_term") String searchTerm, + @QueryParam("role_ids") List roleIds, + @QueryParam("userIdentifier") String userIdentifier) { boolean OTHER_USER = false; String localeCode = session.getLocaleCode(); String noMsgResultsFound = BundleUtil.getStringFromPropertyFile("dataretrieverAPI.noMsgResultsFound", "Bundle", new Locale(localeCode)); - - // For, superusers, the searchUser may differ from the authUser - // - AuthenticatedUser searchUser = null; - if (DEBUG_MODE==true){ // DEBUG: use userIdentifier - authUser = getUserFromIdentifier(userIdentifier); - if (authUser == null){ - return this.getJSONErrorString("Requires authentication", "retrieveMyDataAsJsonString. User not found! Shouldn't be using this anyway"); + if ((session.getUser() != null) && (session.getUser().isAuthenticated())) { + authUser = (AuthenticatedUser) session.getUser(); + } else { + try { + authUser = getRequestAuthenticatedUserOrDie(crc); + } catch (WrappedResponse e) { + return this.getJSONErrorString("Requires authentication. Please login.", "retrieveMyDataAsJsonString. User not found! Shouldn't be using this anyway"); } - } else if ((session.getUser() != null)&&(session.getUser().isAuthenticated())){ - authUser = (AuthenticatedUser)session.getUser(); - - // If person is a superuser, see if a userIdentifier has been specified - // and use that instead - if ((authUser.isSuperuser())&&(userIdentifier != null)&&(!userIdentifier.isEmpty())){ - searchUser = getUserFromIdentifier(userIdentifier); - if (searchUser != null){ - authUser = searchUser; - OTHER_USER = true; - }else{ - return this.getJSONErrorString("No user found for: \"" + userIdentifier + "\"", null); - } - } - } else if (apiToken != null) { // Is this being accessed by an API Token? - - authUser = findUserByApiToken(apiToken); - if (authUser == null){ - return this.getJSONErrorString("Requires authentication. Please login.", "retrieveMyDataAsJsonString. User not found! Shouldn't be using this anyway"); - }else{ - // If person is a superuser, see if a userIdentifier has been specified - // and use that instead - if ((authUser.isSuperuser())&&(userIdentifier != null)&&(!userIdentifier.isEmpty())){ - searchUser = getUserFromIdentifier(userIdentifier); - if (searchUser != null){ - authUser = searchUser; - OTHER_USER = true; - }else{ - return this.getJSONErrorString("No user found for: \"" + userIdentifier + "\"", null); - } - } + } + // For superusers, the searchUser may differ from the authUser + AuthenticatedUser searchUser = null; + // If the user is a superuser, see if a userIdentifier has been specified and use that instead + if ((authUser.isSuperuser()) && (userIdentifier != null) && (!userIdentifier.isEmpty())) { + searchUser = getUserFromIdentifier(userIdentifier); + if (searchUser != null) { + authUser = searchUser; + OTHER_USER = true; + } else { + return this.getJSONErrorString("No user found for: \"" + userIdentifier + "\"", null); } - - } else{ - return this.getJSONErrorString("Requires authentication. Please login.", "retrieveMyDataAsJsonString. User not found! Shouldn't be using this anyway"); } - + roleList = dataverseRoleService.findAll(); rolePermissionHelper = new DataverseRolePermissionHelper(roleList); @@ -462,7 +435,7 @@ public String retrieveMyDataAsJsonString(@QueryParam("dvobject_types") List queryStrings = new ArrayList<>(); for (DatasetFieldType dsfType : metadataFieldList) { if (dsfType.getSearchValue() != null && !dsfType.getSearchValue().equals("")) { - queryStrings.add(constructQuery(dsfType.getSolrField().getNameSearchable(), dsfType.getSearchValue())); + //CVoc fields return term URIs - add quotes around them to avoid solr breaking them into individual search words + queryStrings.add(constructQuery(dsfType.getSolrField().getNameSearchable(), dsfType.getSearchValue(), getCVocConf().containsKey(dsfType.getId()))); } else if (dsfType.getListValues() != null && !dsfType.getListValues().isEmpty()) { List listQueryStrings = new ArrayList<>(); for (String value : dsfType.getListValues()) { diff --git a/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java index f1812ebcae1..fe5160b1e0d 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/IndexServiceBean.java @@ -1497,6 +1497,7 @@ private List findAllLinkingDataverses(DvObject dvObject){ dataset = (Dataset) dvObject; linkingDataverses = dsLinkingService.findLinkingDataverses(dataset.getId()); ancestorList = dataset.getOwner().getOwners(); + ancestorList.add(dataset.getOwner()); //to show dataset in linking dv when parent dv is linked } if(dvObject.isInstanceofDataverse()){ dv = (Dataverse) dvObject; @@ -1661,6 +1662,11 @@ private List retrieveDVOPaths(DvObject dvo) { logger.info("failed to find dataverseSegments for dataversePaths for " + SearchFields.SUBTREE + ": " + ex); } List dataversePaths = getDataversePathsFromSegments(dataverseSegments); + if (dataversePaths.size() > 0 && dvo.isInstanceofDataverse()) { + // removing the dataverse's own id from the paths + // fixes bug where if my parent dv was linked my dv was shown as linked to myself + dataversePaths.remove(dataversePaths.size() - 1); + } /* add linking paths */ diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java b/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java index 1881a85e0c7..3e0b07397f5 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SearchServiceBean.java @@ -43,8 +43,8 @@ import jakarta.persistence.NoResultException; import org.apache.solr.client.solrj.SolrQuery; import org.apache.solr.client.solrj.SolrQuery.SortClause; +import org.apache.solr.client.solrj.impl.BaseHttpSolrClient.RemoteSolrException; import org.apache.solr.client.solrj.SolrServerException; -import org.apache.solr.client.solrj.impl.HttpSolrClient.RemoteSolrException; import org.apache.solr.client.solrj.response.FacetField; import org.apache.solr.client.solrj.response.QueryResponse; import org.apache.solr.client.solrj.response.RangeFacet; diff --git a/src/main/java/edu/harvard/iq/dataverse/search/SearchUtil.java b/src/main/java/edu/harvard/iq/dataverse/search/SearchUtil.java index 8a1045a842c..adcc5825766 100644 --- a/src/main/java/edu/harvard/iq/dataverse/search/SearchUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/search/SearchUtil.java @@ -117,6 +117,10 @@ public static String determineFinalQuery(String userSuppliedQuery) { } public static String constructQuery(String solrField, String userSuppliedQuery) { + return constructQuery(solrField, userSuppliedQuery, false); + } + + public static String constructQuery(String solrField, String userSuppliedQuery, boolean addQuotes) { StringBuilder queryBuilder = new StringBuilder(); String delimiter = "[\"]+"; @@ -134,7 +138,12 @@ public static String constructQuery(String solrField, String userSuppliedQuery) } else { StringTokenizer st = new StringTokenizer(userSuppliedQuery); while (st.hasMoreElements()) { - queryStrings.add(solrField + ":" + st.nextElement()); + String nextElement = (String) st.nextElement(); + //Entries such as URIs will get tokenized into individual words by solr unless they are in quotes + if(addQuotes) { + nextElement = "\"" + nextElement + "\""; + } + queryStrings.add(solrField + ":" + nextElement); } } } diff --git a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java index bc5a73cd958..ed3a161075b 100644 --- a/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java +++ b/src/main/java/edu/harvard/iq/dataverse/settings/JvmSettings.java @@ -44,6 +44,10 @@ public enum JvmSettings { FQDN(PREFIX, "fqdn"), SITE_URL(PREFIX, "siteUrl"), + // FILES SETTINGS + SCOPE_FILES(PREFIX, "files"), + FILES_DIRECTORY(SCOPE_FILES, "directory"), + // SOLR INDEX SETTINGS SCOPE_SOLR(PREFIX, "solr"), SOLR_HOST(SCOPE_SOLR, "host"), diff --git a/src/main/java/edu/harvard/iq/dataverse/util/DataSourceProducer.java b/src/main/java/edu/harvard/iq/dataverse/util/DataSourceProducer.java index 94e6d68dd43..62cd318706f 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/DataSourceProducer.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/DataSourceProducer.java @@ -16,9 +16,13 @@ // HINT: PGSimpleDataSource would work too, but as we use a connection pool, go with a javax.sql.ConnectionPoolDataSource // HINT: PGXADataSource is unnecessary (no distributed transactions used) and breaks ingest. className = "org.postgresql.ds.PGConnectionPoolDataSource", - user = "${MPCONFIG=dataverse.db.user}", + + // BEWARE: as this resource is created before defaults are read from META-INF/microprofile-config.properties, + // defaults must be provided in this Payara-proprietary manner. + user = "${MPCONFIG=dataverse.db.user:dataverse}", password = "${MPCONFIG=dataverse.db.password}", - url = "jdbc:postgresql://${MPCONFIG=dataverse.db.host}:${MPCONFIG=dataverse.db.port}/${MPCONFIG=dataverse.db.name}", + url = "jdbc:postgresql://${MPCONFIG=dataverse.db.host:localhost}:${MPCONFIG=dataverse.db.port:5432}/${MPCONFIG=dataverse.db.name:dataverse}?${MPCONFIG=dataverse.db.parameters:}", + // If we ever need to change these pool settings, we need to remove this class and create the resource // from web.xml. We can use MicroProfile Config in there for these values, impossible to do in the annotation. // @@ -30,18 +34,30 @@ maxPoolSize = 100, // "The number of seconds that a physical connection should remain unused in the pool before the connection is closed for a connection pool. " // Payara DataSourceDefinitionDeployer default value = 300 (seconds) - maxIdleTime = 300) -// It's possible to add additional properties like this... -// -//properties = { -// "fish.payara.log-jdbc-calls=true" -//}) -// -// ... but at this time we don't think we need any. The full list -// of properties can be found at https://docs.payara.fish/community/docs/5.2021.6/documentation/payara-server/jdbc/advanced-connection-pool-properties.html#full-list-of-properties -// -// All these properties cannot be configured via MPCONFIG as Payara doesn't support this (yet). To be enhanced. -// See also https://github.com/payara/Payara/issues/5024 + maxIdleTime = 300, + + // Set more options via MPCONFIG, including defaults where applicable. + // TODO: Future versions of Payara might support setting integer properties like pool size, + // idle times, etc in a Payara-propietary way. See https://github.com/payara/Payara/pull/5272 + properties = { + // The following options are documented here: + // https://docs.payara.fish/community/docs/documentation/payara-server/jdbc/advanced-connection-pool-properties.html + // VALIDATION + "fish.payara.is-connection-validation-required=${MPCONFIG=dataverse.db.is-connection-validation-required:false}", + "fish.payara.connection-validation-method=${MPCONFIG=dataverse.db.connection-validation-method:}", + "fish.payara.validation-table-name=${MPCONFIG=dataverse.db.validation-table-name:}", + "fish.payara.validation-classname=${MPCONFIG=dataverse.db.validation-classname:}", + "fish.payara.validate-atmost-once-period-in-seconds=${MPCONFIG=dataverse.db.validate-atmost-once-period-in-seconds:0}", + // LEAK DETECTION + "fish.payara.connection-leak-timeout-in-seconds=${MPCONFIG=dataverse.db.connection-leak-timeout-in-seconds:0}", + "fish.payara.connection-leak-reclaim=${MPCONFIG=dataverse.db.connection-leak-reclaim:false}", + "fish.payara.statement-leak-timeout-in-seconds=${MPCONFIG=dataverse.db.statement-leak-timeout-in-seconds:0}", + "fish.payara.statement-leak-reclaim=${MPCONFIG=dataverse.db.statement-leak-reclaim:false}", + // LOGGING, SLOWNESS, PERFORMANCE + "fish.payara.statement-timeout-in-seconds=${MPCONFIG=dataverse.db.statement-timeout-in-seconds:-1}", + "fish.payara.slow-query-threshold-in-seconds=${MPCONFIG=dataverse.db.slow-query-threshold-in-seconds:-1}", + "fish.payara.log-jdbc-calls=${MPCONFIG=dataverse.db.log-jdbc-calls:false}" + }) public class DataSourceProducer { @Resource(lookup = "java:app/jdbc/dataverse") diff --git a/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java index f1a4c754a24..5b1a029b293 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/FileUtil.java @@ -40,6 +40,7 @@ import edu.harvard.iq.dataverse.ingest.IngestServiceShapefileHelper; import edu.harvard.iq.dataverse.ingest.IngestableDataChecker; import edu.harvard.iq.dataverse.license.License; +import edu.harvard.iq.dataverse.settings.JvmSettings; import edu.harvard.iq.dataverse.util.file.BagItFileHandler; import edu.harvard.iq.dataverse.util.file.CreateDataFileResult; import edu.harvard.iq.dataverse.util.file.BagItFileHandlerFactory; @@ -1436,11 +1437,8 @@ public static boolean canIngestAsTabular(String mimeType) { } public static String getFilesTempDirectory() { - String filesRootDirectory = System.getProperty("dataverse.files.directory"); - if (filesRootDirectory == null || filesRootDirectory.equals("")) { - filesRootDirectory = "/tmp/files"; - } - + + String filesRootDirectory = JvmSettings.FILES_DIRECTORY.lookup(); String filesTempDirectory = filesRootDirectory + "/temp"; if (!Files.exists(Paths.get(filesTempDirectory))) { @@ -2142,7 +2140,9 @@ public static String jsonArrayOfObjectsToCSV(JsonArray jsonArray, String... head JsonObject jo = (JsonObject) jv; String[] values = new String[headers.length]; for (int i = 0; i < headers.length; i++) { - values[i] = jo.get(headers[i]).toString(); + if(jo.containsKey(headers[i])) { + values[i] = jo.get(headers[i]).toString(); + } } csvSB.append("\n").append(String.join(",", values)); }); diff --git a/src/main/java/edu/harvard/iq/dataverse/util/PersonOrOrgUtil.java b/src/main/java/edu/harvard/iq/dataverse/util/PersonOrOrgUtil.java new file mode 100644 index 00000000000..b38097091ad --- /dev/null +++ b/src/main/java/edu/harvard/iq/dataverse/util/PersonOrOrgUtil.java @@ -0,0 +1,155 @@ +package edu.harvard.iq.dataverse.util; + +import java.util.ArrayList; +import java.util.List; +import java.util.logging.Logger; + +import jakarta.json.JsonArray; +import jakarta.json.JsonObject; +import jakarta.json.JsonObjectBuilder; +import jakarta.json.JsonString; + +import edu.harvard.iq.dataverse.export.openaire.Cleanup; +import edu.harvard.iq.dataverse.export.openaire.FirstNames; +import edu.harvard.iq.dataverse.export.openaire.Organizations; +import edu.harvard.iq.dataverse.util.json.JsonUtil; +import edu.harvard.iq.dataverse.util.json.NullSafeJsonBuilder; + +/** + * + * @author qqmyers + * + * Adapted from earlier code in OpenAireExportUtil + * + * Implements an algorithm derived from code at DataCite to determine + * whether a name is that of a Person or Organization and, if the + * former, to pull out the given and family names. + * + * Adds parameters that can improve accuracy: + * + * * e.g. for curated repositories, allowing the code to assume that all + * Person entries are in , order. + * + * * allow local configuration of specific words/phrases that will + * automatically categorize one-off cases that the algorithm would + * otherwise mis-categorize. For example, the code appears to not + * recognize names ending in "Project" as an Organization. + * + */ + +public class PersonOrOrgUtil { + + private static final Logger logger = Logger.getLogger(PersonOrOrgUtil.class.getCanonicalName()); + + static boolean assumeCommaInPersonName = false; + static List orgPhrases; + + static { + setAssumeCommaInPersonName(Boolean.parseBoolean(System.getProperty("dataverse.personOrOrg.assumeCommaInPersonName", "false"))); + setOrgPhraseArray(System.getProperty("dataverse.personOrOrg.orgPhraseArray", null)); + } + + /** + * This method tries to determine if a name belongs to a person or an + * organization and, if it is a person, what the given and family names are. The + * core algorithm is adapted from a Datacite algorithm, see + * https://github.com/IQSS/dataverse/issues/2243#issuecomment-358615313 + * + * @param name + * - the name to test + * @param organizationIfTied + * - if a given name isn't found, should the name be assumed to be + * from an organization. This could be a generic true/false or + * information from some non-name aspect of the entity, e.g. which + * field is in use, or whether a .edu email exists, etc. + * @param isPerson + * - if this is known to be a person due to other info (i.e. they + * have an ORCID). In this case the algorithm is just looking for + * given/family names. + * @return + */ + public static JsonObject getPersonOrOrganization(String name, boolean organizationIfTied, boolean isPerson) { + name = Cleanup.normalize(name); + + String givenName = null; + String familyName = null; + + boolean isOrganization = !isPerson && Organizations.getInstance().isOrganization(name); + if (!isOrganization) { + for (String phrase : orgPhrases) { + if (name.contains(phrase)) { + isOrganization = true; + break; + } + } + } + if (name.contains(",")) { + givenName = FirstNames.getInstance().getFirstName(name); + // contributorName=, + if (givenName != null && !isOrganization) { + // givenName ok + isOrganization = false; + // contributor_map.put("nameType", "Personal"); + if (!name.replaceFirst(",", "").contains(",")) { + // contributorName=, + String[] fullName = name.split(", "); + givenName = fullName[1]; + familyName = fullName[0]; + } + } else if (isOrganization || organizationIfTied) { + isOrganization = true; + givenName = null; + } + + } else { + if (assumeCommaInPersonName && !isPerson) { + isOrganization = true; + } else { + givenName = FirstNames.getInstance().getFirstName(name); + + if (givenName != null && !isOrganization) { + isOrganization = false; + if (givenName.length() + 1 < name.length()) { + familyName = name.substring(givenName.length() + 1); + } + } else { + // default + if (isOrganization || organizationIfTied) { + isOrganization = true; + givenName=null; + } + } + } + } + JsonObjectBuilder job = new NullSafeJsonBuilder(); + job.add("fullName", name); + job.add("givenName", givenName); + job.add("familyName", familyName); + job.add("isPerson", !isOrganization); + return job.build(); + + } + + // Public for testing + public static void setOrgPhraseArray(String phraseArray) { + orgPhrases = new ArrayList(); + if (!StringUtil.isEmpty(phraseArray)) { + try { + JsonArray phrases = JsonUtil.getJsonArray(phraseArray); + phrases.forEach(val -> { + JsonString strVal = (JsonString) val; + orgPhrases.add(strVal.getString()); + }); + } catch (Exception e) { + logger.warning("Could not parse Org phrase list"); + } + } + + } + + // Public for testing + public static void setAssumeCommaInPersonName(boolean assume) { + assumeCommaInPersonName = assume; + } + +} diff --git a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java index b9d423bc71e..cece2f19865 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/SystemConfig.java @@ -58,11 +58,6 @@ public class SystemConfig { public static final String DATAVERSE_PATH = "/dataverse/"; - /** - * A JVM option for where files are stored on the file system. - */ - public static final String FILES_DIRECTORY = "dataverse.files.directory"; - /** * Some installations may not want download URLs to their files to be * available in Schema.org JSON-LD output. diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java index 138bb6dc65b..6abcbe6c788 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonParser.java @@ -908,6 +908,7 @@ public String parseHarvestingClient(JsonObject obj, HarvestingClient harvestingC harvestingClient.setArchiveDescription(obj.getString("archiveDescription", null)); harvestingClient.setMetadataPrefix(obj.getString("metadataFormat",null)); harvestingClient.setHarvestingSet(obj.getString("set",null)); + harvestingClient.setCustomHttpHeaders(obj.getString("customHeaders", null)); return dataverseAlias; } diff --git a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java index ecfd118845f..2bb605bc3d8 100644 --- a/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java +++ b/src/main/java/edu/harvard/iq/dataverse/util/json/JsonPrinter.java @@ -37,6 +37,7 @@ import edu.harvard.iq.dataverse.dataset.DatasetUtil; import edu.harvard.iq.dataverse.license.License; import edu.harvard.iq.dataverse.globus.FileDetailsHolder; +import edu.harvard.iq.dataverse.harvest.client.HarvestingClient; import edu.harvard.iq.dataverse.privateurl.PrivateUrl; import edu.harvard.iq.dataverse.settings.SettingsServiceBean; import edu.harvard.iq.dataverse.util.DatasetFieldWalker; @@ -580,6 +581,7 @@ public static JsonObjectBuilder json(FileMetadata fmd) { // in a sense that there's no longer the category field in the // fileMetadata object; but there are now multiple, oneToMany file // categories - and we probably need to export them too!) -- L.A. 4.5 + // DONE: catgegories by name .add("description", fmd.getDescription()) .add("label", fmd.getLabel()) // "label" is the filename .add("restricted", fmd.isRestricted()) @@ -617,13 +619,13 @@ public static JsonObjectBuilder json(DataFile df, FileMetadata fileMetadata) { // (TODO...? L.A. 4.5, Aug 7 2016) String fileName = null; - if (fileMetadata != null) { - fileName = fileMetadata.getLabel(); - } else if (df.getFileMetadata() != null) { + if (fileMetadata == null){ // Note that this may not necessarily grab the file metadata from the // version *you want*! (L.A.) - fileName = df.getFileMetadata().getLabel(); + fileMetadata = df.getFileMetadata(); } + + fileName = fileMetadata.getLabel(); String pidURL = ""; @@ -640,7 +642,8 @@ public static JsonObjectBuilder json(DataFile df, FileMetadata fileMetadata) { .add("filename", fileName) .add("contentType", df.getContentType()) .add("filesize", df.getFilesize()) - .add("description", df.getDescription()) + .add("description", fileMetadata.getDescription()) + .add("categories", getFileCategories(fileMetadata)) .add("embargo", embargo) //.add("released", df.isReleased()) //.add("restricted", df.isRestricted()) @@ -667,6 +670,32 @@ public static JsonObjectBuilder json(DataFile df, FileMetadata fileMetadata) { ; } + public static JsonObjectBuilder json(HarvestingClient harvestingClient) { + if (harvestingClient == null) { + return null; + } + + return jsonObjectBuilder().add("nickName", harvestingClient.getName()). + add("dataverseAlias", harvestingClient.getDataverse().getAlias()). + add("type", harvestingClient.getHarvestType()). + add("style", harvestingClient.getHarvestStyle()). + add("harvestUrl", harvestingClient.getHarvestingUrl()). + add("archiveUrl", harvestingClient.getArchiveUrl()). + add("archiveDescription", harvestingClient.getArchiveDescription()). + add("metadataFormat", harvestingClient.getMetadataPrefix()). + add("set", harvestingClient.getHarvestingSet()). + add("schedule", harvestingClient.isScheduled() ? harvestingClient.getScheduleDescription() : "none"). + add("status", harvestingClient.isHarvestingNow() ? "inProgress" : "inActive"). + add("customHeaders", harvestingClient.getCustomHttpHeaders()). + add("lastHarvest", harvestingClient.getLastHarvestTime() == null ? null : harvestingClient.getLastHarvestTime().toString()). + add("lastResult", harvestingClient.getLastResult()). + add("lastSuccessful", harvestingClient.getLastSuccessfulHarvestTime() == null ? null : harvestingClient.getLastSuccessfulHarvestTime().toString()). + add("lastNonEmpty", harvestingClient.getLastNonEmptyHarvestTime() == null ? null : harvestingClient.getLastNonEmptyHarvestTime().toString()). + add("lastDatasetsHarvested", harvestingClient.getLastHarvestedDatasetCount()). // == null ? "N/A" : harvestingClient.getLastHarvestedDatasetCount().toString()). + add("lastDatasetsDeleted", harvestingClient.getLastDeletedDatasetCount()). // == null ? "N/A" : harvestingClient.getLastDeletedDatasetCount().toString()). + add("lastDatasetsFailed", harvestingClient.getLastFailedDatasetCount()); // == null ? "N/A" : harvestingClient.getLastFailedDatasetCount().toString()); + } + public static String format(Date d) { return (d == null) ? null : Util.getDateTimeFormat().format(d); } @@ -703,7 +732,7 @@ public static JsonArrayBuilder getTabularFileTags(DataFile df) { } return tabularTags; } - + private static class DatasetFieldsToJson implements DatasetFieldWalker.Listener { Deque objectStack = new LinkedList<>(); diff --git a/src/main/java/propertyFiles/Bundle.properties b/src/main/java/propertyFiles/Bundle.properties index 62531d32bb2..45807dc7cde 100644 --- a/src/main/java/propertyFiles/Bundle.properties +++ b/src/main/java/propertyFiles/Bundle.properties @@ -73,7 +73,7 @@ delete=Delete copyClipboard=Copy to Clipboard truncateMoreBtn=Read full {0} [+] truncateMoreTip=Click to read the full {0}. -truncateLessBtn=Collapse {0} [+] +truncateLessBtn=Collapse {0} [-] truncateLessTip=Click to collapse the {0}. yes=Yes no=No @@ -232,12 +232,12 @@ notification.access.revoked.datafile=You have been removed from a role in {0}. notification.checksumfail=One or more files in your upload failed checksum validation for dataset {1}. Please re-run the upload script. If the problem persists, please contact support. notification.ingest.completed=Your Dataset {2} has one or more tabular files that completed the tabular ingest process. These files will be available for download in their original formats and other formats for enhanced archival purposes after you publish the dataset. The archival .tab files are displayed in the file table. Please see the guides for more information about ingest and support for tabular files. notification.ingest.completedwitherrors=Your Dataset {2} has one or more tabular files that have been uploaded successfully but are not supported for tabular ingest. After you publish the dataset, these files will not have additional archival features. Please see the guides for more information about ingest and support for tabular files.

Files with incomplete ingest:{5} -notification.mail.import.filesystem=Globus transfer to Dataset {2} ({0}/dataset.xhtml?persistentId={1}) was successful. File(s) have been uploaded and verified. +notification.mail.import.filesystem=Dataset {2} ({0}/dataset.xhtml?persistentId={1}) has been successfully uploaded and verified. notification.mail.globus.upload.completed=Globus transfer to Dataset {2} was successful. File(s) have been uploaded and verified.

{3}
notification.mail.globus.download.completed=Globus transfer of file(s) from the dataset {2} was successful.

{3}
notification.mail.globus.upload.completedWithErrors=Globus transfer to Dataset {2} is complete with errors.

{3}
notification.mail.globus.download.completedWithErrors=Globus transfer from the dataset {2} is complete with errors.

{3}
-notification.import.filesystem=Globus transfer to Dataset {1} was successful. File(s) have been uploaded and verified. +notification.import.filesystem=Dataset {1} has been successfully uploaded and verified. notification.globus.upload.completed=Globus transfer to Dataset {1} was successful. File(s) have been uploaded and verified. notification.globus.download.completed=Globus transfer from the dataset {1} was successful. notification.globus.upload.completedWithErrors=Globus transfer to Dataset {1} is complete with errors. @@ -538,6 +538,10 @@ harvestclients.newClientDialog.nickname.helptext=Consists of letters, digits, un harvestclients.newClientDialog.nickname.required=Client nickname cannot be empty! harvestclients.newClientDialog.nickname.invalid=Client nickname can contain only letters, digits, underscores (_) and dashes (-); and must be at most 30 characters. harvestclients.newClientDialog.nickname.alreadyused=This nickname is already used. +harvestclients.newClientDialog.customHeader=Custom HTTP Header +harvestclients.newClientDialog.customHeader.helptext=(Optional) Custom HTTP header to add to requests, if required by this OAI server. +harvestclients.newClientDialog.customHeader.watermark=Enter an http header, as in header-name: header-value +harvestclients.newClientDialog.customHeader.invalid=Client header name can only contain letters, digits, underscores (_) and dashes (-); the entire header string must be in the form of "header-name: header-value" harvestclients.newClientDialog.type=Server Protocol harvestclients.newClientDialog.type.helptext=Only the OAI server protocol is currently supported. harvestclients.newClientDialog.type.OAI=OAI @@ -1523,7 +1527,7 @@ dataset.subjectDisplay.title=Subject dataset.contact.tip=Use email button above to contact. dataset.asterisk.tip=Asterisks indicate required fields dataset.message.uploadFiles.label=Upload Dataset Files -dataset.message.uploadFilesSingle.message=All file types are supported for upload and download in their original format. If you are uploading Excel, CSV, TSV, RData, Stata, or SPSS files, see the guides for tabular support and limitations. +dataset.message.uploadFilesSingle.message=All file types are supported for upload and download in their original format. If you are uploading Excel, CSV, TSV, RData, Stata, or SPSS files, see the guides for tabular support and limitations. dataset.message.uploadFilesMultiple.message=Multiple file upload/download methods are available for this dataset. Once you upload a file using one of these methods, your choice will be locked in for this dataset. dataset.message.editMetadata.label=Edit Dataset Metadata dataset.message.editMetadata.message=Add more metadata about this dataset to help others easily find it. @@ -2011,6 +2015,7 @@ file.remotelyStored=This file is stored remotely - click for more info file.auxfiles.download.header=Download Auxiliary Files # These types correspond to the AuxiliaryFile.Type enum. file.auxfiles.types.DP=Differentially Private Statistics +file.auxfiles.types.NcML=XML from NetCDF/HDF5 (NcML) # Add more types here file.auxfiles.unspecifiedTypes=Other Auxiliary Files @@ -2551,6 +2556,7 @@ admin.api.deleteUser.success=Authenticated User {0} deleted. #Files.java files.api.metadata.update.duplicateFile=Filename already exists at {0} +files.api.no.draft=No draft available for this file #Datasets.java datasets.api.updatePIDMetadata.failure.dataset.must.be.released=Modify Registration Metadata must be run on a published dataset. diff --git a/src/main/java/propertyFiles/MimeTypeDetectionByFileExtension.properties b/src/main/java/propertyFiles/MimeTypeDetectionByFileExtension.properties index c93bb56151f..97b2eed111c 100644 --- a/src/main/java/propertyFiles/MimeTypeDetectionByFileExtension.properties +++ b/src/main/java/propertyFiles/MimeTypeDetectionByFileExtension.properties @@ -3,6 +3,7 @@ ado=application/x-stata-ado dbf=application/dbf dcm=application/dicom docx=application/vnd.openxmlformats-officedocument.wordprocessingml.document +eln=application/vnd.eln+zip emf=application/x-emf geojson=application/geo+json h5=application/x-h5 diff --git a/src/main/java/propertyFiles/MimeTypeDisplay.properties b/src/main/java/propertyFiles/MimeTypeDisplay.properties index 928419c0405..295ac226fa1 100644 --- a/src/main/java/propertyFiles/MimeTypeDisplay.properties +++ b/src/main/java/propertyFiles/MimeTypeDisplay.properties @@ -169,6 +169,7 @@ application/x-7z-compressed=7Z Archive application/x-xz=XZ Archive application/warc=Web Archive application/x-iso9660-image=Optical Disc Image +application/vnd.eln+zip=ELN Archive # Image image/gif=GIF Image image/jpeg=JPEG Image diff --git a/src/main/java/propertyFiles/MimeTypeFacets.properties b/src/main/java/propertyFiles/MimeTypeFacets.properties index 2cac63a7ad0..aaab66f20ae 100644 --- a/src/main/java/propertyFiles/MimeTypeFacets.properties +++ b/src/main/java/propertyFiles/MimeTypeFacets.properties @@ -170,6 +170,7 @@ application/x-7z-compressed=Archive application/x-xz=Archive application/warc=Archive application/x-iso9660-image=Archive +application/vnd.eln+zip=Archive # Image image/gif=Image image/jpeg=Image @@ -224,4 +225,4 @@ text/xml-graphml=Network Data # Other application/octet-stream=Unknown # Dataverse-specific -application/vnd.dataverse.file-package=Data \ No newline at end of file +application/vnd.dataverse.file-package=Data diff --git a/src/main/java/propertyFiles/astrophysics.properties b/src/main/java/propertyFiles/astrophysics.properties index a49b8b66510..6e04bac590f 100644 --- a/src/main/java/propertyFiles/astrophysics.properties +++ b/src/main/java/propertyFiles/astrophysics.properties @@ -50,9 +50,9 @@ datasetfieldtype.coverage.SkyFraction.description=The fraction of the sky repres datasetfieldtype.coverage.Polarization.description=The polarization coverage datasetfieldtype.redshiftType.description=RedshiftType string C "Redshift"; or "Optical" or "Radio" definitions of Doppler velocity used in the data object. datasetfieldtype.resolution.Redshift.description=The resolution in redshift (unitless) or Doppler velocity (km/s) in the data object. -datasetfieldtype.coverage.RedshiftValue.description=The value of the redshift (unitless) or Doppler velocity (km/s in the data object. -datasetfieldtype.coverage.Redshift.MinimumValue.description=The minimum value of the redshift (unitless) or Doppler velocity (km/s in the data object. -datasetfieldtype.coverage.Redshift.MaximumValue.description=The maximum value of the redshift (unitless) or Doppler velocity (km/s in the data object. +datasetfieldtype.coverage.RedshiftValue.description=The value of the redshift (unitless) or Doppler velocity (km/s) in the data object. +datasetfieldtype.coverage.Redshift.MinimumValue.description=The minimum value of the redshift (unitless) or Doppler velocity (km/s) in the data object. +datasetfieldtype.coverage.Redshift.MaximumValue.description=The maximum value of the redshift (unitless) or Doppler velocity (km/s) in the data object. datasetfieldtype.astroType.watermark= datasetfieldtype.astroFacility.watermark= datasetfieldtype.astroInstrument.watermark= @@ -102,4 +102,4 @@ controlledvocabulary.astroType.observation=Observation controlledvocabulary.astroType.object=Object controlledvocabulary.astroType.value=Value controlledvocabulary.astroType.valuepair=ValuePair -controlledvocabulary.astroType.survey=Survey \ No newline at end of file +controlledvocabulary.astroType.survey=Survey diff --git a/src/main/resources/META-INF/microprofile-config.properties b/src/main/resources/META-INF/microprofile-config.properties index cde43fbff91..58592775a98 100644 --- a/src/main/resources/META-INF/microprofile-config.properties +++ b/src/main/resources/META-INF/microprofile-config.properties @@ -8,6 +8,9 @@ dataverse.build= %ct.dataverse.fqdn=localhost %ct.dataverse.siteUrl=http://${dataverse.fqdn}:8080 +# FILES +dataverse.files.directory=/tmp/dataverse + # SEARCH INDEX dataverse.solr.host=localhost # Activating mp config profile -Dmp.config.profile=ct changes default to "solr" as DNS name diff --git a/src/main/resources/db/migration/V5.13.0.1__8671-sorting_licenses.sql b/src/main/resources/db/migration/V5.12.1.1__8671-sorting_licenses.sql similarity index 100% rename from src/main/resources/db/migration/V5.13.0.1__8671-sorting_licenses.sql rename to src/main/resources/db/migration/V5.12.1.1__8671-sorting_licenses.sql diff --git a/src/main/resources/db/migration/V5.13.0.2__7715-signed-urls-for-tools.sql b/src/main/resources/db/migration/V5.12.1.2__7715-signed-urls-for-tools.sql similarity index 100% rename from src/main/resources/db/migration/V5.13.0.2__7715-signed-urls-for-tools.sql rename to src/main/resources/db/migration/V5.12.1.2__7715-signed-urls-for-tools.sql diff --git a/src/main/resources/db/migration/V5.13.0.3__8840-improve-guestbook-estimates.sql b/src/main/resources/db/migration/V5.12.1.3__8840-improve-guestbook-estimates.sql similarity index 100% rename from src/main/resources/db/migration/V5.13.0.3__8840-improve-guestbook-estimates.sql rename to src/main/resources/db/migration/V5.12.1.3__8840-improve-guestbook-estimates.sql diff --git a/src/main/resources/db/migration/V5.12.1.4__9153-extract-metadata.sql b/src/main/resources/db/migration/V5.12.1.4__9153-extract-metadata.sql new file mode 100644 index 00000000000..48230d21032 --- /dev/null +++ b/src/main/resources/db/migration/V5.12.1.4__9153-extract-metadata.sql @@ -0,0 +1 @@ +ALTER TABLE externaltool ADD COLUMN IF NOT EXISTS requirements TEXT; diff --git a/src/main/resources/db/migration/V5.12.1.5__9231_custom_headers_oai_requests.sql b/src/main/resources/db/migration/V5.12.1.5__9231_custom_headers_oai_requests.sql new file mode 100644 index 00000000000..fe6d717b2a3 --- /dev/null +++ b/src/main/resources/db/migration/V5.12.1.5__9231_custom_headers_oai_requests.sql @@ -0,0 +1 @@ +ALTER TABLE harvestingclient ADD COLUMN IF NOT EXISTS customhttpheaders TEXT; diff --git a/src/main/webapp/dataset-license-terms.xhtml b/src/main/webapp/dataset-license-terms.xhtml index 3f062ec5ca0..86e52092622 100644 --- a/src/main/webapp/dataset-license-terms.xhtml +++ b/src/main/webapp/dataset-license-terms.xhtml @@ -237,11 +237,12 @@ -
+ or !empty termsOfUseAndAccess.studyCompletion}">
  diff --git a/src/main/webapp/dataset.xhtml b/src/main/webapp/dataset.xhtml index 35753374dbb..6b91f815d9a 100644 --- a/src/main/webapp/dataset.xhtml +++ b/src/main/webapp/dataset.xhtml @@ -1790,6 +1790,7 @@ +