From b0b65722c41b95e9c60ea50c190cc88dd6235371 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 12 Sep 2025 14:22:32 -0400 Subject: [PATCH 01/28] start 6.8 release notes #11674 --- doc/release-notes/6.8-release-notes.md | 100 +++++++++++++++++++++++++ 1 file changed, 100 insertions(+) create mode 100644 doc/release-notes/6.8-release-notes.md diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md new file mode 100644 index 00000000000..24e3af40dd1 --- /dev/null +++ b/doc/release-notes/6.8-release-notes.md @@ -0,0 +1,100 @@ +# Dataverse 6.8 + +Please note: To read these instructions in full, please go to https://github.com/IQSS/dataverse/releases/tag/v6.8 rather than the [list of releases](https://github.com/IQSS/dataverse/releases), which will cut them off. + +This release brings new features, enhancements, and bug fixes to Dataverse. Thank you to all of the community members who contributed code, suggestions, bug reports, and other assistance across the project! + +## Release Highlights + +Highlights for Dataverse 6.8 include: + +## Features Added + +## Bugs Fixed + +## API Updates + +## Security Updates + +This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. + +## End-Of-Life (EOL) Announcements + +## New Settings + +## Deprecated Settings + +## Removed Settings + +## Backward Incompatible Changes + +Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. + +## Complete List of Changes + +For the complete list of code changes in this release, see the [6.8 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.8+is%3Aclosed) in GitHub. + +## Getting Help + +For help with upgrading, installing, or general questions please see [getting help](https://guides.dataverse.org/en/latest/installation/intro.html#getting-help) in the Installation Guide. + +## Installation + +If this is a new installation, please follow our [Installation Guide](https://guides.dataverse.org/en/latest/installation/). Please don't be shy about [asking for help](https://guides.dataverse.org/en/latest/installation/intro.html#getting-help) if you need it! + +Once you are in production, we would be delighted to update our [map of Dataverse installations around the world](https://dataverse.org/installations) to include yours! Please [create an issue](https://github.com/IQSS/dataverse-installations/issues) or email us at support@dataverse.org to join the club! + +You are also very welcome to join the [Global Dataverse Community Consortium](https://www.gdcc.io/) (GDCC). + +## Upgrade Instructions + +Upgrading requires a maintenance window and downtime. Please plan accordingly, create backups of your database, etc. + +These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.6. + +0\. These instructions assume that you are upgrading from the immediate previous version. See [tags on GitHub](https://github.com/IQSS/dataverse/tags) for a list of versions. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version. + +If you are running Payara as a non-root user (and you should be!), **remember not to execute the commands below as root**. By default, Payara runs as the `dataverse` user. In the commands below, we use sudo to run the commands as a non-root user. + +Also, we assume that Payara 6 is installed in `/usr/local/payara6`. If not, adjust as needed. + +```shell +export PAYARA=/usr/local/payara6 +``` + +(or `setenv PAYARA /usr/local/payara6` if you are using a `csh`-like shell) + +1\. List deployed applications + +```shell +$PAYARA/bin/asadmin list-applications +``` + +2\. Undeploy the previous version (should match "list-applications" above) + +```shell +$PAYARA/bin/asadmin undeploy dataverse-6.7.1 +``` + +3\. Download and deploy this version + +```shell +wget https://github.com/IQSS/dataverse/releases/download/v6.8/dataverse-6.8.war +$PAYARA/bin/asadmin deploy dataverse-6.8.war +``` + +Note: if you have any trouble deploying, stop Payara, remove the following directories, start Payara, and try to deploy again. + +```shell +sudo service payara stop +sudo rm -rf $PAYARA/glassfish/domains/domain1/generated +sudo rm -rf $PAYARA/glassfish/domains/domain1/osgi-cache +sudo rm -rf $PAYARA/glassfish/domains/domain1/lib/databases +sudo service payara start +``` + +4\. For installations with internationalization or text customizations: + +Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs). + +If you have text customizations you can get the latest English files from . \ No newline at end of file From 3d72e1de27bd3f3c70bb823315ea6abb07796eb8 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 12 Sep 2025 14:55:13 -0400 Subject: [PATCH 02/28] datasetCount Solr field and edit dataset metadata API changes --- doc/release-notes/10190-dataset-count.md | 2 - .../11243-editmetadata-api-extension.md | 7 --- .../11392-edit-file-metadata-empty-values.md | 7 --- doc/release-notes/6.8-release-notes.md | 45 ++++++++++++++++++- 4 files changed, 44 insertions(+), 17 deletions(-) delete mode 100644 doc/release-notes/10190-dataset-count.md delete mode 100644 doc/release-notes/11243-editmetadata-api-extension.md delete mode 100644 doc/release-notes/11392-edit-file-metadata-empty-values.md diff --git a/doc/release-notes/10190-dataset-count.md b/doc/release-notes/10190-dataset-count.md deleted file mode 100644 index a3d3a052f87..00000000000 --- a/doc/release-notes/10190-dataset-count.md +++ /dev/null @@ -1,2 +0,0 @@ -The search index now includes datasetCount for each collection, counting published, linked, and harvested datasets. -Collections can be filtered using datasetCount (e.g., `datasetCount:[1000 TO *]`), and the value is returned in Dataverse search results via the Search API. \ No newline at end of file diff --git a/doc/release-notes/11243-editmetadata-api-extension.md b/doc/release-notes/11243-editmetadata-api-extension.md deleted file mode 100644 index 3666d8bc30a..00000000000 --- a/doc/release-notes/11243-editmetadata-api-extension.md +++ /dev/null @@ -1,7 +0,0 @@ -### Edit Dataset Metadata API extension - -- This endpoint now allows removing fields (by sending empty values), as long as they are not required by the dataset. -- New ``sourceLastUpdateTime`` optional query parameter, which prevents inconsistencies by managing updates that - may occur from other users while a dataset is being edited. - -NOTE: This release note was updated to conform to the refactoring of the validation as part of issue #11392 diff --git a/doc/release-notes/11392-edit-file-metadata-empty-values.md b/doc/release-notes/11392-edit-file-metadata-empty-values.md deleted file mode 100644 index 5839fa100af..00000000000 --- a/doc/release-notes/11392-edit-file-metadata-empty-values.md +++ /dev/null @@ -1,7 +0,0 @@ -### Edit File Metadata empty values should clear data - -Previously the API POST /files/{id}/metadata would ignore fields with empty values. Now the API updates the fields with the empty values essentially clearing the data. Missing fields will still be ignored. - -An optional query parameter (sourceLastUpdateTime) was added to ensure the metadata update doesn't overwrite stale data. - -See also [the guides](https://dataverse-guide--11359.org.readthedocs.build/en/11359/api/native-api.html#updating-file-metadata), #11392, and #11359. diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 24e3af40dd1..73ebefb8eb4 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -10,10 +10,20 @@ Highlights for Dataverse 6.8 include: ## Features Added +- The search index now includes datasetCount for each collection, counting published, linked, and harvested datasets. Collections can be filtered using datasetCount (e.g., `datasetCount:[1000 TO *]`), and the value is returned in Dataverse search results via the Search API. See #10190. + ## Bugs Fixed ## API Updates +### Edit File Metadata: Empty Values Clear Data + +Previously the API POST /files/{id}/metadata would ignore fields with empty values. Now the API updates the fields with the empty values essentially clearing the data. Missing fields will still be ignored. + +An optional query parameter (sourceLastUpdateTime) was added to ensure the metadata update doesn't overwrite stale data. + +See also [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#updating-file-metadata), #11392, #11439, and the [API Changelog](https://guides.dataverse.org/en/6.8/api/changelog.html) (also listed under Backward Incompatible Changes, below). + ## Security Updates This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. @@ -97,4 +107,37 @@ sudo service payara start Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs). -If you have text customizations you can get the latest English files from . \ No newline at end of file +If you have text customizations you can get the latest English files from . + +5\. Update Solr schema and reindex + +Due to changes in the Solr schema (the addition of field "datasetCount"), updating the Solr schema and reindexing is required. + +Download the updated `schema.xml` file: + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.8/conf/solr/schema.xml +cp schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf +``` + +5a\. For installations with additional metadata blocks or external controlled vocabulary scripts, update fields + +- Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.8/installation/prerequisites.html#solr-init-script)). + +- Run the `update-fields.sh` script that we supply, as in the example below (modify the command lines as needed to reflect the correct path of your Solr installation): + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.8/conf/solr/update-fields.sh +chmod +x update-fields.sh +curl "http://localhost:8080/api/admin/index/solr/schema" | ./update-fields.sh /usr/local/solr/solr-9.8.0/server/solr/collection1/conf/schema.xml +``` + +Note that Docker-based installations use a different directory: `solr/data/data/collection1/conf/schema.xml`. + +- Start Solr instance (usually `service solr start` depending on Solr/OS). + +6\. Reindex Solr + +```shell +curl http://localhost:8080/api/admin/index +``` \ No newline at end of file From 8a0f8835f35ddd83d35ca382314f25f8ea938012 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 12 Sep 2025 15:05:34 -0400 Subject: [PATCH 03/28] add shib incommon update --- .../11404-shibboleth-mdq-wayfinder.md | 44 ------------------- doc/release-notes/6.8-release-notes.md | 43 +++++++++++++++++- 2 files changed, 42 insertions(+), 45 deletions(-) delete mode 100644 doc/release-notes/11404-shibboleth-mdq-wayfinder.md diff --git a/doc/release-notes/11404-shibboleth-mdq-wayfinder.md b/doc/release-notes/11404-shibboleth-mdq-wayfinder.md deleted file mode 100644 index 2674cbd797c..00000000000 --- a/doc/release-notes/11404-shibboleth-mdq-wayfinder.md +++ /dev/null @@ -1,44 +0,0 @@ -### For Dataverse instances that use Shibboleth as members of InCommon federation - -Please note that most of the known Dataverse instances that support Shibboleth logins do so without being part of InCommon, and therefore are not affected. All such instances will be able to continue using the old login workflow without needing to make any configuration changes. - -For the relatively few instances using InCommon: Since InCommon discontinued their old-style federation metadata feed, a new Shibboleth implementation has been added to utilize the recommended replacements: the MDQ protocol and the WayFinder service. In order to continue using InCommon, such instances will need to modify their shibd configuration and their registration with Incommon, plus set a new feature flag. See the upgrade instructions for details. - - -### New Settings - -- dataverse.feature.shibboleth-use-wayfinder -- dataverse.feature.shibboleth-use-localhost - -### For the Upgrade Instruction: - -[(strip this from the real release note) this should be the very last of the optional upgrade steps; as it's been pointed out to me that there may not be any instances affected by this aside from Harvard and UNC, both of which have been active participants in the development of the underlying code and the upgrade process below. ... which kind of makes including them in the release note somewhat unnecessary (?). but I figure we should include them anyway, in case there's an instance out there we are not aware of. - L.A.] - - -If your instance is offering institutional Shibboleth logins as part of the InCommon federation, you must make some changes to your service configuration: - -Note that if your Dataverse instance is using Shibboleth outside of InCommon, your login workflow should continue working unchanged, so please skip this section. - -a. Configure your Service Provider (SP) in the InCommon Federation Manager to use WayFinder following [their instructions](https://spaces.at.internet2.edu/display/federation/how-to-configure-service-to-use-wayfinder). - -b. Reconfigure your locally-running `shibd` service to use WayFinder and the new MDQ metadata retrieval protocol. -Download and place the new [production signing key](https://spaces.at.internet2.edu/display/MDQ/production-mdq-signing-key) in `/etc/shibboleth` and name it `inc-md-cert-mdq.pem`. -Change the `SSO` and `MetadataProvider` sections of the `/etc/shibboleth/shibboleth2.xml` configuration file as follows: - -``` - - SAML2 SAML1 - -``` -and -``` - - - - -``` -See [How to configure a Shibboleth service provider (SP) to use MDQ](https://spaces.at.internet2.edu/display/MDQ/how-to-configure-shib-sp-to-use-mdq) for more information. - -c. Set the feature flag `dataverse.feature.shibboleth-use-wayfinder=true`. - diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 73ebefb8eb4..b9ce4877628 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -30,8 +30,17 @@ This release contains important security updates. If you are not receiving secur ## End-Of-Life (EOL) Announcements +### For Dataverse instances that Use Shibboleth as Members of the InCommon Federation + +Please note that most of the known Dataverse instances that support Shibboleth logins do so without being part of InCommon, and therefore are not affected. All such instances will be able to continue using the old login workflow without needing to make any configuration changes. + +For the relatively few instances using InCommon: Since InCommon discontinued their old-style federation metadata feed, a new Shibboleth implementation has been added to utilize the recommended replacements: the MDQ protocol and the WayFinder service. In order to continue using InCommon, such instances will need to modify their shibd configuration and their registration with Incommon, plus set a new feature flag. See the upgrade instructions below for details. See also #11404 and #11502. + ## New Settings +- dataverse.feature.shibboleth-use-wayfinder +- dataverse.feature.shibboleth-use-localhost + ## Deprecated Settings ## Removed Settings @@ -140,4 +149,36 @@ Note that Docker-based installations use a different directory: `solr/data/data/ ```shell curl http://localhost:8080/api/admin/index -``` \ No newline at end of file +``` + +7\. InCommon federation login update + +If your instance is offering institutional Shibboleth logins as part of the InCommon federation, you must make some changes to your service configuration: + +Note that if your Dataverse instance is using Shibboleth outside of InCommon, your login workflow should continue working unchanged, so please skip this section. + +a. Configure your Service Provider (SP) in the InCommon Federation Manager to use WayFinder following [their instructions](https://spaces.at.internet2.edu/display/federation/how-to-configure-service-to-use-wayfinder). + +b. Reconfigure your locally-running `shibd` service to use WayFinder and the new MDQ metadata retrieval protocol. +Download and place the new [production signing key](https://spaces.at.internet2.edu/display/MDQ/production-mdq-signing-key) in `/etc/shibboleth` and name it `inc-md-cert-mdq.pem`. +Change the `SSO` and `MetadataProvider` sections of the `/etc/shibboleth/shibboleth2.xml` configuration file as follows: + +``` + + SAML2 SAML1 + +``` + +and + +``` + + + + +``` + +See [How to configure a Shibboleth service provider (SP) to use MDQ](https://spaces.at.internet2.edu/display/MDQ/how-to-configure-shib-sp-to-use-mdq) for more information. + +c. Set the feature flag `dataverse.feature.shibboleth-use-wayfinder=true`. \ No newline at end of file From c8736cb02d8baa1e011afc6d3f85d800a3128e45 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 12 Sep 2025 15:18:54 -0400 Subject: [PATCH 04/28] dataset links and get customization files --- .../11448-api-endpoint-for-analytics-html.md | 5 ----- doc/release-notes/11492-list-dataset-links.md | 1 - doc/release-notes/6.8-release-notes.md | 8 ++++++++ 3 files changed, 8 insertions(+), 6 deletions(-) delete mode 100644 doc/release-notes/11448-api-endpoint-for-analytics-html.md delete mode 100644 doc/release-notes/11492-list-dataset-links.md diff --git a/doc/release-notes/11448-api-endpoint-for-analytics-html.md b/doc/release-notes/11448-api-endpoint-for-analytics-html.md deleted file mode 100644 index ac62a1b5257..00000000000 --- a/doc/release-notes/11448-api-endpoint-for-analytics-html.md +++ /dev/null @@ -1,5 +0,0 @@ -### Feature Request: API endpoint for analytics.html - -New API to get the analytics.html from settings for SPA (Also can be used to get homePage, header, footer, style, and logo) - -See also [the guides](https://dataverse-guide--11359.org.readthedocs.build/en/11359/installation/config.html#web-analytics-code), #11448. diff --git a/doc/release-notes/11492-list-dataset-links.md b/doc/release-notes/11492-list-dataset-links.md deleted file mode 100644 index 0a5a6f7a198..00000000000 --- a/doc/release-notes/11492-list-dataset-links.md +++ /dev/null @@ -1 +0,0 @@ -The [API for listing the collections a dataset has been linked to](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset) (`api/datasets/$linked-dataset-id/links`) is no longer restricted to superusers. For unpublished datasets, users need the "View Unpublished Dataset" permission to access the API. Unpublished collections in the list require the "View Unpublished Dataverse" permission; otherwise, they are hidden. \ No newline at end of file diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index b9ce4877628..b4e21f0d30f 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -24,6 +24,14 @@ An optional query parameter (sourceLastUpdateTime) was added to ensure the metad See also [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#updating-file-metadata), #11392, #11439, and the [API Changelog](https://guides.dataverse.org/en/6.8/api/changelog.html) (also listed under Backward Incompatible Changes, below). +### Listing Collections a Dataset Has Been Linked To + +The [API for listing the collections a dataset has been linked to](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset) (`api/datasets/$linked-dataset-id/links`) is no longer restricted to superusers. For unpublished datasets, users need the "View Unpublished Dataset" permission to access the API. Unpublished collections in the list require the "View Unpublished Dataverse" permission; otherwise, they are hidden. See #11492. + +### Get Customization File Contents API + +A new API has been added to get customization file contents: analytics, homepage, header, footer, style, and logo. See [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#get-customization-file-contents), #11448, and #11467. + ## Security Updates This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. From cdbfe3e3d5e1d2b8232eb412c4e1962d0e21843b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 12 Sep 2025 15:38:29 -0400 Subject: [PATCH 05/28] mpconfig --- doc/release-notes/11485-mpconfig-personororg.md | 7 ------- doc/release-notes/6.8-release-notes.md | 14 +++++++++++++- 2 files changed, 13 insertions(+), 8 deletions(-) delete mode 100644 doc/release-notes/11485-mpconfig-personororg.md diff --git a/doc/release-notes/11485-mpconfig-personororg.md b/doc/release-notes/11485-mpconfig-personororg.md deleted file mode 100644 index c30ef3829c1..00000000000 --- a/doc/release-notes/11485-mpconfig-personororg.md +++ /dev/null @@ -1,7 +0,0 @@ -The settings `dataverse.personOrOrg.assumeCommaInPersonName` and `dataverse.personOrOrg.orgPhraseArray` now support configuration via MicroProfile Config. - -They have been renamed to `dataverse.person-or-org.assume-comma-in-person-name` and `dataverse.person-or-org.org-phrase-array` for consistency with naming conventions. - -In addition to the existing `asadmin` JVM option method, any [supported MicroProfile Config API source](https://docs.payara.fish/community/docs/Technical%20Documentation/MicroProfile/Config/Overview.html) can now be used to set their values. - -For backwards compatibility, `dataverse.personOrOrg.assumeCommaInPersonName` is still supported. However, `dataverse.personOrOrg.orgPhraseArray` is not, due to a change in the expected value format. `dataverse.person-or-org.org-phrase-array` now expects a comma-separated list of phrases as a value instead of a JsonArray of strings. Please update both the name and value format if using the old setting. \ No newline at end of file diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index b4e21f0d30f..fe6188b8dc5 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -48,6 +48,10 @@ For the relatively few instances using InCommon: Since InCommon discontinued the - dataverse.feature.shibboleth-use-wayfinder - dataverse.feature.shibboleth-use-localhost +- dataverse.person-or-org.assume-comma-in-person-name +- dataverse.person-or-org.org-phrase-array + +The settings `dataverse.personOrOrg.assumeCommaInPersonName` and `dataverse.personOrOrg.orgPhraseArray` now support configuration via MicroProfile Config (MPConfig). (Previously, they were only configurable as JVM options.) Their MPConfig names are `dataverse.person-or-org.assume-comma-in-person-name` and `dataverse.person-or-org.org-phrase-array`, respectively, for consistency with naming conventions. In addition to the existing `asadmin` JVM option method, any [supported MicroProfile Config API source](https://docs.payara.fish/community/docs/Technical%20Documentation/MicroProfile/Config/Overview.html) can now be used to set their values (as with all other MPConfig settings). For backwards compatibility, `dataverse.personOrOrg.assumeCommaInPersonName` is still supported. However, `dataverse.personOrOrg.orgPhraseArray` is not, due to a change in the expected value format, as mentioned under Backward Incompatible Changes, below. `dataverse.person-or-org.org-phrase-array` now expects a comma-separated list of phrases as a value instead of a JsonArray of strings. The upgrade instructions below indicate to update both the name and value format if using the old setting. See #11485. ## Deprecated Settings @@ -57,6 +61,10 @@ For the relatively few instances using InCommon: Since InCommon discontinued the Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/latest/api/changelog.html) for a list of backward-incompatible API changes. +### dataverse.personOrOrg.orgPhraseArray + +The setting `dataverse.personOrOrg.orgPhraseArray` has been renamed to `dataverse.person-or-org.org-phrase-array` and now expects a comma-separated list of phrases as a value instead of a JsonArray of strings. + ## Complete List of Changes For the complete list of code changes in this release, see the [6.8 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.8+is%3Aclosed) in GitHub. @@ -159,7 +167,11 @@ Note that Docker-based installations use a different directory: `solr/data/data/ curl http://localhost:8080/api/admin/index ``` -7\. InCommon federation login update +7\. Update dataverse.personOrOrg.orgPhraseArray, if used. + +If you are using the `dataverse.personOrOrg.orgPhraseArray` setting, rename it to `dataverse.person-or-org.org-phrase-array` and replace the JSON array of strings with a comma-separated list. See also the ([docs](https://guides.dataverse.org/en/6.8/installation/config.html#dataverse-person-or-org-org-phrase-array)) for this settings and the New Settings and Backward Incompatible Changes sections above. + +8\. InCommon federation login update If your instance is offering institutional Shibboleth logins as part of the InCommon federation, you must make some changes to your service configuration: From bb9c01aa9b28c015fe864beff87c0d154dc676bc Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 12 Sep 2025 16:02:29 -0400 Subject: [PATCH 06/28] my data list and show collections --- ...llections-a-user-can-create-datasets-in.md | 5 ----- doc/release-notes/11558-show-collections.md | 12 ---------- doc/release-notes/6.8-release-notes.md | 22 +++++++++++++++++++ 3 files changed, 22 insertions(+), 17 deletions(-) delete mode 100644 doc/release-notes/11525-retrieve-collections-a-user-can-create-datasets-in.md delete mode 100644 doc/release-notes/11558-show-collections.md diff --git a/doc/release-notes/11525-retrieve-collections-a-user-can-create-datasets-in.md b/doc/release-notes/11525-retrieve-collections-a-user-can-create-datasets-in.md deleted file mode 100644 index 91a3a83efa5..00000000000 --- a/doc/release-notes/11525-retrieve-collections-a-user-can-create-datasets-in.md +++ /dev/null @@ -1,5 +0,0 @@ -### New API to retrieve a list of collections that an authenticated user can create a dataset in - -The API GET /api/mydata/retrieve/collectionList will return all the dataverse objects that the user can add to - -See also [the guides](https://guides.dataverse.org/en/latest/api/native-api.html#mydata) and #11525. diff --git a/doc/release-notes/11558-show-collections.md b/doc/release-notes/11558-show-collections.md deleted file mode 100644 index b204da62387..00000000000 --- a/doc/release-notes/11558-show-collections.md +++ /dev/null @@ -1,12 +0,0 @@ -The Search API now supports a `show_collections` parameter for dataset results. -When the parameter is set, each result includes a `collections` array showing the dataset’s parent and linked collections. Each entry includes `id`, `name`, and `alias`, for example: - -```json -"collections": [ - { - "id": 11, - "name": "My cool collection", - "alias": "dvcb50a190" - } -] -``` \ No newline at end of file diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index fe6188b8dc5..e16d788b14e 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -16,6 +16,28 @@ Highlights for Dataverse 6.8 include: ## API Updates +### MyData Collection List API + +The MyData Collection List API is used to get a list of the collections an authenticated user can create a Dataset in. Param userIdentifier={userName} is used by a superuser to get the collections for a specific user. + +See also [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#mydata-collection-list) #11525, and #11681. + +### Search API show_collections Parameter + +The Search API now supports a `show_collections` parameter for dataset results. When the parameter is set, each result includes a `collections` array showing the dataset's parent and linked collections. Each entry includes `id`, `name`, and `alias`, for example: + +```json +"collections": [ + { + "id": 42, + "name": "My cool collection", + "alias": "myCoolCollection" + } +] +``` + +See also [the guides](https://guides.dataverse.org/en/6.8/api/search.html) and #11558. + ### Edit File Metadata: Empty Values Clear Data Previously the API POST /files/{id}/metadata would ignore fields with empty values. Now the API updates the fields with the empty values essentially clearing the data. Missing fields will still be ignored. From fe757735a055f82264e6362c5c306219d7d105b1 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Fri, 12 Sep 2025 16:16:09 -0400 Subject: [PATCH 07/28] linking permission split off from publishing --- doc/release-notes/11534-link-permissions.md | 3 --- doc/release-notes/6.8-release-notes.md | 1 + 2 files changed, 1 insertion(+), 3 deletions(-) delete mode 100644 doc/release-notes/11534-link-permissions.md diff --git a/doc/release-notes/11534-link-permissions.md b/doc/release-notes/11534-link-permissions.md deleted file mode 100644 index 29251c1b7d9..00000000000 --- a/doc/release-notes/11534-link-permissions.md +++ /dev/null @@ -1,3 +0,0 @@ -Linking or unlinking a dataset or dataverse now requires the new "Link Dataset/Dataverse" permission. -Previously, this action was covered by the "Publish Dataset/Dataverse" permission. -Linking and publishing permissions can now be granted separately, allowing for more fine-grained access control. \ No newline at end of file diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index e16d788b14e..88de3efb32a 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -11,6 +11,7 @@ Highlights for Dataverse 6.8 include: ## Features Added - The search index now includes datasetCount for each collection, counting published, linked, and harvested datasets. Collections can be filtered using datasetCount (e.g., `datasetCount:[1000 TO *]`), and the value is returned in Dataverse search results via the Search API. See #10190. +- Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" or "LinkDataverse" permissions, respectively. Previously, this action was covered by the "PublishDataset" or "PublishDataverse" permission. Splitting linking into its own permission, separate from publishing, allows for more fine-grained access control, if you choose to implement custom roles that differ from the roles that Dataverse ships with. A (Flyway) database migration script will be run automatically such that all roles that have permission to publish will continue to have permission to link. See #11691. ## Bugs Fixed From 2141bb2264cbb155288012ac57762fc35da87fce Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 09:25:38 -0400 Subject: [PATCH 08/28] list various bugs --- doc/release-notes/11592-HandleParsing_fix.md | 1 - doc/release-notes/11629-CSLFix.md | 1 - doc/release-notes/11685-CurationStatus_fix.md | 6 ------ doc/release-notes/11722-configbaker-bc-missing.md | 1 - doc/release-notes/11776- index fix.md | 1 - doc/release-notes/6.8-release-notes.md | 6 ++++++ 6 files changed, 6 insertions(+), 10 deletions(-) delete mode 100644 doc/release-notes/11592-HandleParsing_fix.md delete mode 100644 doc/release-notes/11629-CSLFix.md delete mode 100644 doc/release-notes/11685-CurationStatus_fix.md delete mode 100644 doc/release-notes/11722-configbaker-bc-missing.md delete mode 100644 doc/release-notes/11776- index fix.md diff --git a/doc/release-notes/11592-HandleParsing_fix.md b/doc/release-notes/11592-HandleParsing_fix.md deleted file mode 100644 index 087655d77c4..00000000000 --- a/doc/release-notes/11592-HandleParsing_fix.md +++ /dev/null @@ -1 +0,0 @@ -A bug introduced in v6.5 broken Handle parsing when using a lower-case shoulder. This is now fixed. \ No newline at end of file diff --git a/doc/release-notes/11629-CSLFix.md b/doc/release-notes/11629-CSLFix.md deleted file mode 100644 index 374a0dc78be..00000000000 --- a/doc/release-notes/11629-CSLFix.md +++ /dev/null @@ -1 +0,0 @@ -The styled citations available through the "View Styled Citations" menu were including extra characters, e.g. 'doi:' in the URL form of the PIDs in the citation. This is now fixed. \ No newline at end of file diff --git a/doc/release-notes/11685-CurationStatus_fix.md b/doc/release-notes/11685-CurationStatus_fix.md deleted file mode 100644 index 18bb6c8ad1d..00000000000 --- a/doc/release-notes/11685-CurationStatus_fix.md +++ /dev/null @@ -1,6 +0,0 @@ -The updates to support keeping the history of curation status labels added in #11268 -will incorrectly show curation statuses added prior to v6.7 as the current one, regardless of -whether newer statuses exist. This PR corrects the problem. - -(As a work-around for 6.7 admins can add createtime dates (must be prior to when 6.7 was installed) to the curationstatus table - for entries that have null createtimes. The code fix in this version properly handles null dates as indicating older/pre v6.7 curationstatuses.) \ No newline at end of file diff --git a/doc/release-notes/11722-configbaker-bc-missing.md b/doc/release-notes/11722-configbaker-bc-missing.md deleted file mode 100644 index 3a2074bbf44..00000000000 --- a/doc/release-notes/11722-configbaker-bc-missing.md +++ /dev/null @@ -1 +0,0 @@ -When following the container demo tutorial, it was not possible to update Solr fields after adding additional metadata blocks. This has been fixed. See #11722 and #11723 diff --git a/doc/release-notes/11776- index fix.md b/doc/release-notes/11776- index fix.md deleted file mode 100644 index 959b1ca3e95..00000000000 --- a/doc/release-notes/11776- index fix.md +++ /dev/null @@ -1 +0,0 @@ -A bug, introduced in v6.7, that caused files in draft versions that were added after the initial dataset version was published, has been fixed. \ No newline at end of file diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 88de3efb32a..ca1a21b2277 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -15,6 +15,12 @@ Highlights for Dataverse 6.8 include: ## Bugs Fixed +- A bug, introduced in v6.7, that caused files to not be indexed in draft versions that were added after the initial dataset version was published, has been fixed. See #11776 and #11779. +- The styled citations available through the "View Styled Citations" menu were including extra characters, e.g. 'doi:' in the URL form of the PIDs in the citation. This is now fixed. See #11629. +- The updates to support keeping the history of curation status labels added in #11268 will incorrectly show curation statuses added prior to v6.7 as the current one, regardless of whether newer statuses exist. This bug has been fixed. See #11685. (As a work-around for 6.7 admins can add createtime dates (must be prior to when 6.7 was installed) to the curationstatus table for entries that have null createtimes. The code fix in this version properly handles null dates as indicating older/pre v6.7 curationstatuses.) +- A bug introduced in v6.5 broken Handle parsing when using a lower-case shoulder. This is now fixed. See #11592. +- When following the container demo tutorial, it was not possible to update Solr fields after adding additional metadata blocks. This has been fixed. See #11722 and #11723. + ## API Updates ### MyData Collection List API From 127377271b187bf1b67c390b7dd36c639ec36da6 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 09:27:38 -0400 Subject: [PATCH 09/28] add dev update section with exporters feature --- doc/release-notes/6.8-release-notes.md | 6 ++++++ doc/release-notes/issue-10523.md | 6 ------ 2 files changed, 6 insertions(+), 6 deletions(-) delete mode 100644 doc/release-notes/issue-10523.md diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index ca1a21b2277..cb7dbfef017 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -65,6 +65,12 @@ A new API has been added to get customization file contents: analytics, homepage This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. +## Developer Updates + +### Writing External Exporters + +The getDatasetFileDetails data structure now contains "directoryLabel" (file path). See #10523 and #11618. + ## End-Of-Life (EOL) Announcements ### For Dataverse instances that Use Shibboleth as Members of the InCommon Federation diff --git a/doc/release-notes/issue-10523.md b/doc/release-notes/issue-10523.md deleted file mode 100644 index 3ce292646a7..00000000000 --- a/doc/release-notes/issue-10523.md +++ /dev/null @@ -1,6 +0,0 @@ - -## Developer Updates - -### Writing External Exporters - -The getDatasetFileDetails data structure now contains "directoryLabel" (file path). See #10523 and #11618. \ No newline at end of file From 2a84214bb4b19766f95a83773361259c24467cb6 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 09:37:54 -0400 Subject: [PATCH 10/28] OOD and PID failure tracking --- doc/release-notes/11601-pid-fail-tracking.md | 3 --- doc/release-notes/11768-ood.md | 3 --- doc/release-notes/6.8-release-notes.md | 8 +++++++- 3 files changed, 7 insertions(+), 7 deletions(-) delete mode 100644 doc/release-notes/11601-pid-fail-tracking.md delete mode 100644 doc/release-notes/11768-ood.md diff --git a/doc/release-notes/11601-pid-fail-tracking.md b/doc/release-notes/11601-pid-fail-tracking.md deleted file mode 100644 index 8a020c18c8e..00000000000 --- a/doc/release-notes/11601-pid-fail-tracking.md +++ /dev/null @@ -1,3 +0,0 @@ -This version of Dataverse includes a new feature flag - ``dataverse.feature.enable-pid-failure-log``. When set, Dataverse will log requests for dataset and file pages via persistentId that fail in monthly log files of the form PIDFailures_.log. These potentially indicate when someone has shared a draft PID without publishing or cases where a '.' or other character has been added to the PID, which may be of interest to site administrators. - -The new log files can be used in concert with the pidreport.py script at https://github.com/gdcc/dataverse-recipes/tree/main/python/pid_reports to generate and email monthly PID failure reports. \ No newline at end of file diff --git a/doc/release-notes/11768-ood.md b/doc/release-notes/11768-ood.md deleted file mode 100644 index 1ac699c0484..00000000000 --- a/doc/release-notes/11768-ood.md +++ /dev/null @@ -1,3 +0,0 @@ -### Open OnDemand Integration - -Open OnDemand, a web frontend to High Performance Computing (HPC) resources, has been integrated with Dataverse, allowing files to be downloaded from Dataverse installations around the world or from a specific dataset landing page if an external tool is enabled. Additionally, after computation is complete, datasets can be created in Dataverse from Open OnDemand and files can be uploaded. See the [docs](https://dataverse-guide--11769.org.readthedocs.build/en/11769/admin/integrations.html#open-ondemand), #11768 and #11769. diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index cb7dbfef017..196b70dfdcd 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -8,8 +8,13 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.8 include: +- Open OnDemand Integration +- Dignosing PID failures + ## Features Added +- Open OnDemand, a web frontend to High Performance Computing (HPC) resources, has been integrated with Dataverse, allowing files to be downloaded from Dataverse installations around the world or from a specific dataset landing page if an external tool is enabled. Additionally, after computation is complete, datasets can be created in Dataverse from Open OnDemand and files can be uploaded. See the [docs](https://guides.dataverse.org/en/6.8/admin/integrations.html#open-ondemand), #11768 and #11769. +- A new [feature flag](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags) called `enable-pid-failure-log` can be enabled to help diagnose PID failures. When set, Dataverse will log requests for dataset and file pages via persistentId that fail in monthly log files of the form `PIDFailures_.log`. These potentially indicate when someone has shared a draft PID without publishing or cases where a '.' or other character has been added to the PID, which may be of interest to site administrators. The new log files can be used in concert with the [pidreport.py](https://github.com/gdcc/dataverse-recipes/tree/main/python/pid_reports) script to generate and email monthly PID failure reports. See #11601. - The search index now includes datasetCount for each collection, counting published, linked, and harvested datasets. Collections can be filtered using datasetCount (e.g., `datasetCount:[1000 TO *]`), and the value is returned in Dataverse search results via the Search API. See #10190. - Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" or "LinkDataverse" permissions, respectively. Previously, this action was covered by the "PublishDataset" or "PublishDataverse" permission. Splitting linking into its own permission, separate from publishing, allows for more fine-grained access control, if you choose to implement custom roles that differ from the roles that Dataverse ships with. A (Flyway) database migration script will be run automatically such that all roles that have permission to publish will continue to have permission to link. See #11691. @@ -81,8 +86,9 @@ For the relatively few instances using InCommon: Since InCommon discontinued the ## New Settings -- dataverse.feature.shibboleth-use-wayfinder +- dataverse.feature.enable-pid-failure-log - dataverse.feature.shibboleth-use-localhost +- dataverse.feature.shibboleth-use-wayfinder - dataverse.person-or-org.assume-comma-in-person-name - dataverse.person-or-org.org-phrase-array From f31832313a8220206b0a410b38651d0e8e35f711 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 10:05:39 -0400 Subject: [PATCH 11/28] auth updates --- .../11605-existing-shib-external-users-auth.md | 1 - .../11645-existing-oauth-external-users-api-auth.md | 6 ------ .../11689-builtin-users-api-bearer-auth-enhance.md | 9 --------- doc/release-notes/6.8-release-notes.md | 12 ++++++++++++ 4 files changed, 12 insertions(+), 16 deletions(-) delete mode 100644 doc/release-notes/11605-existing-shib-external-users-auth.md delete mode 100644 doc/release-notes/11645-existing-oauth-external-users-api-auth.md delete mode 100644 doc/release-notes/11689-builtin-users-api-bearer-auth-enhance.md diff --git a/doc/release-notes/11605-existing-shib-external-users-auth.md b/doc/release-notes/11605-existing-shib-external-users-auth.md deleted file mode 100644 index a836c7737bb..00000000000 --- a/doc/release-notes/11605-existing-shib-external-users-auth.md +++ /dev/null @@ -1 +0,0 @@ -Implemented a new feature flag ``dataverse.feature.api-bearer-auth-use-shib-user-on-id-match``, which supports the use of the new Dataverse client in instances that have historically allowed login via Shibboleth. Specifically, with this flag enabled, when an OIDC bridge is configured to allow OIDC login with validation by the bridged Shibboleth providers, users with existing Shibboleth-based accounts in Dataverse can log in to those accounts, thereby maintaining access to their existing content and retaining their roles. (For security reasons, Dataverse's current support for direct login via Shibboleth cannot be used in browser-based clients.) \ No newline at end of file diff --git a/doc/release-notes/11645-existing-oauth-external-users-api-auth.md b/doc/release-notes/11645-existing-oauth-external-users-api-auth.md deleted file mode 100644 index 4afc5a38fb2..00000000000 --- a/doc/release-notes/11645-existing-oauth-external-users-api-auth.md +++ /dev/null @@ -1,6 +0,0 @@ -Implemented a new feature flag ``dataverse.feature.api-bearer-auth-use-oauth-user-on-id-match``, which supports the use of the new Dataverse client in instances that have historically allowed login via GitHub, ORCID, or Google. Specifically, with this flag enabled, when an OIDC bridge is configured to allow OIDC login with validation by the bridged OAuth providers, users with existing GitHub, ORCID, or Google accounts in Dataverse can log in to those accounts, thereby maintaining access to their existing content and retaining their roles. - -## New Settings - -- dataverse.feature.api-bearer-auth-use-oauth-user-on-id-match - diff --git a/doc/release-notes/11689-builtin-users-api-bearer-auth-enhance.md b/doc/release-notes/11689-builtin-users-api-bearer-auth-enhance.md deleted file mode 100644 index 671a5225781..00000000000 --- a/doc/release-notes/11689-builtin-users-api-bearer-auth-enhance.md +++ /dev/null @@ -1,9 +0,0 @@ -## Security improvements for `api-bearer-auth-use-builtin-user-on-id-match` - -We’ve strengthened the security of the `api-bearer-auth-use-builtin-user-on-id-match` feature flag. It will now only work when the provided bearer token includes an `idp` claim that matches the Keycloak Service Provider identifier. - -By enforcing this check, the risk of impersonation from other identity providers is significantly reduced, since they would need to be explicitly configured with this specific, non-standard identifier. - -See: -- [#11622 (comment)](https://github.com/IQSS/dataverse/pull/11622#discussion_r2216017175) -- [#11689](https://github.com/IQSS/dataverse/issues/11689) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 196b70dfdcd..7beb4a9fc0c 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -70,6 +70,16 @@ A new API has been added to get customization file contents: analytics, homepage This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. +## Authentication Updates + +Authentication updates listed below were introduced as we work toward allowing the [new Dataverse frontend](https://github.com/IQSS/dataverse-frontend), a React-based Single Page Application (SPA). We list them here for completeness but unless you are experimenting with the new frontend or playing with OIDC directly, they probably will have no impact on your installation. + +- We've strengthened the security of the `api-bearer-auth-use-builtin-user-on-id-match` feature flag. It will now only work when the provided bearer token includes an `idp` claim that matches the Keycloak Service Provider identifier. By enforcing this check, the risk of impersonation from other identity providers is significantly reduced, since they would need to be explici tly configured with this specific, non-standard identifier. See the list of [feature flags](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags), #11689, and #11763. +- A new feature flag `api-bearer-auth-use-shib-user-on-id-match` supports the use of clients in instances that have historically allowed login via Shibboleth. Specifically, with this flag enabled, when an OIDC bridge is configured to allow OIDC login with validation by the bridged Shibboleth providers, users with existing Shibboleth-based accounts in Dataverse can log in to those accounts, thereby maintaining access to their existing content and retaining their roles. (For security reasons, Dataverse's current support for direct login via Shibboleth cannot be used in browser-based clients.) See the list of [feature flags](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags), #11605, and #11622. +- A new feature flag `api-bearer-auth-use-oauth-user-on-id-match` supports the use of clients in instances that have historically allowed login via GitHub, ORCID, or Google. Specifically, with this flag enabled, when an OIDC bridge is configured to allow OIDC login with validation by the bridged OAuth providers, users with existing GitHub, ORCID, or Google accounts in Dataverse can log in to those accounts, thereby maintaining access to their existing content and retaining their roles. See the list of [feature flags](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags), #11671, and #11645. + +Finally, there is one other authenticated-related update that has the potential to affect a small number for Dataverse installations. See the EOL announcement below about the InCommon Federation feed for details. + ## Developer Updates ### Writing External Exporters @@ -86,6 +96,8 @@ For the relatively few instances using InCommon: Since InCommon discontinued the ## New Settings +- dataverse.feature.api-bearer-auth-use-oauth-user-on-id-match +- dataverse.feature.api-bearer-auth-use-shib-user-on-id-match - dataverse.feature.enable-pid-failure-log - dataverse.feature.shibboleth-use-localhost - dataverse.feature.shibboleth-use-wayfinder From 071968f04c3b874500d0ff6a0575f057f51cf12e Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 10:12:54 -0400 Subject: [PATCH 12/28] commons-lang3 bug fix --- .../11632-commons-lang3-update.md | 20 -------------- doc/release-notes/6.8-release-notes.md | 27 ++++++++++++++----- 2 files changed, 21 insertions(+), 26 deletions(-) delete mode 100644 doc/release-notes/11632-commons-lang3-update.md diff --git a/doc/release-notes/11632-commons-lang3-update.md b/doc/release-notes/11632-commons-lang3-update.md deleted file mode 100644 index c03929dfe02..00000000000 --- a/doc/release-notes/11632-commons-lang3-update.md +++ /dev/null @@ -1,20 +0,0 @@ -Due to changes in how the commons-lang3 library handles a non-ascii chararacter, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. Translations will need to address this. - -controlledvocabulary.language.magɨ_(madang_province) => controlledvocabulary.language.magi_(madang_province) -controlledvocabulary.language.magɨyi => controlledvocabulary.language.magiyi - -## Upgrade Instructions - -x\. Update metadata blocks - -These changes reflect incremental improvements made to the handling of core metadata fields. - -Reload the citation.tsv file to handle the commons-lang3 change mentioned above. - -Expect the loading of the citation block to take several seconds because of its size (especially due to the number of languages). - -```shell -wget https://raw.githubusercontent.com/IQSS/dataverse/v6.8/scripts/api/data/metadatablocks/citation.tsv - -curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv -``` diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 7beb4a9fc0c..5e5a043c6c6 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -24,6 +24,7 @@ Highlights for Dataverse 6.8 include: - The styled citations available through the "View Styled Citations" menu were including extra characters, e.g. 'doi:' in the URL form of the PIDs in the citation. This is now fixed. See #11629. - The updates to support keeping the history of curation status labels added in #11268 will incorrectly show curation statuses added prior to v6.7 as the current one, regardless of whether newer statuses exist. This bug has been fixed. See #11685. (As a work-around for 6.7 admins can add createtime dates (must be prior to when 6.7 was installed) to the curationstatus table for entries that have null createtimes. The code fix in this version properly handles null dates as indicating older/pre v6.7 curationstatuses.) - A bug introduced in v6.5 broken Handle parsing when using a lower-case shoulder. This is now fixed. See #11592. +- Due to changes in how the commons-lang3 Java library handles a non-ascii chararacter, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. `controlledvocabulary.language.magɨ_(madang_province)` has been changed to `controlledvocabulary.language.magi_(madang_province)` and `controlledvocabulary.language.magɨyi` has been changed to `controlledvocabulary.language.magiyi`. In the upgrade instructions below, we indicate that the citation metadata block should be reloaded. Translations will need make the same adjustment. See #11632. - When following the container demo tutorial, it was not possible to update Solr fields after adding additional metadata blocks. This has been fixed. See #11722 and #11723. ## API Updates @@ -181,13 +182,27 @@ sudo rm -rf $PAYARA/glassfish/domains/domain1/lib/databases sudo service payara start ``` -4\. For installations with internationalization or text customizations: +4\. Update metadata blocks + +These changes reflect incremental improvements made to the handling of core metadata fields. + +Reload the citation.tsv file to handle the commons-lang3 change mentioned above. + +Expect the loading of the citation block to take several seconds because of its size (especially due to the number of languages). + +```shell +wget https://raw.githubusercontent.com/IQSS/dataverse/v6.8/scripts/api/data/metadatablocks/citation.tsv + +curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv +``` + +5\. For installations with internationalization or text customizations: Please remember to update translations via [Dataverse language packs](https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs). If you have text customizations you can get the latest English files from . -5\. Update Solr schema and reindex +6\. Update Solr schema and reindex Due to changes in the Solr schema (the addition of field "datasetCount"), updating the Solr schema and reindexing is required. @@ -198,7 +213,7 @@ wget https://raw.githubusercontent.com/IQSS/dataverse/v6.8/conf/solr/schema.xml cp schema.xml /usr/local/solr/solr-9.8.0/server/solr/collection1/conf ``` -5a\. For installations with additional metadata blocks or external controlled vocabulary scripts, update fields +6a\. For installations with additional metadata blocks or external controlled vocabulary scripts, update fields - Stop Solr instance (usually `service solr stop`, depending on Solr installation/OS, see the [Installation Guide](https://guides.dataverse.org/en/6.8/installation/prerequisites.html#solr-init-script)). @@ -214,17 +229,17 @@ Note that Docker-based installations use a different directory: `solr/data/data/ - Start Solr instance (usually `service solr start` depending on Solr/OS). -6\. Reindex Solr +7\. Reindex Solr ```shell curl http://localhost:8080/api/admin/index ``` -7\. Update dataverse.personOrOrg.orgPhraseArray, if used. +8\. Update dataverse.personOrOrg.orgPhraseArray, if used. If you are using the `dataverse.personOrOrg.orgPhraseArray` setting, rename it to `dataverse.person-or-org.org-phrase-array` and replace the JSON array of strings with a comma-separated list. See also the ([docs](https://guides.dataverse.org/en/6.8/installation/config.html#dataverse-person-or-org-org-phrase-array)) for this settings and the New Settings and Backward Incompatible Changes sections above. -8\. InCommon federation login update +9\. InCommon federation login update If your instance is offering institutional Shibboleth logins as part of the InCommon federation, you must make some changes to your service configuration: From 72b002488c7f10b432da296737a68a10fbd72daf Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 10:34:40 -0400 Subject: [PATCH 13/28] API updates: links, isAdvancedSearchFieldType --- .../11614-include-isAdvancedSeachField-property.md | 3 --- .../11633-list-dataverse-links-api-change.md | 1 - ...11724-extend-list-dataverse-collection-links.md | 1 - doc/release-notes/6.8-release-notes.md | 14 +++++++++++++- 4 files changed, 13 insertions(+), 6 deletions(-) delete mode 100644 doc/release-notes/11614-include-isAdvancedSeachField-property.md delete mode 100644 doc/release-notes/11633-list-dataverse-links-api-change.md delete mode 100644 doc/release-notes/11724-extend-list-dataverse-collection-links.md diff --git a/doc/release-notes/11614-include-isAdvancedSeachField-property.md b/doc/release-notes/11614-include-isAdvancedSeachField-property.md deleted file mode 100644 index a544570c52e..00000000000 --- a/doc/release-notes/11614-include-isAdvancedSeachField-property.md +++ /dev/null @@ -1,3 +0,0 @@ -The API endpoints `api/{dataverse-alias}/metadatablocks` and `/api/metadatablocks/{block_id}` have been extended to include the following field: - -- `isAdvancedSearchFieldType`: Whether the field can be used in advanced search or not. \ No newline at end of file diff --git a/doc/release-notes/11633-list-dataverse-links-api-change.md b/doc/release-notes/11633-list-dataverse-links-api-change.md deleted file mode 100644 index 9026349045b..00000000000 --- a/doc/release-notes/11633-list-dataverse-links-api-change.md +++ /dev/null @@ -1 +0,0 @@ -The [API for listing the collections a dataverse has been linked to](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#list-dataverse-collection-links) (`api/dataverses/$dataverse-alias/links`) has been refactored to return a new Json format. This is a breaking API. diff --git a/doc/release-notes/11724-extend-list-dataverse-collection-links.md b/doc/release-notes/11724-extend-list-dataverse-collection-links.md deleted file mode 100644 index 9026349045b..00000000000 --- a/doc/release-notes/11724-extend-list-dataverse-collection-links.md +++ /dev/null @@ -1 +0,0 @@ -The [API for listing the collections a dataverse has been linked to](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#list-dataverse-collection-links) (`api/dataverses/$dataverse-alias/links`) has been refactored to return a new Json format. This is a breaking API. diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 5e5a043c6c6..cf19f6b407e 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -61,12 +61,20 @@ See also [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#up ### Listing Collections a Dataset Has Been Linked To -The [API for listing the collections a dataset has been linked to](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset) (`api/datasets/$linked-dataset-id/links`) is no longer restricted to superusers. For unpublished datasets, users need the "View Unpublished Dataset" permission to access the API. Unpublished collections in the list require the "View Unpublished Dataverse" permission; otherwise, they are hidden. See #11492. +The [API for listing the collections a dataset has been linked to](https://guides.dataverse.org/en/6.7/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset) (`api/datasets/$linked-dataset-id/links`) is no longer restricted to superusers. For unpublished datasets, users need the "View Unpublished Dataset" permission to access the API. Unpublished collections in the list require the "View Unpublished Dataverse" permission; otherwise, they are hidden. See #11492. + +### Listing Collections a Collection Has Been Linked To + +The [API for listing the collections a collection has been linked to](https://guides.dataverse.org/en/6.8/admin/dataverses-datasets.html#list-dataverse-collection-links) now returns a different, backward-incompatible JSON format. See #11633, #11669, and the [API Changelog](https://guides.dataverse.org/en/6.8/api/changelog.html) (also listed under Backward Incompatible Changes, below). Also, additional fields are now being returned. See #11724 and #11728. ### Get Customization File Contents API A new API has been added to get customization file contents: analytics, homepage, header, footer, style, and logo. See [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#get-customization-file-contents), #11448, and #11467. +### Listing Metadata Blocks: isAdvancedSearchFieldType + +The API endpoints `api/{dataverse-alias}/metadatablocks` and `/api/metadatablocks/{block_id}` have been extended to include `isAdvancedSearchFieldType` to know whether the field can be used in advanced search or not. See #11614 and #11617. + ## Security Updates This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. @@ -119,6 +127,10 @@ Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/late The setting `dataverse.personOrOrg.orgPhraseArray` has been renamed to `dataverse.person-or-org.org-phrase-array` and now expects a comma-separated list of phrases as a value instead of a JsonArray of strings. +### Different JSON Format When Listing Collection Links + +The [API for listing the collections a collection has been linked to](https://guides.dataverse.org/en/6.8/admin/dataverses-datasets.html#list-dataverse-collection-links) now returns a different, backward-incompatible JSON format. See #11633, #11669. + ## Complete List of Changes For the complete list of code changes in this release, see the [6.8 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.8+is%3Aclosed) in GitHub. From 783540ded120a930634e2c0308df2e5fd7f024f2 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 10:46:46 -0400 Subject: [PATCH 14/28] add externalTool API changes --- doc/release-notes/11760-tool url apis.md | 10 ---------- doc/release-notes/6.8-release-notes.md | 13 +++++++++++++ 2 files changed, 13 insertions(+), 10 deletions(-) delete mode 100644 doc/release-notes/11760-tool url apis.md diff --git a/doc/release-notes/11760-tool url apis.md b/doc/release-notes/11760-tool url apis.md deleted file mode 100644 index a54641a29e6..00000000000 --- a/doc/release-notes/11760-tool url apis.md +++ /dev/null @@ -1,10 +0,0 @@ -New API calls have been added to retrieve the URLs needed to launch external tools on specific datasets and files: - -/api/datasets/$DATASET_ID/externalTool/$TOOL_ID/toolUrl -and -/api/files/$FILE_ID/externalTool/$TOOL_ID/toolUrl - -If the dataset/file is not public, the caller must authenticate and have permission to view the dataset/file. In such cases, the generated URL will include a callback token containing a signed URL the tool can use to retrieve all the parameters it is configured for. - -Backward incompatibility: -The responses from the GET /api/externalTools and /api/externalTools/{id} are now formatted as JSON (previously the toolParameters and allowedApiCalls were JSON serialized as strings) and any confiugured "requirements" are included. \ No newline at end of file diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index cf19f6b407e..4f7de2e89c9 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -75,6 +75,15 @@ A new API has been added to get customization file contents: analytics, homepage The API endpoints `api/{dataverse-alias}/metadatablocks` and `/api/metadatablocks/{block_id}` have been extended to include `isAdvancedSearchFieldType` to know whether the field can be used in advanced search or not. See #11614 and #11617. +### Retrieving URLs to Launch External Tools + +New API calls have been added to retrieve the URLs needed to launch external tools on specific datasets and files: + +- `/api/datasets/$DATASET_ID/externalTool/$TOOL_ID/toolUrl`: [docs](https://guides.dataverse.org/en/6.8/api/native-api.html#get-dataset-external-tool-url) +- `/api/files/$FILE_ID/externalTool/$TOOL_ID/toolUrl`: [docs](https://guides.dataverse.org/en/6.8/api/native-api.html#get-file-external-tool-url) + +If the dataset/file is not public, the caller must authenticate and have permission to view the dataset/file. In such cases, the generated URL will include a callback token containing a signed URL the tool can use to retrieve all the parameters it is configured for. See Backward Incompatible Changes, below for a change to the JSON response. See #11760. + ## Security Updates This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. @@ -131,6 +140,10 @@ The setting `dataverse.personOrOrg.orgPhraseArray` has been renamed to `datavers The [API for listing the collections a collection has been linked to](https://guides.dataverse.org/en/6.8/admin/dataverses-datasets.html#list-dataverse-collection-links) now returns a different, backward-incompatible JSON format. See #11633, #11669. +### /api/externalTools Response + +The responses from the `GET` `/api/externalTools` and `/api/externalTools/{id}` are now formatted as JSON (previously the toolParameters and allowedApiCalls were JSON serialized as strings) and any configured "requirements" are included. See #11760. + ## Complete List of Changes For the complete list of code changes in this release, see the [6.8 milestone](https://github.com/IQSS/dataverse/issues?q=milestone%3A6.8+is%3Aclosed) in GitHub. From a082bb1542499b357810f25ca494ab1e85440c0b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 10:49:23 -0400 Subject: [PATCH 15/28] reorder API updates --- doc/release-notes/6.8-release-notes.md | 24 ++++++++++++------------ 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 4f7de2e89c9..ae059bed158 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -51,14 +51,6 @@ The Search API now supports a `show_collections` parameter for dataset results. See also [the guides](https://guides.dataverse.org/en/6.8/api/search.html) and #11558. -### Edit File Metadata: Empty Values Clear Data - -Previously the API POST /files/{id}/metadata would ignore fields with empty values. Now the API updates the fields with the empty values essentially clearing the data. Missing fields will still be ignored. - -An optional query parameter (sourceLastUpdateTime) was added to ensure the metadata update doesn't overwrite stale data. - -See also [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#updating-file-metadata), #11392, #11439, and the [API Changelog](https://guides.dataverse.org/en/6.8/api/changelog.html) (also listed under Backward Incompatible Changes, below). - ### Listing Collections a Dataset Has Been Linked To The [API for listing the collections a dataset has been linked to](https://guides.dataverse.org/en/6.7/admin/dataverses-datasets.html#list-collections-that-are-linked-from-a-dataset) (`api/datasets/$linked-dataset-id/links`) is no longer restricted to superusers. For unpublished datasets, users need the "View Unpublished Dataset" permission to access the API. Unpublished collections in the list require the "View Unpublished Dataverse" permission; otherwise, they are hidden. See #11492. @@ -67,14 +59,22 @@ The [API for listing the collections a dataset has been linked to](https://guide The [API for listing the collections a collection has been linked to](https://guides.dataverse.org/en/6.8/admin/dataverses-datasets.html#list-dataverse-collection-links) now returns a different, backward-incompatible JSON format. See #11633, #11669, and the [API Changelog](https://guides.dataverse.org/en/6.8/api/changelog.html) (also listed under Backward Incompatible Changes, below). Also, additional fields are now being returned. See #11724 and #11728. -### Get Customization File Contents API - -A new API has been added to get customization file contents: analytics, homepage, header, footer, style, and logo. See [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#get-customization-file-contents), #11448, and #11467. - ### Listing Metadata Blocks: isAdvancedSearchFieldType The API endpoints `api/{dataverse-alias}/metadatablocks` and `/api/metadatablocks/{block_id}` have been extended to include `isAdvancedSearchFieldType` to know whether the field can be used in advanced search or not. See #11614 and #11617. +### Edit File Metadata: Empty Values Clear Data + +Previously the API POST /files/{id}/metadata would ignore fields with empty values. Now the API updates the fields with the empty values essentially clearing the data. Missing fields will still be ignored. + +An optional query parameter (sourceLastUpdateTime) was added to ensure the metadata update doesn't overwrite stale data. + +See also [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#updating-file-metadata), #11392, #11439, and the [API Changelog](https://guides.dataverse.org/en/6.8/api/changelog.html) (also listed under Backward Incompatible Changes, below). + +### Get Customization File Contents API + +A new API has been added to get customization file contents: analytics, homepage, header, footer, style, and logo. See [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#get-customization-file-contents), #11448, and #11467. + ### Retrieving URLs to Launch External Tools New API calls have been added to retrieve the URLs needed to launch external tools on specific datasets and files: From f1b791e2646de52b799efb066d1b08b301cacb7a Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:16:21 -0400 Subject: [PATCH 16/28] APIs: templates, file categories, notifications --- doc/release-notes/11562-templates-api.md | 4 --- ...t-dataset-file-available-categories-api.md | 4 --- .../11648-notifications-api-extension.md | 7 ------ doc/release-notes/11650-unread.md | 11 -------- ...3-notifications-and-templates-api-fixes.md | 13 ---------- doc/release-notes/6.8-release-notes.md | 25 +++++++++++++++++++ 6 files changed, 25 insertions(+), 39 deletions(-) delete mode 100644 doc/release-notes/11562-templates-api.md delete mode 100644 doc/release-notes/11634-get-dataset-file-available-categories-api.md delete mode 100644 doc/release-notes/11648-notifications-api-extension.md delete mode 100644 doc/release-notes/11650-unread.md delete mode 100644 doc/release-notes/11703-notifications-and-templates-api-fixes.md diff --git a/doc/release-notes/11562-templates-api.md b/doc/release-notes/11562-templates-api.md deleted file mode 100644 index 30e35687a33..00000000000 --- a/doc/release-notes/11562-templates-api.md +++ /dev/null @@ -1,4 +0,0 @@ -New endpoints have been implemented in the Dataverses API for the management of dataverse templates: - -- POST `/dataverses/{id}/templates`: Creates a template for a given Dataverse collection ``id``. -- GET `/dataverses/{id}/templates`: Lists the templates for a given Dataverse collection ``id``. diff --git a/doc/release-notes/11634-get-dataset-file-available-categories-api.md b/doc/release-notes/11634-get-dataset-file-available-categories-api.md deleted file mode 100644 index 38fe61b8eed..00000000000 --- a/doc/release-notes/11634-get-dataset-file-available-categories-api.md +++ /dev/null @@ -1,4 +0,0 @@ -### Get Dataset File Available Categories API - -- This new endpoint allows the user to get all of the available file categories for a dataset, both built-in and custom. - diff --git a/doc/release-notes/11648-notifications-api-extension.md b/doc/release-notes/11648-notifications-api-extension.md deleted file mode 100644 index ee5aa22863d..00000000000 --- a/doc/release-notes/11648-notifications-api-extension.md +++ /dev/null @@ -1,7 +0,0 @@ -# getAllNotificationsForUser API extension - -- Extended endpoint getAllNotificationsForUser(``/notifications/all``), which now supports an optional query parameter ``inAppNotificationFormat`` which, if sent as ``true``, retrieves the fields needed to build the in-app notifications for the Notifications section of the Dataverse UI, omitting fields related to email notifications. See also #11648 and #11696. - -# Notifications triggered by API endpoints - -The addDataset and addDataverse API endpoints now trigger user notifications upon successful execution. See also #1342 and #11696. diff --git a/doc/release-notes/11650-unread.md b/doc/release-notes/11650-unread.md deleted file mode 100644 index 07ab852e24d..00000000000 --- a/doc/release-notes/11650-unread.md +++ /dev/null @@ -1,11 +0,0 @@ -## API Updates - -### Support read/unread status for notifications - -The API for managing notifications has been extended. - -- displayAsRead boolean added to "get all" -- new GET unreadCount API endpoint -- new PUT markAsRead API endpoint - -See also [the guides](https://dataverse-guide--11664.org.readthedocs.build/en/11664/api/native-api.html#notifications), #11650, and #11664. diff --git a/doc/release-notes/11703-notifications-and-templates-api-fixes.md b/doc/release-notes/11703-notifications-and-templates-api-fixes.md deleted file mode 100644 index 27e5484cd12..00000000000 --- a/doc/release-notes/11703-notifications-and-templates-api-fixes.md +++ /dev/null @@ -1,13 +0,0 @@ -# Release Notes - -## Templates API fixes -- Added the missing `isDefault` field to the **Get Dataset Templates** API response. -- Added the missing `fileAccessRequest` field of terms of use and access to the **Get Dataset Templates** API response. -- Added the missing `contactForAccess` field of terms of use and access to the **Get Dataset Templates** API response. -- Resolved an API **500 error** that occurred when working with templates containing custom license terms. -- Updated API behavior so templates are **no longer returned from parent dataverses** when the "Include Templates from Root" option is unchecked in the UI. - -## Notifications API fixes -- Fixed API errors caused by NullPointerException when retrieving notifications without a requestor. - -See #11704 diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index ae059bed158..e5e7972456e 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -23,12 +23,27 @@ Highlights for Dataverse 6.8 include: - A bug, introduced in v6.7, that caused files to not be indexed in draft versions that were added after the initial dataset version was published, has been fixed. See #11776 and #11779. - The styled citations available through the "View Styled Citations" menu were including extra characters, e.g. 'doi:' in the URL form of the PIDs in the citation. This is now fixed. See #11629. - The updates to support keeping the history of curation status labels added in #11268 will incorrectly show curation statuses added prior to v6.7 as the current one, regardless of whether newer statuses exist. This bug has been fixed. See #11685. (As a work-around for 6.7 admins can add createtime dates (must be prior to when 6.7 was installed) to the curationstatus table for entries that have null createtimes. The code fix in this version properly handles null dates as indicating older/pre v6.7 curationstatuses.) +- The addDataset and addDataverse API endpoints now trigger user notifications See #1342 and #11696. - A bug introduced in v6.5 broken Handle parsing when using a lower-case shoulder. This is now fixed. See #11592. + - Due to changes in how the commons-lang3 Java library handles a non-ascii chararacter, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. `controlledvocabulary.language.magɨ_(madang_province)` has been changed to `controlledvocabulary.language.magi_(madang_province)` and `controlledvocabulary.language.magɨyi` has been changed to `controlledvocabulary.language.magiyi`. In the upgrade instructions below, we indicate that the citation metadata block should be reloaded. Translations will need make the same adjustment. See #11632. - When following the container demo tutorial, it was not possible to update Solr fields after adding additional metadata blocks. This has been fixed. See #11722 and #11723. ## API Updates +### Templates API + +New endpoints have been implemented in the Dataverses API for the management of dataset templates: + +- POST `/dataverses/{id}/templates`: Creates a template for a given collection `id`. +- GET `/dataverses/{id}/templates`: Lists the templates for a given collection `id`. + +See [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#list-templates-of-a-collection), #11562, #11565, #11703, and #11704. + +### File Categories API + +A new API was added that returns a list of categories (both built-in and custom) that may be applied to the files of a given dataset. See [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#get-available-dataset-file-categories), #11634, and #11668. + ### MyData Collection List API The MyData Collection List API is used to get a list of the collections an authenticated user can create a Dataset in. Param userIdentifier={userName} is used by a superuser to get the collections for a specific user. @@ -63,6 +78,16 @@ The [API for listing the collections a collection has been linked to](https://gu The API endpoints `api/{dataverse-alias}/metadatablocks` and `/api/metadatablocks/{block_id}` have been extended to include `isAdvancedSearchFieldType` to know whether the field can be used in advanced search or not. See #11614 and #11617. +### Notifications API: unreadCount, markAsRead, inAppNotificationFormat + +The Notifications API has been updated in various ways: + +- The JSON returned from [listing notifications](https://guides.dataverse.org/en/6.8/api/native-api.html#get-all-notifications-by-user) now includes a "displayAsRead" boolean to indicated if a notification has been read. See #11650 and #11664. +- You can get an count of unread notifications via a new [unreadCount](https://guides.dataverse.org/en/6.8/api/native-api.html#get-unread-count) API endpoint. +- You can mark a notification as read via a new [markAsRead](https://guides.dataverse.org/en/6.8/api/native-api.html#mark-notification-as-read) API endpoint. +- The JSON can be returned using `inAppNotificationFormat`. See #11648 and #11696. +- A bug was fixed where a NullPointerException was being thrown when retrieving notifications without a requestor. See #11703, and #11704. + ### Edit File Metadata: Empty Values Clear Data Previously the API POST /files/{id}/metadata would ignore fields with empty values. Now the API updates the fields with the empty values essentially clearing the data. Missing fields will still be ignored. From 25260ce5a506b69f486f7b7fd81e483b1c22790c Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:30:51 -0400 Subject: [PATCH 17/28] add postgres EOL reminder --- doc/release-notes/6.8-release-notes.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index e5e7972456e..6398bc8498f 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -131,6 +131,10 @@ The getDatasetFileDetails data structure now contains "directoryLabel" (file pat ## End-Of-Life (EOL) Announcements +### PostgreSQL 13 Reaches EOL on 13 November 2025 + +We mentioned this in the [Dataverse 6.6 release notes](https://github.com/IQSS/dataverse/releases/tag/v6.6), but as a reminder, according to https://www.postgresql.org/support/versioning/ PostgreSQL 13 reaches EOL on 13 November 2025. As mentioned in the [Installation Guide](https://guides.dataverse.org/en/6.8/installation/prerequisites.html#postgresql), we recommend running PostgreSQL 16 since it is the version we test with in our continous integration ([since](https://github.com/gdcc/dataverse-ansible/commit/8ebbd84ad2cf3903b8f995f0d34578250f4223ff) February 2025). The [Dataverse 5.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v5.4) explained the upgrade process from 9 to 13 (e.g. pg_dumpall, etc.) and the steps will be similar. If you have any problems, please feel free to reach out (see "getting help" in these release notes). + ### For Dataverse instances that Use Shibboleth as Members of the InCommon Federation Please note that most of the known Dataverse instances that support Shibboleth logins do so without being part of InCommon, and therefore are not affected. All such instances will be able to continue using the old login workflow without needing to make any configuration changes. From c2ec2d2faf8f9710a60b6df95459bbdf6bf66156 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:34:53 -0400 Subject: [PATCH 18/28] group search API improvements --- doc/release-notes/6.8-release-notes.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 6398bc8498f..9433cab3c1e 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -15,7 +15,6 @@ Highlights for Dataverse 6.8 include: - Open OnDemand, a web frontend to High Performance Computing (HPC) resources, has been integrated with Dataverse, allowing files to be downloaded from Dataverse installations around the world or from a specific dataset landing page if an external tool is enabled. Additionally, after computation is complete, datasets can be created in Dataverse from Open OnDemand and files can be uploaded. See the [docs](https://guides.dataverse.org/en/6.8/admin/integrations.html#open-ondemand), #11768 and #11769. - A new [feature flag](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags) called `enable-pid-failure-log` can be enabled to help diagnose PID failures. When set, Dataverse will log requests for dataset and file pages via persistentId that fail in monthly log files of the form `PIDFailures_.log`. These potentially indicate when someone has shared a draft PID without publishing or cases where a '.' or other character has been added to the PID, which may be of interest to site administrators. The new log files can be used in concert with the [pidreport.py](https://github.com/gdcc/dataverse-recipes/tree/main/python/pid_reports) script to generate and email monthly PID failure reports. See #11601. -- The search index now includes datasetCount for each collection, counting published, linked, and harvested datasets. Collections can be filtered using datasetCount (e.g., `datasetCount:[1000 TO *]`), and the value is returned in Dataverse search results via the Search API. See #10190. - Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" or "LinkDataverse" permissions, respectively. Previously, this action was covered by the "PublishDataset" or "PublishDataverse" permission. Splitting linking into its own permission, separate from publishing, allows for more fine-grained access control, if you choose to implement custom roles that differ from the roles that Dataverse ships with. A (Flyway) database migration script will be run automatically such that all roles that have permission to publish will continue to have permission to link. See #11691. ## Bugs Fixed @@ -50,7 +49,9 @@ The MyData Collection List API is used to get a list of the collections an authe See also [the guides](https://guides.dataverse.org/en/6.8/api/native-api.html#mydata-collection-list) #11525, and #11681. -### Search API show_collections Parameter +### Search API: datasetCount and show_collections + +The search index now includes datasetCount for each collection, counting published, linked, and harvested datasets. Collections can be filtered using datasetCount (e.g., `datasetCount:[1000 TO *]`), and the value is returned in Dataverse search results via the Search API. See #10190. The Search API now supports a `show_collections` parameter for dataset results. When the parameter is set, each result includes a `collections` array showing the dataset's parent and linked collections. Each entry includes `id`, `name`, and `alias`, for example: From 7bd2bf440399dca45b41a2868bae13dcb4ac927b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:38:28 -0400 Subject: [PATCH 19/28] update highlights, expand features --- doc/release-notes/6.8-release-notes.md | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 9433cab3c1e..1d009efea31 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -8,14 +8,25 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.8 include: -- Open OnDemand Integration -- Dignosing PID failures +- Open OnDemand integration +- Logs for dignosing PID failures +- Link permission split off from publish permission +- New and improved APIs +- Bug fixes ## Features Added -- Open OnDemand, a web frontend to High Performance Computing (HPC) resources, has been integrated with Dataverse, allowing files to be downloaded from Dataverse installations around the world or from a specific dataset landing page if an external tool is enabled. Additionally, after computation is complete, datasets can be created in Dataverse from Open OnDemand and files can be uploaded. See the [docs](https://guides.dataverse.org/en/6.8/admin/integrations.html#open-ondemand), #11768 and #11769. -- A new [feature flag](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags) called `enable-pid-failure-log` can be enabled to help diagnose PID failures. When set, Dataverse will log requests for dataset and file pages via persistentId that fail in monthly log files of the form `PIDFailures_.log`. These potentially indicate when someone has shared a draft PID without publishing or cases where a '.' or other character has been added to the PID, which may be of interest to site administrators. The new log files can be used in concert with the [pidreport.py](https://github.com/gdcc/dataverse-recipes/tree/main/python/pid_reports) script to generate and email monthly PID failure reports. See #11601. -- Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" or "LinkDataverse" permissions, respectively. Previously, this action was covered by the "PublishDataset" or "PublishDataverse" permission. Splitting linking into its own permission, separate from publishing, allows for more fine-grained access control, if you choose to implement custom roles that differ from the roles that Dataverse ships with. A (Flyway) database migration script will be run automatically such that all roles that have permission to publish will continue to have permission to link. See #11691. +### Open OnDemand Integration + +Open OnDemand, a web frontend to High Performance Computing (HPC) resources, has been integrated with Dataverse, allowing files to be downloaded from Dataverse installations around the world or from a specific dataset landing page if an external tool is enabled. Additionally, after computation is complete, datasets can be created in Dataverse from Open OnDemand and files can be uploaded. See the [docs](https://guides.dataverse.org/en/6.8/admin/integrations.html#open-ondemand), #11768 and #11769. + +### Logs for Dignosing PID Failures + +A new [feature flag](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags) called `enable-pid-failure-log` can be enabled to help diagnose PID failures. When set, Dataverse will log requests for dataset and file pages via persistentId that fail in monthly log files of the form `PIDFailures_.log`. These potentially indicate when someone has shared a draft PID without publishing or cases where a '.' or other character has been added to the PID, which may be of interest to site administrators. The new log files can be used in concert with the [pidreport.py](https://github.com/gdcc/dataverse-recipes/tree/main/python/pid_reports) script to generate and email monthly PID failure reports. See #11601. + +### Link Permission Split Off from Publish Permission + +Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" or "LinkDataverse" permissions, respectively. Previously, this action was covered by the "PublishDataset" or "PublishDataverse" permission. Splitting linking into its own permission, separate from publishing, allows for more fine-grained access control, if you choose to implement custom roles that differ from the roles that Dataverse ships with. A (Flyway) database migration script will be run automatically such that all roles that have permission to publish will continue to have permission to link. See #11691. ## Bugs Fixed From bc58ac3f11a5e73678f61dcedfeb3d577d48c60c Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:39:01 -0400 Subject: [PATCH 20/28] remove extra newline --- doc/release-notes/6.8-release-notes.md | 1 - 1 file changed, 1 deletion(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 1d009efea31..a81cbc57c73 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -35,7 +35,6 @@ Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" o - The updates to support keeping the history of curation status labels added in #11268 will incorrectly show curation statuses added prior to v6.7 as the current one, regardless of whether newer statuses exist. This bug has been fixed. See #11685. (As a work-around for 6.7 admins can add createtime dates (must be prior to when 6.7 was installed) to the curationstatus table for entries that have null createtimes. The code fix in this version properly handles null dates as indicating older/pre v6.7 curationstatuses.) - The addDataset and addDataverse API endpoints now trigger user notifications See #1342 and #11696. - A bug introduced in v6.5 broken Handle parsing when using a lower-case shoulder. This is now fixed. See #11592. - - Due to changes in how the commons-lang3 Java library handles a non-ascii chararacter, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. `controlledvocabulary.language.magɨ_(madang_province)` has been changed to `controlledvocabulary.language.magi_(madang_province)` and `controlledvocabulary.language.magɨyi` has been changed to `controlledvocabulary.language.magiyi`. In the upgrade instructions below, we indicate that the citation metadata block should be reloaded. Translations will need make the same adjustment. See #11632. - When following the container demo tutorial, it was not possible to update Solr fields after adding additional metadata blocks. This has been fixed. See #11722 and #11723. From 7c981890ef79383662623f4f830d9277b57b9a73 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:41:06 -0400 Subject: [PATCH 21/28] no security updates --- doc/release-notes/6.8-release-notes.md | 4 ---- 1 file changed, 4 deletions(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index a81cbc57c73..41c608b30f7 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -120,10 +120,6 @@ New API calls have been added to retrieve the URLs needed to launch external too If the dataset/file is not public, the caller must authenticate and have permission to view the dataset/file. In such cases, the generated URL will include a callback token containing a signed URL the tool can use to retrieve all the parameters it is configured for. See Backward Incompatible Changes, below for a change to the JSON response. See #11760. -## Security Updates - -This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. - ## Authentication Updates Authentication updates listed below were introduced as we work toward allowing the [new Dataverse frontend](https://github.com/IQSS/dataverse-frontend), a React-based Single Page Application (SPA). We list them here for completeness but unless you are experimenting with the new frontend or playing with OIDC directly, they probably will have no impact on your installation. From b26946a0dcf21f5d6b26bb0e65bc91474d309930 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:44:08 -0400 Subject: [PATCH 22/28] no removed settings, update deprecated --- doc/release-notes/6.8-release-notes.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 41c608b30f7..f29248124ce 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -162,7 +162,8 @@ The settings `dataverse.personOrOrg.assumeCommaInPersonName` and `dataverse.pers ## Deprecated Settings -## Removed Settings +- dataverse.personOrOrg.assumeCommaInPersonName +- dataverse.personOrOrg.orgPhraseArray ## Backward Incompatible Changes From 714b28cf73d71e51b8e2ad2031917930aa8a7ffd Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 12:48:32 -0400 Subject: [PATCH 23/28] improve backward incompat section --- doc/release-notes/6.8-release-notes.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index f29248124ce..4f264be8f96 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -171,7 +171,12 @@ Generally speaking, see the [API Changelog](https://guides.dataverse.org/en/late ### dataverse.personOrOrg.orgPhraseArray -The setting `dataverse.personOrOrg.orgPhraseArray` has been renamed to `dataverse.person-or-org.org-phrase-array` and now expects a comma-separated list of phrases as a value instead of a JsonArray of strings. +The setting `dataverse.personOrOrg.orgPhraseArray` has been renamed to `dataverse.person-or-org.org-phrase-array` and now expects a comma-separated list of phrases as a value instead of a JsonArray of strings. See #11485. + +### Edit Metadata API Changes + +- For POST /api/files/{id}/metadata passing an empty string ("description":"") or array ("categories":[]) will no longer be ignored. Empty fields will now clear out the values in the file's metadata. To ignore the fields simply do not include them in the JSON string. See #11439. +- For PUT /api/datasets/{id}/editMetadata the query parameter "sourceInternalVersionNumber" has been removed and replaced with "sourceLastUpdateTime" to verify that the data being edited hasn't been modified and isn't stale. See #11439. ### Different JSON Format When Listing Collection Links From 157beef6e11d3a6dbf9b98a2ca2a0010968ff67b Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Mon, 15 Sep 2025 15:13:19 -0400 Subject: [PATCH 24/28] Apply suggestions from code review Co-authored-by: Omer Fahim --- doc/release-notes/6.8-release-notes.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 4f264be8f96..deb217cb0a6 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -9,7 +9,7 @@ This release brings new features, enhancements, and bug fixes to Dataverse. Than Highlights for Dataverse 6.8 include: - Open OnDemand integration -- Logs for dignosing PID failures +- Logs for diagnosing PID failures - Link permission split off from publish permission - New and improved APIs - Bug fixes @@ -20,7 +20,7 @@ Highlights for Dataverse 6.8 include: Open OnDemand, a web frontend to High Performance Computing (HPC) resources, has been integrated with Dataverse, allowing files to be downloaded from Dataverse installations around the world or from a specific dataset landing page if an external tool is enabled. Additionally, after computation is complete, datasets can be created in Dataverse from Open OnDemand and files can be uploaded. See the [docs](https://guides.dataverse.org/en/6.8/admin/integrations.html#open-ondemand), #11768 and #11769. -### Logs for Dignosing PID Failures +### Logs for Diagnosing PID Failures A new [feature flag](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags) called `enable-pid-failure-log` can be enabled to help diagnose PID failures. When set, Dataverse will log requests for dataset and file pages via persistentId that fail in monthly log files of the form `PIDFailures_.log`. These potentially indicate when someone has shared a draft PID without publishing or cases where a '.' or other character has been added to the PID, which may be of interest to site administrators. The new log files can be used in concert with the [pidreport.py](https://github.com/gdcc/dataverse-recipes/tree/main/python/pid_reports) script to generate and email monthly PID failure reports. See #11601. @@ -33,9 +33,9 @@ Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" o - A bug, introduced in v6.7, that caused files to not be indexed in draft versions that were added after the initial dataset version was published, has been fixed. See #11776 and #11779. - The styled citations available through the "View Styled Citations" menu were including extra characters, e.g. 'doi:' in the URL form of the PIDs in the citation. This is now fixed. See #11629. - The updates to support keeping the history of curation status labels added in #11268 will incorrectly show curation statuses added prior to v6.7 as the current one, regardless of whether newer statuses exist. This bug has been fixed. See #11685. (As a work-around for 6.7 admins can add createtime dates (must be prior to when 6.7 was installed) to the curationstatus table for entries that have null createtimes. The code fix in this version properly handles null dates as indicating older/pre v6.7 curationstatuses.) -- The addDataset and addDataverse API endpoints now trigger user notifications See #1342 and #11696. -- A bug introduced in v6.5 broken Handle parsing when using a lower-case shoulder. This is now fixed. See #11592. -- Due to changes in how the commons-lang3 Java library handles a non-ascii chararacter, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. `controlledvocabulary.language.magɨ_(madang_province)` has been changed to `controlledvocabulary.language.magi_(madang_province)` and `controlledvocabulary.language.magɨyi` has been changed to `controlledvocabulary.language.magiyi`. In the upgrade instructions below, we indicate that the citation metadata block should be reloaded. Translations will need make the same adjustment. See #11632. +- The addDataset and addDataverse API endpoints now trigger user notifications. See #1342 and #11696. +- A bug introduced in v6.5 broke Handle parsing when using a lowercase shoulder. This is now fixed. See #11592. +- Due to changes in how the commons-lang3 Java library handles a non-ascii chararacter, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. `controlledvocabulary.language.magɨ_(madang_province)` has been changed to `controlledvocabulary.language.magi_(madang_province)` and `controlledvocabulary.language.magɨyi` has been changed to `controlledvocabulary.language.magiyi`. In the upgrade instructions below, we indicate that the citation metadata block should be reloaded. Translations will need to make the same adjustment. See #11632. - When following the container demo tutorial, it was not possible to update Solr fields after adding additional metadata blocks. This has been fixed. See #11722 and #11723. ## API Updates @@ -93,8 +93,8 @@ The API endpoints `api/{dataverse-alias}/metadatablocks` and `/api/metadatablock The Notifications API has been updated in various ways: -- The JSON returned from [listing notifications](https://guides.dataverse.org/en/6.8/api/native-api.html#get-all-notifications-by-user) now includes a "displayAsRead" boolean to indicated if a notification has been read. See #11650 and #11664. -- You can get an count of unread notifications via a new [unreadCount](https://guides.dataverse.org/en/6.8/api/native-api.html#get-unread-count) API endpoint. +- The JSON returned from [listing notifications](https://guides.dataverse.org/en/6.8/api/native-api.html#get-all-notifications-by-user) now includes a "displayAsRead" boolean to indicate if a notification has been read. See #11650 and #11664. +- You can get a count of unread notifications via a new [unreadCount](https://guides.dataverse.org/en/6.8/api/native-api.html#get-unread-count) API endpoint. - You can mark a notification as read via a new [markAsRead](https://guides.dataverse.org/en/6.8/api/native-api.html#mark-notification-as-read) API endpoint. - The JSON can be returned using `inAppNotificationFormat`. See #11648 and #11696. - A bug was fixed where a NullPointerException was being thrown when retrieving notifications without a requestor. See #11703, and #11704. @@ -124,7 +124,7 @@ If the dataset/file is not public, the caller must authenticate and have permiss Authentication updates listed below were introduced as we work toward allowing the [new Dataverse frontend](https://github.com/IQSS/dataverse-frontend), a React-based Single Page Application (SPA). We list them here for completeness but unless you are experimenting with the new frontend or playing with OIDC directly, they probably will have no impact on your installation. -- We've strengthened the security of the `api-bearer-auth-use-builtin-user-on-id-match` feature flag. It will now only work when the provided bearer token includes an `idp` claim that matches the Keycloak Service Provider identifier. By enforcing this check, the risk of impersonation from other identity providers is significantly reduced, since they would need to be explici tly configured with this specific, non-standard identifier. See the list of [feature flags](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags), #11689, and #11763. +- We've strengthened the security of the `api-bearer-auth-use-builtin-user-on-id-match` feature flag. It will now only work when the provided bearer token includes an `idp` claim that matches the Keycloak Service Provider identifier. By enforcing this check, the risk of impersonation from other identity providers is significantly reduced, since they would need to be explicitly configured with this specific, non-standard identifier. See the list of [feature flags](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags), #11689, and #11763. - A new feature flag `api-bearer-auth-use-shib-user-on-id-match` supports the use of clients in instances that have historically allowed login via Shibboleth. Specifically, with this flag enabled, when an OIDC bridge is configured to allow OIDC login with validation by the bridged Shibboleth providers, users with existing Shibboleth-based accounts in Dataverse can log in to those accounts, thereby maintaining access to their existing content and retaining their roles. (For security reasons, Dataverse's current support for direct login via Shibboleth cannot be used in browser-based clients.) See the list of [feature flags](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags), #11605, and #11622. - A new feature flag `api-bearer-auth-use-oauth-user-on-id-match` supports the use of clients in instances that have historically allowed login via GitHub, ORCID, or Google. Specifically, with this flag enabled, when an OIDC bridge is configured to allow OIDC login with validation by the bridged OAuth providers, users with existing GitHub, ORCID, or Google accounts in Dataverse can log in to those accounts, thereby maintaining access to their existing content and retaining their roles. See the list of [feature flags](https://guides.dataverse.org/en/6.8/installation/config.html#feature-flags), #11671, and #11645. @@ -140,13 +140,13 @@ The getDatasetFileDetails data structure now contains "directoryLabel" (file pat ### PostgreSQL 13 Reaches EOL on 13 November 2025 -We mentioned this in the [Dataverse 6.6 release notes](https://github.com/IQSS/dataverse/releases/tag/v6.6), but as a reminder, according to https://www.postgresql.org/support/versioning/ PostgreSQL 13 reaches EOL on 13 November 2025. As mentioned in the [Installation Guide](https://guides.dataverse.org/en/6.8/installation/prerequisites.html#postgresql), we recommend running PostgreSQL 16 since it is the version we test with in our continous integration ([since](https://github.com/gdcc/dataverse-ansible/commit/8ebbd84ad2cf3903b8f995f0d34578250f4223ff) February 2025). The [Dataverse 5.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v5.4) explained the upgrade process from 9 to 13 (e.g. pg_dumpall, etc.) and the steps will be similar. If you have any problems, please feel free to reach out (see "getting help" in these release notes). +We mentioned this in the [Dataverse 6.6 release notes](https://github.com/IQSS/dataverse/releases/tag/v6.6), but as a reminder, according to https://www.postgresql.org/support/versioning/ PostgreSQL 13 reaches EOL on 13 November 2025. As mentioned in the [Installation Guide](https://guides.dataverse.org/en/6.8/installation/prerequisites.html#postgresql), we recommend running PostgreSQL 16 since it is the version we test with in our continuous integration ([since](https://github.com/gdcc/dataverse-ansible/commit/8ebbd84ad2cf3903b8f995f0d34578250f4223ff) February 2025). The [Dataverse 5.4 release notes](https://github.com/IQSS/dataverse/releases/tag/v5.4) explained the upgrade process from 9 to 13 (e.g. pg_dumpall, etc.) and the steps will be similar. If you have any problems, please feel free to reach out (see "getting help" in these release notes). ### For Dataverse instances that Use Shibboleth as Members of the InCommon Federation Please note that most of the known Dataverse instances that support Shibboleth logins do so without being part of InCommon, and therefore are not affected. All such instances will be able to continue using the old login workflow without needing to make any configuration changes. -For the relatively few instances using InCommon: Since InCommon discontinued their old-style federation metadata feed, a new Shibboleth implementation has been added to utilize the recommended replacements: the MDQ protocol and the WayFinder service. In order to continue using InCommon, such instances will need to modify their shibd configuration and their registration with Incommon, plus set a new feature flag. See the upgrade instructions below for details. See also #11404 and #11502. +For the relatively few instances using InCommon: Since InCommon discontinued their old-style federation metadata feed, a new Shibboleth implementation has been added to utilize the recommended replacements: the MDQ protocol and the WayFinder service. In order to continue using InCommon, such instances will need to modify their shibd configuration and their registration with InCommon, plus set a new feature flag. See the upgrade instructions below for details. See also #11404 and #11502. ## New Settings From e63da69cbbab322f77c71e5db7c08b568b1e6840 Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Tue, 16 Sep 2025 13:57:09 -0400 Subject: [PATCH 25/28] Revert "no security updates" This reverts commit 7c981890ef79383662623f4f830d9277b57b9a73. --- doc/release-notes/6.8-release-notes.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index deb217cb0a6..d9d78541535 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -120,6 +120,10 @@ New API calls have been added to retrieve the URLs needed to launch external too If the dataset/file is not public, the caller must authenticate and have permission to view the dataset/file. In such cases, the generated URL will include a callback token containing a signed URL the tool can use to retrieve all the parameters it is configured for. See Backward Incompatible Changes, below for a change to the JSON response. See #11760. +## Security Updates + +This release contains important security updates. If you are not receiving security notices, please sign up by following [the steps](https://guides.dataverse.org/en/latest/installation/config.html#ongoing-security-of-your-installation) in the guides. + ## Authentication Updates Authentication updates listed below were introduced as we work toward allowing the [new Dataverse frontend](https://github.com/IQSS/dataverse-frontend), a React-based Single Page Application (SPA). We list them here for completeness but unless you are experimenting with the new frontend or playing with OIDC directly, they probably will have no impact on your installation. From 56ede777b735a50547e4c738630fd072b5a9167d Mon Sep 17 00:00:00 2001 From: Philip Durbin Date: Wed, 17 Sep 2025 12:03:26 -0400 Subject: [PATCH 26/28] fix latest expected version before upgrade --- doc/release-notes/6.8-release-notes.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index d9d78541535..ee970687066 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -210,7 +210,7 @@ You are also very welcome to join the [Global Dataverse Community Consortium](ht Upgrading requires a maintenance window and downtime. Please plan accordingly, create backups of your database, etc. -These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.6. +These instructions assume that you've already upgraded through all the 5.x releases and are now running Dataverse 6.7.1. 0\. These instructions assume that you are upgrading from the immediate previous version. See [tags on GitHub](https://github.com/IQSS/dataverse/tags) for a list of versions. If you are running an earlier version, the only supported way to upgrade is to progress through the upgrades to all the releases in between before attempting the upgrade to this version. @@ -340,4 +340,4 @@ and See [How to configure a Shibboleth service provider (SP) to use MDQ](https://spaces.at.internet2.edu/display/MDQ/how-to-configure-shib-sp-to-use-mdq) for more information. -c. Set the feature flag `dataverse.feature.shibboleth-use-wayfinder=true`. \ No newline at end of file +c. Set the feature flag `dataverse.feature.shibboleth-use-wayfinder=true`. From 3e32cdf2c5b76fc4a6335ec72b7ef442806fcc0b Mon Sep 17 00:00:00 2001 From: Omer Fahim Date: Tue, 23 Sep 2025 14:50:47 -0400 Subject: [PATCH 27/28] Update doc/release-notes/6.8-release-notes.md typo fix --- doc/release-notes/6.8-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index ee970687066..7395061da8f 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -35,7 +35,7 @@ Linking or unlinking a dataset or dataverse now requires the new "LinkDataset" o - The updates to support keeping the history of curation status labels added in #11268 will incorrectly show curation statuses added prior to v6.7 as the current one, regardless of whether newer statuses exist. This bug has been fixed. See #11685. (As a work-around for 6.7 admins can add createtime dates (must be prior to when 6.7 was installed) to the curationstatus table for entries that have null createtimes. The code fix in this version properly handles null dates as indicating older/pre v6.7 curationstatuses.) - The addDataset and addDataverse API endpoints now trigger user notifications. See #1342 and #11696. - A bug introduced in v6.5 broke Handle parsing when using a lowercase shoulder. This is now fixed. See #11592. -- Due to changes in how the commons-lang3 Java library handles a non-ascii chararacter, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. `controlledvocabulary.language.magɨ_(madang_province)` has been changed to `controlledvocabulary.language.magi_(madang_province)` and `controlledvocabulary.language.magɨyi` has been changed to `controlledvocabulary.language.magiyi`. In the upgrade instructions below, we indicate that the citation metadata block should be reloaded. Translations will need to make the same adjustment. See #11632. +- Due to changes in how the commons-lang3 Java library handles a non-ascii character, two keys in the citation.properties and citation.tsv files have changed to include i instead of ɨ. `controlledvocabulary.language.magɨ_(madang_province)` has been changed to `controlledvocabulary.language.magi_(madang_province)` and `controlledvocabulary.language.magɨyi` has been changed to `controlledvocabulary.language.magiyi`. In the upgrade instructions below, we indicate that the citation metadata block should be reloaded. Translations will need to make the same adjustment. See #11632. - When following the container demo tutorial, it was not possible to update Solr fields after adding additional metadata blocks. This has been fixed. See #11722 and #11723. ## API Updates From 888f9d49a8f8352191d932fe9cd1a9ea66f2d51b Mon Sep 17 00:00:00 2001 From: Omer Fahim Date: Wed, 24 Sep 2025 13:56:11 -0400 Subject: [PATCH 28/28] Update doc/release-notes/6.8-release-notes.md update to step 6 Co-authored-by: Philip Durbin --- doc/release-notes/6.8-release-notes.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/release-notes/6.8-release-notes.md b/doc/release-notes/6.8-release-notes.md index 7395061da8f..8e60e4978ff 100644 --- a/doc/release-notes/6.8-release-notes.md +++ b/doc/release-notes/6.8-release-notes.md @@ -273,7 +273,7 @@ Please remember to update translations via [Dataverse language packs](https://gi If you have text customizations you can get the latest English files from . -6\. Update Solr schema and reindex +6\. Update Solr schema Due to changes in the Solr schema (the addition of field "datasetCount"), updating the Solr schema and reindexing is required.